Include stack traces when we recover a panic in EndSpan

When we panic inside a span, right now you only end up seeing the panic message, e.g. EOF, but not the associated stack trace.
Because we also recover in main, we get a stack trace on the main function, but it would be much more helpful to get the stack on the originating span.

I've updated our recover in EndSpan so that we record the stack trace when a traced function panics. Eventually we won't need the last resort recover in main, but we aren't quite there yet so the panic is double logged.

Signed-off-by: Carolyn Van Slyck <[email protected]>

pkg/tracing/traceLogger.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"runtime"
+	"runtime/debug"
 	"strings"
 
 	"go.opentelemetry.io/otel/attribute"
@@ -123,10 +124,12 @@ func newTraceLogger(ctx context.Context, span trace.Span, logger *zap.Logger, tr
 func (l traceLogger) EndSpan(opts ...trace.SpanEndOption) {
 	defer l.span.End(opts...)
 
-	// If there was a panic, mark the span
-	if p := recover(); p != nil {
-		l.Errorf("panic: %s", p)
-		panic(p) // retrow
+	// If there was a panic, mark the span and include the stack trace
+	if panicErr := recover(); panicErr != nil {
+		l.Error(fmt.Errorf("%s", panicErr),
+			attribute.Bool("panic", true),
+			attribute.String("stackTrace", string(debug.Stack())))
+		panic(panicErr) // rethrow
 	}
 }