feat: OpenTelemetry configuration and BaseApp instrumentation #25516
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| _ "cosmossdk.io/api/cosmos/app/v1alpha1" | ||
| fmt "fmt" | ||
| io "io" | ||
| reflect "reflect" |
Check notice
Code scanning / CodeQL
Sensitive package import Note
| _ "cosmossdk.io/api/cosmos/app/v1alpha1" | ||
| fmt "fmt" | ||
| io "io" | ||
| reflect "reflect" |
Check notice
Code scanning / CodeQL
Sensitive package import Note
| import ( | ||
| fmt "fmt" | ||
| io "io" | ||
| reflect "reflect" |
Check notice
Code scanning / CodeQL
Sensitive package import Note
| _ "cosmossdk.io/api/cosmos/app/v1alpha1" | ||
| fmt "fmt" | ||
| io "io" | ||
| reflect "reflect" |
Check notice
Code scanning / CodeQL
Sensitive package import Note
telemetry/config.go
Outdated
| // InitializeOpenTelemetry initializes the OpenTelemetry SDK. | ||
| // We assume that the otel configuration file is in `~/.<your_node_home>/config/otel.yaml`. | ||
| // An empty otel.yaml is automatically placed in the directory above in the `appd init` command. | ||
| func InitializeOpenTelemetry(homeDir string) error { |
There was a problem hiding this comment.
I'd prefer we just pass in a filename here instead of a dir. Then the user can use whatever filename they want. That would be better for testing too
There was a problem hiding this comment.
Also, a more future proof approach here would be either an options struct, ex:
type Options struct {
ConfigFilename string
}
func InitializeOpenTelemetry(Options) error {or vararg Options ex:
type Option interface {
applyOption(*options)
}
func WithConfigFile(filename string) Option { ...}
func InitializeOpenTelemetry(...Option) error {otel go universally uses the ...Option pattern FWIW
There was a problem hiding this comment.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25516 +/- ##
==========================================
+ Coverage 70.35% 70.43% +0.08%
==========================================
Files 825 833 +8
Lines 53809 54387 +578
==========================================
+ Hits 37857 38308 +451
- Misses 15952 16079 +127
... and 10 files with indirect coverage changes 🚀 New features to boost your workflow:
|
baseapp/abci.go
Outdated
| // are used as part of gRPC paths | ||
| if grpcHandler := app.grpcQueryRouter.Route(req.Path); grpcHandler != nil { | ||
| return app.handleQueryGRPC(grpcHandler, req), nil | ||
| return app.handleQueryGRPC(grpcHandler, req, ctx), nil |
There was a problem hiding this comment.
makes more sense to have the ctx be first argument here
server/start.go
Outdated
| traceCleanupFn() | ||
|
|
||
| // shutdown telemetry with a 5 second timeout | ||
| shutdownCtx, _ := context.WithTimeout(context.Background(), 5*time.Second) |
There was a problem hiding this comment.
why is it ok to discard this cancel?
telemetry/config.go
Outdated
| ) | ||
|
|
||
| var ( | ||
| sdk *otelconf.SDK |
There was a problem hiding this comment.
can we create a more descriptive name for this var?
There was a problem hiding this comment.
|
|
||
| // parse cosmos extra config | ||
| var extraCfg extraConfig | ||
| err = yaml.Unmarshal(bz, &extraCfg) |
There was a problem hiding this comment.
so, we can support all of these as sinks simultaneously?
There was a problem hiding this comment.
yes - these are all independent exporters. it just tells otel to forward all:
- metrics if metricsfile is set
- traces if tracesfile is set
- logs if logsfile is set
Eric-Warehime
left a comment
Eric-Warehime
left a comment
There was a problem hiding this comment.
lgtm--please document additional config values.
| type cosmosExtra struct { | ||
| TraceFile string `json:"trace_file" yaml:"trace_file" mapstructure:"trace_file"` | ||
| MetricsFile string `json:"metrics_file" yaml:"metrics_file" mapstructure:"metrics_file"` | ||
| MetricsFileInterval string `json:"metrics_file_interval" yaml:"metrics_file_interval" mapstructure:"metrics_file_interval"` | ||
| LogsFile string `json:"logs_file" yaml:"logs_file" mapstructure:"logs_file"` | ||
| InstrumentHost bool `json:"instrument_host" yaml:"instrument_host" mapstructure:"instrument_host"` | ||
| InstrumentRuntime bool `json:"instrument_runtime" yaml:"instrument_runtime" mapstructure:"instrument_runtime"` | ||
| Propagators []string `json:"propagators" yaml:"propagators" mapstructure:"propagators"` | ||
| } |
There was a problem hiding this comment.
Can you document what these fields are and the potential valid values for them?
There was a problem hiding this comment.
I would also note that this stuff should be considered experimental. Otel contrib will update their configuration setup upstream and based on what they do we may want to change this. I expect they will include some native support for dumping to the file soon. They should be adding support for propagators soon too.
There was a problem hiding this comment.
|
lint failing just on deprecated fields |
github-actions
bot
added
C:x/staking
C:x/gov
C:x/slashing
C:x/distribution
should be fixed |
Description
This PR:
telemetry/package via https://pkg.go.dev/go.opentelemetry.io/contrib/otelconf/v0.3.0telemetry/which are based on https://pkg.go.dev/github.com/hashicorp/go-metrics which is under-maintained and requires mutex locks and map lookups on every telemetry methodBaseAppwith OpenTelemetry spans and block and tx counter metricsTestingMainfunction for tests to export telemetry dataconfigures thelog/slogdefault logger to send logs to OpenTelemetry and allows otelslog bridged to be used for logging. NOTE: this leaves a bit of a disconnect between the SDK's existing logging infrastructure which currently just writes to stdout - this can either be default with in this PR or dealt with in a follow upThe OpenTelemetry go libraries are very actively maintained, most vendors in the space are adding OpenTelemetry support and generally it seems like the industry is headed in this direction. Much of our existing telemetry code is to configure basic telemetry exporting, but with otelconf declarative config, we don't need to maintain any of this ourselves and the out of the box experience is quite simple even for usage in testing.
Closes: SDK-427