log correlation + webpack may not work

[trentm]

(I'm splitting out one of the listed issues on #1967 (comment) to this new issue.)

Scenario:

• A node.js project using winston, and @elastic/ecs-winston-format for ECS logging formatting.
• It is using elastic-apm-node for tracing and should also log correlation with the ecs formatter.
• It is also using webpack bundling.

Is there a way to configure and use webpack so that (a) the APM agent works and (b) log correlation works?
See coming docs from #2837 about using the agent with webpack. Those docs should potentially be updated if there are changes required to get the above to work.

Using 'externals' in webpack.config.js like this:

...
externals: {
    'elastic-apm-node': 'commonjs elastic-apm-node'
},
...

naively works to get instrumentation of core node modules only, e.g. the http module. It may work for adding other modules as well:

...
externals: {
    'elastic-apm-node': 'commonjs elastic-apm-node',
    'pg': 'commonjs pg'
},
...

However, it does not work for log correlation via:

...
externals: {
    'elastic-apm-node': 'commonjs elastic-apm-node',
    '@elastic/ecs-winston-format': 'commonjs @elastic/ecs-winston-format'
},
...

because I found that the built "dist/app.js" replaces the require('elastic-apm-node') in ecs-winston-format with the __webpack_require__, breaking the sniffing. Solutions:

1. Something about the more complete 'webpack-node-externals' module works (and for all packages). So this is probably our best current workaround answer. However it may defeat the user's use case for webpack, perhaps not.

   // webpack.config.js
   const nodeExternals = require('webpack-node-externals')
   ...
   externals: [nodeExternals()],
   ...

2. If we switch to the global var, this might just work. We should probably do that anyway, to handle the "weird, hard" case.

3. Perhaps, however, it would be better for log correlation here to be from the APM side. It shouldn't just be Elastic APM here. The circular dep issue might be different here then.

[sibelius]

can we build a "native" solution for bundlers?

you could hook when the webpack is compiling a given package like pg and modify to have your instrumentation version

[sibelius]

we can have a list of packages that needs to be external to make elastic apm instrumentation to work

[sibelius]

can elastic-apm-node logs what modules/dependencies/packages are being instrumented?

[trentm]

can we build a "native" solution for bundlers?

@sibelius Hi. A native solution for bundlers will likely require a separate package/plugin/something for each bundler (e.g. one for webpack, one for esbuild, etc.), assuming the given bundler supports customization like that. I believe doing so is possible, yes, but it will take some investigation and prioritizing doing that. It is on my radar, but nothing is currently planned.

we can have a list of packages that needs to be external to make elastic apm instrumentation to work

The best such list in here:

apm-agent-nodejs/lib/instrumentation/index.js

Lines 37 to 86 in 5c372b6

	var MODULES = [
	'@apollo/server',
	'@aws-sdk/smithy-client', // Instrument the base client which all AWS-SDK v3 clients extends
	['@elastic/elasticsearch', '@elastic/elasticsearch-canary'],
	'@node-redis/client/dist/lib/client',
	'@node-redis/client/dist/lib/client/commands-queue',
	'@opentelemetry/api',
	'@opentelemetry/sdk-metrics',
	'@redis/client/dist/lib/client',
	'@redis/client/dist/lib/client/commands-queue',
	'apollo-server-core',
	'aws-sdk',
	'bluebird',
	'cassandra-driver',
	'elasticsearch',
	'express',
	'express-graphql',
	'express-queue',
	'fastify',
	'finalhandler',
	'generic-pool',
	'graphql',
	'handlebars',
	['hapi', '@hapi/hapi'],
	'http',
	'https',
	'http2',
	'ioredis',
	'jade',
	'knex',
	'koa',
	['koa-router', '@koa/router'],
	'memcached',
	'mimic-response',
	'mongodb-core',
	'mongodb',
	'mysql',
	'mysql2',
	'next/dist/server/api-utils/node',
	'next/dist/server/dev/next-dev-server',
	'next/dist/server/next',
	'next/dist/server/next-server',
	'pg',
	'pug',
	'redis',
	'restify',
	'tedious',
	'undici',
	'ws'
	]

These are all the modules that the APM agent will possible instrument.

can elastic-apm-node logs what modules/dependencies/packages are being instrumented?

It often does log at "debug" level when it is instrumenting a particular part of a module, e.g.:

agent.logger.debug('shimming generic-pool.Pool')
agent.logger.debug('shimming koa-router prototype.match function')
agent.logger.debug('wrapping fastify build function')

However, it does not currently do so consistently.

[sibelius]

I would like to see logs that packages that should be instrumented but were not instrumented

like

we tried to instrument mongodb but didn't work

but I think this won't be easy or even possible

[trentm]

we tried to instrument mongodb but didn't work

but I think this won't be easy or even possible

The issue in a bundler is that the hook the APM agent puts on require() doesn't get called at all, so the APM agent cannot know when mongodb is require()d. So, yes, in the current state of things, it isn't possible for the APM agent to warn at runtime.

[sibelius]

here is an article https://dev.to/woovi/using-webpack-to-slim-your-docker-image-1b1n

of how we are using elastic apm with webpack (bundling), and enabling instrumentation

we move all required packages that need instrumentation outside bundling, as externals

and install them in the docker afterwards