Log environment details for forensic purposes #1994
Description
Our systems have a lot of complexity... We weave a tangled web, and sometimes we need to follow some of those sticky fibers back to their ends. The more pertinent information we have in our logs about a running system, the easier it will be to do that tracing.
Building on #1702 by further incorporating ideas described in cmu-delphi/delphi-epidata#987 , we can create a better view into the state and context of our running systems. Such datapoints will enable us to better explain and identify discrepancies in performance/behavior/output/etc.
For instance, from a dependency versioning perspective, such detailed environment info in the logs might have helped identify the problem in #1988 faster, or made it easier to narrow down the causes for #1966 .
TL;DR: save fingerprints of the stuff we run, like metadata about code versioning, git refs, docker builds, environment status, and library/dependency details.