Missing Metrics

[gkramer]

Hey guys,

Wondering if someone could assist with an issue I'm having with BigGraphite [BG]. It currently receives a large number of metrics, but appears to drop a noticable proportion randomly... this was highlighted when looking at metrics from Apache Spark, which has frequent gaps per hour (of one minute each).

Infrastructure Setup:

• Within EKS (1.20)
• internal AWS NLB
• Traffic Flow: NLB -> Carbon Container -> {elasticsearch + cassandra}
• Carbon: Running inside an upstream Alpine container
• PS:
  1 root 0:00 {entrypoint} /bin/sh /entrypoint
  49 root 0:00 runsvdir -P /etc/service
  51 root 0:00 runsv bg-carbon
  52 root 0:03 runsv brubeck
  53 root 0:00 runsv carbon
  54 root 0:00 runsv carbon-aggregator
  55 root 0:03 runsv carbon-relay
  56 root 0:03 runsv collectd
  57 root 0:00 runsv cron
  58 root 0:00 runsv go-carbon
  59 root 0:00 runsv graphite
  60 root 0:00 runsv nginx
  61 root 0:03 runsv redis
  62 root 0:00 runsv statsd
  63 root 0:00 tee -a /var/log/carbon.log
  65 root 0:00 tee -a /var/log/carbon-relay.log
  68 root 0:00 tee -a /var/log/statsd.log
  69 root 0:01 {gunicorn} /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn wsgi --pythonpath=/opt/graphite/webapp/graphite --preload --threads=1 --worker-class=sync --workers=4 --limit-request-line=0 --max-requests=1000 --timeout=65 --bind=0.0
  70 root 0:09 {node} statsd /opt/statsd/config/tcp.js
  71 root 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
  76 root 0:00 /usr/sbin/crond -f
  79 nginx 0:00 nginx: worker process
  80 nginx 0:00 nginx: worker process
  81 nginx 0:00 nginx: worker process
  82 nginx 0:00 nginx: worker process
  85 root 0:35 tee -a /var/log/bg-carbon.log
  86 root 45:27 /opt/graphite/bin/python3 /opt/graphite/bin/bg-carbon-cache start --nodaemon --debug
  88 root 0:00 tee -a /var/log/carbon-aggregator.log
  156 root 0:41 {gunicorn} /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn wsgi --pythonpath=/opt/graphite/webapp/graphite --preload --threads=1 --worker-class=sync --workers=4 --limit-request-line=0 --max-requests=1000 --timeout=65 --bind=0.0
  157 root 0:49 {gunicorn} /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn wsgi --pythonpath=/opt/graphite/webapp/graphite --preload --threads=1 --worker-class=sync --workers=4 --limit-request-line=0 --max-requests=1000 --timeout=65 --bind=0.0
  158 root 0:46 {gunicorn} /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn wsgi --pythonpath=/opt/graphite/webapp/graphite --preload --threads=1 --worker-class=sync --workers=4 --limit-request-line=0 --max-requests=1000 --timeout=65 --bind=0.0
  159 root 0:47 {gunicorn} /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn wsgi --pythonpath=/opt/graphite/webapp/graphite --preload --threads=1 --worker-class=sync --workers=4 --limit-request-line=0 --max-requests=1000 --timeout=65 --bind=0.0

I can see traffic coming in to the interface (tcpdump/tcpflow), and can see logs to bg-carbon.log with references to 'cache query', but almost no datapoint logs for spark metrics.

Any assistance in troubleshooting would be greatly appreciated!

[geobeau]

If you look on the Cassandra side:

• do you have errors?
• do you see a drop in write ops when you notice the drops?

Inside your container, does carbon restarts by itself?

[gkramer]

Apologies for the delay in coming back to you!

I've rebuilt the cache container to only run carbon cache. Previously, it was running statds+carbon+etc, and this was all under supervisord, or similar. The container now runs carbon exclusively.

At first, and under low load, there were no metric drop-outs at all. We were shipping all metrics for spark, and it was bulletproof. As soon as we started shipping more metrics from other services, we began to see drop-outs of 1-2 minutes. across multiple metrics. Another interesting observation is that metrics appear to disappear at times - I'm not sure if they are being overwritten by null values? What I can tell you is that metrics are being fed into now what is a dedicated carbon ingress, and being inspected from another graphite endpoint, so whisper data is not a thing.

I've made multiple tweaks to the configs, but I'm at a bit of a loss as to how to eradicate the intermittent data loss.

Any help would be GREATLY appreciated!

TIA!