TT-12073 Tyk Splunk Pump Ignores Custom meta.fields Configuration, Always Returns Default Fields

[ebnou28]

When configuring the Tyk Pump for Splunk with custom fields specified in the meta.fields section of pump.conf, the expected behavior is for these fields to populate the Splunk events. Cf this part of the code:

tyk-pump/pumps/splunk.go

Line 209 in 1c28c84

if len(p.config.Fields) > 0 {

However, despite correct configuration, Tyk Pump disregards these specifications and only sends the default set of fields.

My configuration setup:

I'm running this docker-compose with 3 services :

• Tyk Pump version: latest
• Tyk Gateway v2.9.2
• Redis as the analytics storage

version: "3"

services:
  redis:
    image: redis:5-alpine
    ports:
      - 6379:6379
    networks:
    - tyk
  pump:
    image: tykio/tyk-pump-docker-pub:latest
    depends_on:
      - redis
    networks:
      - tyk
    volumes:
      - ./data/pump.conf:/opt/tyk-pump/pump.conf
    # env_file: ./pump.env
    environment:
      - TYK_PMP_PUMPS_SPLUNK_META_FIELDS=host
  tyk:
    image: tykio/tyk-gateway:v2.9.2
    depends_on:
      - redis
    networks:
      - tyk
    ports:
     - 8080:8080
    tty: true
    volumes:
      - ./tyk.standalone.conf:/opt/tyk-gateway/tyk.conf
      - ./apps:/opt/tyk-gateway/apps
    environment:
      - TYK_GW_ENABLEANALYTICS=true
      - TYK_GW_HEALTHCHECK_ENABLEHEALTHCHECKS=false
    command: tyk

networks:
  tyk:
    driver: bridge

Here is my Tyk pump.conf file (where I'm configuring the Splunk conf to send to a collector_url of an app running on my host machine on port 7000 :

{
    "analytics_storage_type": "redis",
    "analytics_storage_config": {
        "type": "redis",
        "host": "redis",
        "port": 6379,
        "hosts": null,
        "username": "",
        "password": "",
        "database": 0,
        "optimisation_max_idle": 100,
        "optimisation_max_active": 0,
        "enable_cluster": false
    },
    "purge_delay": 10,
  "pumps": {
      "splunk": {
        "type": "splunk",
        "meta": {
          "collector_token": "test",
          "collector_url": "http://host.docker.internal:7000/",
          "ssl_insecure_skip_verify": true,
          "obfuscate_api_keys": true,
          "obfuscate_api_keys_length": 10,
          "enable_batch":true,
          "max_retries": 2,
          "fields": [
            "method",
            "host",
            "path",
            "raw_path",
            "content_length",
            "user_agent",
            "response_code",
            "api_key",
            "time_stamp",
            "api_version",
            "api_name",
            "api_id",
            "org_id",
            "oauth_id",
            "raw_request",
            "request_time",
            "raw_response",
            "ip_address",
            "geo",
            "network",
            "latency",
            "tags",
            "alias",
            "track_path"
        ]
        }
      }
    },
    "dont_purge_uptime_data": true
}

Issue: even with fields explicitly set in the meta.fields field, I'm still receiving on the default list as in this else statement

tyk-pump/pumps/splunk.go

Line 226 in 1c28c84

} else {

ps: I'm receiving the collector token sent in the config, which makes me wonder whether the statement len(p.config.Fields) > 0 is true or not.

---

Currently, the only method I've found to successfully retrieve the desired fields involves enabling "enable_detailed_recording": true in the tyk.standalone.conf file (utilized for the Tyk Gateway). This approach provides me with the raw_request, which includes a significant amount of the targeted data.

Is my pump.conf configuration might be incorrect ? or is there an alternative method to get these fields without having to depend encoded raw_request?

I appreciate any insights.
Thanks