request help: upstream related header is missing when using external plugins

[yujinchoi-94]

Description

Hi,

I am using ext-plugin-post-req, ext-plugin-post-resp and sending logs to loki using loki-logger. However, when i enabled external plugins, upstream related header is missing in logs.

It seems that upstream related variables is using nginx variable. I tried to log them in the external plugin code but it returns nothing.

TestFilter

    @Override
    public void filter(HttpRequest request, HttpResponse response, PluginFilterChain chain) {
        LOG.error("[TestFilter - postReq]" + NginxVars.UPSTREAM_RESPONSE_TIME.getValue() + ": " + request.getVars(NginxVars.UPSTREAM_RESPONSE_TIME.getValue()));
        LOG.error("[TestFilter - postReq]" + NginxVars.UPSTREAM_ADDR.getValue() + ": " + request.getVars(NginxVars.UPSTREAM_ADDR.getValue()));
        LOG.error("[TestFilter - postReq]" + NginxVars.REQUEST_URI.getValue() + ": " + request.getVars(NginxVars.REQUEST_URI.getValue()));
        chain.filter(request, response);
    }

    @Override
    public void postFilter(PostRequest request, PostResponse response, PluginFilterChain chain) {
        response.setHeader(NginxVars.UPSTREAM_ADDR.getValue(),
                request.getVars(NginxVars.UPSTREAM_ADDR.getValue()));
        response.setHeader(NginxVars.UPSTREAM_RESPONSE_TIME.getValue(),
                request.getVars(NginxVars.UPSTREAM_RESPONSE_TIME.getValue()));

        LOG.error("[TestFilter - postRes]" + NginxVars.UPSTREAM_RESPONSE_TIME.getValue() + ": " + request.getVars(NginxVars.UPSTREAM_RESPONSE_TIME.getValue()));
        LOG.error("[TestFilter - postRes]" + NginxVars.UPSTREAM_ADDR.getValue() + ": " + request.getVars(NginxVars.UPSTREAM_ADDR.getValue()));
        LOG.error("[TestFilter - postRes]" + NginxVars.REQUEST_URI.getValue() + ": " + request.getVars(NginxVars.REQUEST_URI.getValue()));
        chain.postFilter(request, response);
    }

    @Override
    public List<String> requiredVars() {
        List<String> vars = new ArrayList<>();
        vars.add(NginxVars.HTTP_ACCEPT.getValue());
        vars.add(NginxVars.REQUEST_URI.getValue());
        vars.add(NginxVars.HTTP_OPENAPI_ORIGINAL_ACCEPT.getValue());
        vars.add(NginxVars.HTTP_OPENAPIKEY.getValue());
        vars.add(NginxVars.UPSTREAM_RESPONSE_TIME.getValue());
        vars.add(NginxVars.UPSTREAM_ADDR.getValue());
        return vars;
    }

Plugins Logs

2023-10-23 01:53:43,844 ERROR epollEventLoopGroup-2-5          - [TestFilter - postReq]upstream_response_time:
 2023-10-23 01:53:43,844 ERROR epollEventLoopGroup-2-5          - [TestFilter - postReq]upstream_addr:
 2023-10-23 01:53:43,844 ERROR epollEventLoopGroup-2-5          - [TestFilter - postReq]request_uri: /rest/.../...(omit for security)
 2023-10-23 01:53:43,862 ERROR epollEventLoopGroup-2-5          - [TestFilter - postRes]upstream_response_time:
 2023-10-23 01:53:43,862 ERROR epollEventLoopGroup-2-5          - [TestFilter - postRes]upstream_addr:
 2023-10-23 01:53:43,863 ERROR epollEventLoopGroup-2-5          - [TestFilter - postRes]request_uri: /rest/../...(omit for security)

Loki (without external plugins)

hide IP for security reason

Loki (with external plugins)

Here's my plugin configuration

      plugins:
        - config:
            client_headers:
              ...
            request_headers:
              ...
            request_method: GET
            upstream_headers:
              ...
            uri: '...'
          enable: true
          name: forward-auth
        - config:
            disable: false
            prefer_name: true
          enable: true
          name: prometheus
        - config:
            endpoint_addrs:
              - >-
                http://loki-loki-distributed-gateway.monitor.svc.cluster.local:80
            include_req_body: true
            include_req_body_expr:
              - - http_response_status
                - ~*
                - '200'
            include_resp_body: true
            include_resp_body_expr:
              - - http_response_status
                - ~*
                - '200'
            tenant_id: dev-openapi-logging
          enable: true
          name: loki-logger
        - config:
            headers:
              ...
            uri: ...
          enable: true
          name: proxy-rewrite
        - config:
            conf:
              - name: TestFilter
                value: |
                  test
          enable: true
          name: ext-plugin-post-req
        - config:
            conf:
              - name: TestFilter
                value: |
                  test
          enable: true
          name: ext-plugin-post-resp

Could you help me to get upstream related headers?
Thanks,

Environment

• APISIX version (run apisix version):

apisix version
/usr/local/openresty//luajit/bin/luajit /usr/local/apisix/apisix/cli/apisix.lua version
3.4.0

• Operating system (run uname -a):

uname -a
Linux dev-apisix-56c7dc6b9-m2mfx 5.10.192-183.736.amzn2.x86_64 #1 SMP Wed Sep 6 21:15:41 UTC 2023 x86_64 GNU/Linux

• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners: apisix-java-plugin-runner 0.4.0
• LuaRocks version, for installation issues (run luarocks --version):

[DaveRoey]

No one is maintaining this plugin anymore, let's use the Kong gateway!

[monkeyDluffy6017]

Contributions to the community are welcome!