proxy option not configured for webpagetest results polling

[garnaud]

Hi,

I would like to use a webpagetest server behind my company proxy but the result polling is blocked.
It seems that the proxy parameter is well configured for the webpagetest server call (I can see the launched job on the server) but not for retrieving the test result.

I use docker image sitespeedio/sitespeed.io:7.0.1.

My configuration file is:

{
  "webpagetest":{
    "host": "some.webpagetest.com",
    "proxy": "http://my.company.proxy:80"
  }
}

The stack error:

[2018-06-04 16:21:00] ERROR: Error creating WebPageTest result  {"cause":{"code":"ENOTFOUND","errno":"ENOTFOUND","syscall":"getaddrinfo","hostname":"some.webpagetest.com","host":"some.webpagetest.com","port":80},"isOperational":true,"code":"ENOTFOUND","errno":"ENOTFOUND","syscall":"getaddrinfo","hostname":"some.webpagetest.com","host":"some.webpagetest.com","port":80}
    at errnoException (dns.js:50:10)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:92:26)
    at Socket.Readable.on (_stream_readable.js:772:35)
    at tickOnSocket (_http_client.js:643:10)
    at onSocketNT (_http_client.js:673:5)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
    at ClientRequest.onSocket (_http_client.js:661:11)
    at handleSocketCreation_Inner (_http_agent.js:351:15)
    at oncreate (_http_agent.js:224:5)
    at Agent.createSocket (_http_agent.js:229:5)
    at Agent.addRequest (_http_agent.js:185:10)
    at new ClientRequest (_http_client.js:258:16)
    at request (http.js:38:10)
    at Object.get (http.js:42:13)
    at WebPageTest.get (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:101:19)
From previous event:
    at Object.analyzeUrl (/usr/src/app/lib/plugins/webpagetest/analyzer.js:24:22)
    at Object.processMessage (/usr/src/app/lib/plugins/webpagetest/index.js:175:12)
    at queue.process.message (/usr/src/app/lib/core/queueHandler.js:208:32)
    at drainItem (/usr/src/app/node_modules/concurrent-queue/index.js:96:21)
    at Immediate.drain [as _onImmediate] (/usr/src/app/node_modules/concurrent-queue/index.js:77:84)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)
    at WebPageTest.get (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:151:6)
    at WebPageTest.api (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:198:9)
    at WebPageTest.getTestResults (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:295:14)
    at ontimeout (timers.js:482:11)
    at tryOnTimeout (timers.js:317:5)
    at Timer.listOnTimeout (timers.js:277:5)
    at WebPageTest.poll (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:362:17)
    at WebPageTest.testCallback (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:381:10)
    at callbackYield (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:159:14)
    at WebPageTest.apiCallback (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:228:7)
    at IncomingMessage.onEnd (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:144:11)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
    at IncomingMessage.Readable.on (_stream_readable.js:772:35)
    at ClientRequest.getResponse (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:120:11)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at HTTPParser.parserOnIncomingClient (_http_client.js:551:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:115:23)
    at Socket.socketOnData (_http_client.js:440:20)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10)
    at TCP.onread (net.js:607:20)
    at ClientRequest.once (events.js:341:8)
    at new ClientRequest (_http_client.js:164:10)
    at request (http.js:38:10)
    at Object.get (http.js:42:13)
    at WebPageTest.get (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:101:19)
    at WebPageTest.api (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:198:9)
    at WebPageTest.runTest (/usr/src/app/node_modules/webpagetest/lib/webpagetest.js:434:16)
    at Object.analyzeUrl (/usr/src/app/lib/plugins/webpagetest/analyzer.js:24:22)
    at Object.processMessage (/usr/src/app/lib/plugins/webpagetest/index.js:175:12)
    at queue.process.message (/usr/src/app/lib/core/queueHandler.js:208:32)
    at drainItem (/usr/src/app/node_modules/concurrent-queue/index.js:96:21)
    at Immediate.drain [as _onImmediate] (/usr/src/app/node_modules/concurrent-queue/index.js:77:84)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)

[soulgalore]

Hi @garnaud thanks for creating the issue. Hmm we use https://github.com/marcelduran/webpagetest-api in the backend to connect with the WebPageTest server but I don't see a proxy option in the documentation for that, can you show we where you found it we can look into it?

Best
Peter

[garnaud]

Hi,

I had to investigate code but I hadn't see it was an external library.

You can see the executed code here. And the call to the method here:

    get.call(this, config, pathname, options.proxy, options.agent, function apiCallback(err, data, info) {
...

I was lucky that options comes from sitespeed.io configuration element webpagetest:

{
  "webpagetest":{
    "host": "some.webpagetest.com",
    "proxy": "http://my.company.proxy:80"
  }
}

But that work only for the first call to the api. For polling the result test, it doesn't reuse the configuration. I don't know why and I'm a little bit lost in nodejs code :)

Do you think I need to open an issue to https://github.com/marcelduran/webpagetest-api ?

Best

Guillaume

[garnaud]

gasp, an issue since 2014 :(

catchpoint/WebPageTest.api-nodejs#31

[soulgalore]

@garnaud the webpagetest api is often open for PRs so if you have time and try to fix it I think Marcel can merge it.

[soulgalore]

Closing this but if it's fixed in the webpagetest api we can implement it here too.