BC artifact downloads from bcartifacts AFD endpoint highly variable on hosted CI runners (30s → 5-6 min)

[StefanMaron]

Symptom

When downloading BC artifacts from bcartifacts-exdbf9fwegejdqak.b02.azurefd.net on hosted CI runners (GitHub Actions ubuntu-latest, also reproduced on hosted Azure DevOps agents), wall-clock download time varies wildly between runs against the same artifact and from the same runner pool:

• Fast path (typical): ~30 s for app (~850 MB) + platform (~1.3 GB) — i.e. ~70-90 MB/s aggregate.
• Slow path (occasional): 5-6 min for the same payload — effective rate drops to <5 MB/s.

There's no error, no retry, no timeout — the GETs just complete slowly. So curl --retry doesn't help: the connection is "successful", just slow.

Evidence

A recent run on a Microsoft-hosted ubuntu-latest runner that hit the fast path:

[artifacts] App URL:      https://bcartifacts-exdbf9fwegejdqak.b02.azurefd.net/sandbox/27.5.46862.48612/w1
[artifacts] Platform URL: https://bcartifacts-exdbf9fwegejdqak.b02.azurefd.net/sandbox/27.5.46862.48612/platform
[artifacts] Downloading app + platform in parallel...
[artifacts] Downloaded: app=856M platform=1.3G in 23546ms (~88 MB/s)
[artifacts] Extracting app...
[artifacts] Extracted in 27094ms | Total: 50654ms | Disk: 3.0G

So the network and runner are clearly capable of ~88 MB/s. But other runs against the same URL on the same runner type take 5-6 min for the same bytes.

Download shape

Two concurrent HTTP/1.1 GETs (one for <type>/<version>/<country>, one for <type>/<version>/platform), each on a single TCP connection:

curl -sSL --retry 3 --retry-all-errors --http1.1 "$APP_URL"      -o app.zip      &
curl -sSL --retry 3 --retry-all-errors --http1.1 "$PLATFORM_URL" -o platform.zip &
wait

--http1.1 is intentional — without it, HTTP/2 multiplexing on AFD made things consistently worse in our testing.

Suspected cause

The host is Azure Front Door (.azurefd.net), which has documented per-route knobs:

• responseTimeoutSeconds (origin-fetch + total response budget)
• per-connection bandwidth caps / origin pool sizing
• origin fetch behaviour on cold cache miss

The slow runs have the shape of a per-connection bandwidth cap that's set tight, or an origin-fetch timeout that's just barely large enough on the fast path and falls back to a slower path on the slow one. Since we use one connection per file, there's no way for the client to compensate when AFD throttles that connection.

What's been tried client-side (and why it doesn't fix it)

• ✅ Forcing HTTP/1.1 — helps, but doesn't eliminate the slow path
• ✅ Parallelising app + platform — lifts aggregate, but each connection is still capped
• ❌ curl --retry — the slow connection completes "successfully", so retry never triggers
• ❌ Range-request parallelism (4-8 byte ranges per file) would multiply throughput on the slow path, but it's a non-trivial rewrite and shouldn't be necessary if AFD limits are right

The ask

Could the AFD route limits for bcartifacts-exdbf9fwegejdqak.b02.azurefd.net be reviewed? Specifically:

1. responseTimeoutSeconds and origin-fetch timeout for the /sandbox/<version>/... and /onprem/<version>/... routes
2. Whether there's a per-connection bandwidth cap, and if so what it's set to
3. Whether cold-cache misses on these blobs go through a slower origin path that could be warmed up or cached more aggressively

Stefan mentioned in our discussion that he remembers a previous request like this leading to some AFD values being adjusted — same thing, please. 🙂

Affects

• navcontainerhelper artifact downloads (same URL, same shape)
• The download-artifacts.sh used by StefanMaron/MsDyn365Bc.On.Linux (Linux BC runtime), which is what surfaced this for me
• Likely anyone hitting bcartifacts-exdbf9fwegejdqak.b02.azurefd.net from CI

cc Freddy Kristiansen (@freddydk)

[aholstrup1]

This variability is most likely explained by whether the artifact you are downloading is in our Azure Front Door cache or not. GitHub Runners are located in the US whereas the BCArtifacts are located in West Europe, so when you don't hit the local cache, the runner has to fetch the artifact from a Storage Account in West Europe. The storage accounts contain a lot of artifacts so caching them all is probably not feasible from our side.

You could try using https://github.com/actions/cache to improve the caching on your repo so that you don't have to fetch the artifacts from our storage account every time.

[StefanMaron]

Thanks for the quick reply. The cache hit/miss explanation doesn't really fit my pattern though — test-versions.yml runs the same 6 artifacts over and over all day, and I can pull two runs from the same SHA hours apart against the byte-identical URL where one downloads in ~30s and the other takes 5+ minutes. Happy to share specific run IDs and the per-phase timings, workflow-summary.sh already records all of it.

What I'm actually trying to do here is build a pipeline that runs cold, end to end, in under 5 minutes. If the artifact download alone takes 5 minutes on the slow path, that pretty much doubles my total pipeline time just on download — which is the whole reason I noticed this in the first place. On the fast path the same single connection saturates at ~88 MB/s, so the runner and the network are clearly capable; something between curl and the storage account is intermittently limiting that connection. And the distribution is bimodal — 30s OR 5min, very little in between. If it were purely a US runner → West Europe origin distance thing, I'd expect every run to be uniformly slow. The bimodal pattern looks mcch more like a per-connection cap or an origin path the route falls into intermittently than a geographic penalty.

On the actions/cache suggestion — I get the idea but it doesn't really work for what I'm doing here. The full BC artifact set is around 3 GB per version, and Microsoft ships hotfixes constantly, so even covering 4-5 versions blows past the 10 GB per-repo cache quota almost immediately. To make caching feasible I'd basically have to lock my pipelines to a weekly release cadence, which defeats the point. And I'm deliberately not caching in this repo specifically because I'm building the pipelines, not just running them — I need to see the real cold-start numbers, otherwise I'm tuning against a benchmark that doesn't reflect what consumers will actually experience. For real-world projects local caching is the right answer, sure, but that's a workaround, not a fix. It also doesn't help anyone on Azure DevOps, and it doesn't help cold runs (forks, new branches, first push of the day, etc.) — all of which still hit AFD raw and pay the slow-path tax.

So really my ask hasn't changed: could the AFD route limits for bcartifacts-exdbf9fwegejdqak.b02.azurefd.net get reviewed? Specifically responseTimeoutSeconds and whether there's a per-connection bandwidth cap on /sandbox//.... Even just confirming the cap is set and what it's set to would tell me whether range-request parallelism is the right next workaround on my end or whether I should just keep waiting for a real fix.

For reference https://github.com/StefanMaron/MsDyn365Bc.On.Linux/actions/workflows/test-versions.yml