Skip to content

Resolve a DRS URI into a workspace, against hosts named in advance - #80

Open
jorgeMFS wants to merge 3 commits into
datasource-interfacefrom
drs-datasource
Open

jorgeMFS wants to merge 3 commits into
datasource-interfacefrom
drs-datasource

Conversation

@jorgeMFS

@jorgeMFS jorgeMFS commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Closes F2 (#21). Based on datasource-interface (#69), the D1 it depends
on, and nothing else — it adds a third provider behind that seam, which is what
that file’s own docstring says F1 and F2 should be.

The design follows from one fact

drs://host/id resolves to https://host/ga4gh/drs/v1/objects/id. The URI
names its own host, and the URI comes from the client.
Following one unchecked
makes this service a request generator pointed wherever a workflow says — from
inside a TRE, at whatever that TRE can reach and the caller cannot.

So BIOCHEF_DRS_HOSTS is an allowlist with no default, and unset disables
the source. Same shape as BIOCHEF_LOCAL_ROOT, whose comment says it confines
itself because the client chooses the path. Here the client chooses the host.

Compact identifiers are refused. Resolving drs://prefix:accession means
asking a third-party resolver which host to contact — taking an endpoint out of
a document and then trusting it completely, which is a mistake this service has
already made twice.

I read the specification before writing this time, which is why the endpoints,
the access_id exchange and the checksum handling are the spec’s rather than my
guess at them.

What the audit then found

Redirects walked around the allowlist. urllib follows 3xx by default, so
an allowlisted DRS server answering 302 sent this service wherever it liked. The
allowlist checked the URI it was handed while the request ended up somewhere
else, which makes the control decorative. The DRS API now refuses redirects
outright; the bytes still follow them, because an access_url is expected to
bounce to storage and the allowlist never covered that host — but only while it
stays on https.

A download failing its checksum was left under the workflow’s own name,
holding whatever the server sent. That is the C1 gap reopened by a different
protocol. Bytes are staged, verified, then renamed.

A test that was worth nothing until the mutation said so

The staging test asserted "nothing is left behind" — and passed with the
staging removed
, because both designs clean up after a failure. The difference
only exists while the bytes are in flight.

It now watches the workspace on every chunk and asserts the workflow’s own file
never appears before the checksum has been checked. That version fails against
the unstaged one.

Other decisions

  • An object id becomes a path segment, so one containing a slash or a leading
    dot is refused before anything is opened — with a test asserting nothing is
    opened at all for a host outside the allowlist, since an allowlist applied
    after the request documents a policy already violated.
  • Checked against the sha-256 or md5 the object declares; one declaring
    neither is refused rather than taken on trust.
  • A server sending far more than its declared size is cut off, because the
    workspace writer has no reason to know what was claimed.

241 passed, 4 skipped. Nothing in the tests touches the network — the
transport is injected, so these are about what this service decides.

Note

I pushed one commit here with a pre-existing test red: the file that enumerates
the providers names them exactly, deliberately, and I added a third without
updating it. Fixed in the commit after, but it should not have gone out.

F2. Read the DRS specification first this time, and the endpoint shapes are its
own: hostname-based drs://host/id resolves to https on 443 under
/ga4gh/drs/v1/objects, and an object either carries an access_url directly or an
access_id to exchange at /access/{access_id}.

The design follows from one fact. A DRS URI names its own host and the URI comes
from the client, so following one unchecked makes this service a request
generator pointed wherever a workflow says, from inside a TRE, at whatever that
TRE can reach and the caller cannot. BIOCHEF_DRS_HOSTS is therefore an allowlist
with no default, and unset disables the source -- the same shape as
BIOCHEF_LOCAL_ROOT, whose comment says it confines itself because the client
chooses the path. Here the client chooses the host.

Compact identifiers are refused. Resolving drs://prefix:accession means asking a
third-party resolver which host to contact, which is taking an endpoint out of a
document and trusting it completely, and that is a mistake this service has
already made twice this week.

An object id becomes a path segment on the DRS server, so one containing a slash
or a leading dot is refused before anything is opened. A test asserts nothing is
opened at all for a host outside the allowlist, because an allowlist applied
after the request documents a policy already violated.

What arrives is checked against the sha-256 or md5 the object declares, and an
object declaring neither is refused rather than taken on trust. That is the gap
C1 exists to close and it does not stop being one because a different protocol
opened it. A server sending far more than its declared size is cut off, since
the workspace writer has no reason to know what was claimed.

An access_url may point at another host entirely, which is what a presigned S3
or GCS URL is, and that is why the allowlist covers the DRS server rather than
the bytes and why the checksum carries the weight.
The list was exact, deliberately, so that adding a provider is a decision
somebody makes rather than something that happens quietly. Adding drs without
updating it is exactly the case it was written for, and I pushed the previous
commit with it red.
Audit of the DRS source, and both findings are patterns this project has already
paid for once.

urllib follows 3xx by default, so an allowlisted DRS server answering 302 sent
this service wherever it liked, including hosts reachable only from inside. The
allowlist checks the URI it was handed and the request went somewhere else,
which makes the control decorative. The DRS API is now opened with redirects
refused outright. The bytes are the other case and keep following them, because
an access_url is expected to bounce to storage and the allowlist never covered
that host, but only while it stays on https.

A download that failed its checksum was left in the workspace under the name the
workflow asked for, holding whatever the server sent. That is the gap C1 closed
for tool bundles, reopened by a different protocol. Bytes are staged, verified,
and only then given the name.

The test for that was worthless at first and the mutation said so: removing the
staging left it passing, because it asserted nothing was left behind and both
designs clean up after a failure. The difference only exists while the bytes are
in flight, so the reader now looks at the workspace on every chunk and the
assertion is that the workflow's own file never appears until the checksum has
been checked. That version fails against the unstaged one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant