Improve ergonomics of sending streams

[einarmo]

I was recently migrating some code from requests to httpx, and came across some pitfalls related to streaming large streams. I need to be able to send BinaryIO objects raw over a connection somewhere, and critically I need to set a Content-Length on the messages I send.

In requests, this is possible to hack by (ab)using the super_len function, which tries to infer the length of the stream by guessing at len, fileno, etc. httpx does something similar, in its peek_filelike_length method, except it doesn't try to guess at len, only fileno.

Like requests, this falls back on tell and seek if this fails, which is where my issue happens. Not all streams are harmlessly seekable, so if you naively pass along a stream that isn't an actual file handle to the files argument, you risk loading the entire stream, which in my case could be hundreds of gigabytes of data.

In the end I ended up creating a Request manually and using the stream argument, which isn't exposed anywhere else from what I can tell. The ergonomics of this is also not great, since I then have to manually set all headers.

Personally I really dislike the whole "try to guess the file length" business, but that is what it is. There are a few things that could be done to greatly improve ergonomics here:

• Do not set or try to guess Content-Length or Transfer-Encoding if either are specified by the user. This one seems like a no-brainer to me, especially the AsyncIterable case in encode_content. If the user knows the content length, there is no real reason to set Transfer-Encoding: chunked or to try to guess the length.
• Create a protocol type or something else you can pass as RequestContent or somewhere else that encapsulates a "stream with length".

[ketpatil77]

For a raw upload with a trusted, known remaining length, you can use content= with a chunk iterator and an explicit Content-Length. client.build_request() still applies the client's normal headers, cookies, and authentication; you do not need to build an isolated Request with every header manually.

import io
import httpx

class NonSeekable:
    def __init__(self, data: bytes):
        self.buffer = io.BytesIO(data)
        self.read_sizes = []
    def read(self, size: int) -> bytes:
        self.read_sizes.append(size)
        return self.buffer.read(size)
    def seek(self, *args):
        raise AssertionError("seek must not be called")
    def tell(self):
        raise AssertionError("tell must not be called")

def chunks(source, chunk_size=64 * 1024):
    while chunk := source.read(chunk_size):
        yield chunk

payload = b"x" * 150000
source = NonSeekable(payload)

def handler(request: httpx.Request) -> httpx.Response:
    assert request.headers["content-length"] == str(len(payload))
    assert "transfer-encoding" not in request.headers
    assert request.headers["x-example"] == "preserved"
    assert request.read() == payload
    return httpx.Response(200)

with httpx.Client(
    headers={"X-Example": "preserved"}, transport=httpx.MockTransport(handler)
) as client:
    request = client.build_request(
        "POST", "https://example.com/upload", content=chunks(source),
        headers={"Content-Length": str(len(payload))},
    )
    assert source.read_sizes == []  # Building the request did not inspect the source.
    response = client.send(request)
    response.raise_for_status()
assert source.read_sizes and max(source.read_sizes) == 64 * 1024

For real I/O, replace the mock transport and pass your source object to chunks. The generator hides file-like length inference, so constructing this request does not call seek or tell. The source must support bounded read(size) and return bytes, with empty bytes indicating EOF.

The header must describe the remaining bytes this iterator will yield, not necessarily the source's full size. Do not guess it or reuse a consumed iterator for retries. The example is a raw-body upload, not multipart files=; this workaround does not change multipart length inference. Manage the source's lifetime separately.

References: streaming uploads, client request construction.

AI-assisted answer using OpenAI Codex. Tested with Python 3.11.9 and HTTPX 0.28.1: 150,000 bytes, bounded 64 KiB reads, no seek/tell, preserved client header, explicit Content-Length, and no Transfer-Encoding. MockTransport does not test wire-level enforcement of an incorrect length.