Filesystem: Migrate from `HttpSource` to `fsspec.HTTPFileSystem`

[amotl]

About

It looks like fsspec's HTTPFileSystem is not yet used within omniload. It might be needed to address CDN providers like CloudFront, no?

• Introduce a http based resources support by filesystem source dlt-hub/dlt#3028
• Feature: Introduce support of http based resources for fs source dlt-hub/dlt#3029

Evaluation

CodeRabbit shared this suggestion:

HTTP / HTTPS files

HTTP files already work for a public CloudFront URL:

omniload ingest \
  --source-uri 'https://d123.cloudfront.net/data/events.parquet' \
  --source-table events \
  --dest-uri 'duckdb:///warehouse.duckdb' \
  --dest-table raw.events

However, this is the older, separate HttpSource, not fsspec.HTTPFileSystem:

• src/omniload/source/http/readers.py uses requests.get().
• It downloads the entire response into memory, despite passing stream=True.
• It only supports CSV, headerless CSV, JSON/JSONL, and Parquet.
• It does not share dlt_filesystem.source.format.registry or the richer filesystem reader options.
• It currently exposes no URI-level headers/auth, retry policy, byte limit, or redirect policy.

So yes: HTTP deserves integration into the filesystem family, but CloudFront support alone does not require HTTPFileSystem.

I recommend adding an HTTPFilesystemSource under dlt_filesystem.source.impl, registered for both http and https, which reuses resource_for_reader and the shared format registry. Use fsspec’s HTTPFileSystem for single-object access and configurable headers/storage options. Do not promise generic directory scans or globs: HTTP has no portable listing protocol, and CDN origins rarely expose indexes.

The resulting URI contract should keep the URL as the object address and use the existing fragment/hint model for format and reader options. Test public CloudFront, redirect handling, a signed URL, an authenticated-header case, compression, and a large Parquet file to ensure it no longer unconditionally buffers the whole response.