latch_registry_export

Latch Registry export to DuckDB

Usage

Log in to Latch and select the workspace:

latch login
latch workspace

Write a config

A TOML config lists the Registry tables to export. You can write it from the start, or generate a config for the workspace and then edit it by hand.

To generate a config, run:

uv run latch_registry_export generate-config --output tables.toml

The generated config has one [[tables]] entry for each table in the workspace. Each entry has a suggested name and lists every column in include_columns. If two tables get the same name, the table with the lowest id keeps it and the others get their id as a suffix. A comment above each entry gives the project and table display name. A table with no columns has no include_columns, so the export includes every column added to it later. Edit the file to remove tables, remove columns, or change names, then export it as usual. generate-config does not replace an existing file unless you set --overwrite, so it does not overwrite your edits by accident. It also accepts --log-level.

When you edit a generated config:

• If you remove a table, also remove the link columns in other tables that point to it. Otherwise the export fails because a link targets a table that is not in the config.
• The export uses only the columns in include_columns. Columns that someone adds to the Registry later are not exported. To export new columns automatically, replace include_columns with exclude_columns, or remove include_columns.

A config that you write from the start can be short:

[[tables]]
id = "11730"
name = "samples"   # optional DuckDB table-name override

[[tables]]
id = "12146"

To export only some columns of a table, set include_columns or exclude_columns (not both) to a list of exact Registry column keys:

[[tables]]
id = "11730"
exclude_columns = ["Transcript Sequence"]   # export all columns except these

[[tables]]
id = "12146"
include_columns = ["Gene", "Sample"]        # export only these columns

The export always includes the record name column. Keys are case-sensitive, and the export fails if a listed key is not a column of the table. If you exclude a link column, the export does not create its foreign key, so the linked table need not be in the config. The exporter still fetches all columns from the Registry, so column selection makes the output smaller but not the fetch faster.

Export

Export the tables to a DuckDB file:

uv run latch_registry_export export --config tables.toml --output registry.duckdb

Options for export:

• --page-size N — page size for streaming records from the Registry (default 100).
• --overwrite — replace an existing output file.
• --log-level LEVEL — one of DEBUG, INFO, WARNING, ERROR (default INFO). The tool never logs GraphQL request and response bodies, at any level.

The export writes provenance and data-quality metadata into the output database, alongside the exported tables:

• _export_run — one row with the tool, dependency, and workspace versions for the run.
• _export_tables — one row per exported table, with its row count.
• _export_columns — one row per exported column, including its foreign-key status.
• _export_issues — rows for values that could not convert cleanly (for example, an integer that overflows its target type).

The reproducibility target is content-equivalence of the DATA tables, plus _export_tables, _export_columns, and _export_issues. _export_run varies run-to-run by design (timestamp, dependency versions, page size), so exclude it when you compare two exports.

Migration

The exporter is now the export subcommand. Change latch_registry_export --config ... to latch_registry_export export --config .... All flags are the same.

Recommended Installation

Install the Python package and dependency management tool uv using official documentation.

Install the dependencies of the project with:

uv sync --locked

To check successful installation, run:

uv run latch_registry_export --help

Development and Testing

See the contributing guide for more information.