spec refresh

Documentation restructure (consolidated Specification section, developer guide grouped
by audience, refreshed concepts/user-guide) plus a specification of the on-disk/on-wire
encoding format (validity contract, per-encoding byte layouts, array/scalar/file/dtype
framing) and a docs-conformance harness that derives the stable-encoding set from the code.

Signed-off-by: mprammer <martin@spiraldb.com>
Co-Authored-By: Claude <noreply@anthropic.com>

Author: mprammer

.github/workflows/ci.yml

@@ -64,6 +64,17 @@ jobs:
             -c .yamllint.yaml \
             .github/
 
+  docs-conformance:
+    name: "Docs conformance"
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6
+      - name: Check docs facts match their source of truth
+        run: python3 scripts/check-docs-conformance.py
+      - name: Self-test the conformance checker (catch matcher regressions)
+        run: python3 scripts/check-docs-conformance.py --self-test
+
   python-lint:
     name: "Python (lint)"
     runs-on: >-

.github/workflows/docs.yml

@@ -20,6 +20,12 @@ jobs:
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v7
+      - name: Check docs conformance (fail before publishing stale docs)
+        # Runs FIRST, before the changelog generator writes generated release-note pages into docs/
+        # (their historical text would otherwise be scanned and could fail the current-docs lint).
+        run: |
+          python3 scripts/check-docs-conformance.py
+          python3 scripts/check-docs-conformance.py --self-test
       - uses: ./.github/actions/setup-rust
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}

README.md

@@ -6,7 +6,7 @@
 [![CodSpeed Badge](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/vortex-data/vortex)
 [![Crates.io](https://img.shields.io/crates/v/vortex.svg)](https://crates.io/crates/vortex)
 [![PyPI - Version](https://img.shields.io/pypi/v/vortex-data)](https://pypi.org/project/vortex-data/)
-[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark)
+[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark_2.13)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark_2.13)
 [![codecov](https://codecov.io/github/vortex-data/vortex/graph/badge.svg)](https://codecov.io/github/vortex-data/vortex)
 [![Cite](https://img.shields.io/badge/cite-CITATION.cff-blue)](CITATION.cff)
 

docs/_extra/_redirects

@@ -0,0 +1,16 @@
+# Cloudflare Pages redirects for docs.vortex.dev.
+# Copied verbatim to the site root via `html_extra_path` in conf.py.
+# More specific rules must come before broader splat rules.
+
+# The internals/serialization page was split across several specification pages
+# (array-format, ipc-format, reading-a-file); send it to the Specification index, which
+# fans out to all of them.
+/developer-guide/internals/serialization      /specification/  301
+/developer-guide/internals/serialization.html /specification/  301
+
+# The specs/ section was renamed to specification/. Cover the section root explicitly
+# (the splat does not match the bare /specs path) before the child-path splat rule.
+/specs            /specification/  301
+/specs/           /specification/  301
+/specs/index.html /specification/  301
+/specs/*          /specification/:splat  301

docs/api/c/index.rst

@@ -1,9 +1,10 @@
 C API
 =====
 
-The Vortex C API provides a low-level FFI interface to the Vortex library. It is the foundation for
-other language bindings (including C++) and is suitable for embedding Vortex into C applications or
-building higher-level wrappers.
+The Vortex C API provides a low-level FFI interface to the Vortex library. It is the intended
+foundation for other language bindings and is suitable for embedding Vortex into C applications or
+building higher-level wrappers. (The C++ binding currently uses a direct ``cxx`` Rust bridge; a future
+migration to build it on top of this C FFI is planned.)
 
 .. warning::
     This API should be considered entirely unstable. It *will* change. Please reach out if a stable

docs/api/cpp/index.rst

@@ -1,9 +1,10 @@
 C++ API
 =======
 
-The Vortex C++ API provides an idiomatic C++ wrapper around the Vortex C FFI, built using
-`cxx <https://cxx.rs/>`_. It currently supports reading and writing Vortex files and integrates
-with the Arrow C Data Interface via `nanoarrow <https://arrow.apache.org/nanoarrow/>`_.
+The Vortex C++ API is an idiomatic C++ binding built with `cxx <https://cxx.rs/>`_, which generates
+a direct bridge between C++ and Rust. It supports reading and writing Vortex files and integrates
+with the Arrow C Data Interface via `nanoarrow <https://arrow.apache.org/nanoarrow/>`_. A future
+migration to build the C++ API on top of the C FFI is planned.
 
 In the future we will expand the C++ API to cover Vortex's plugin and extension points. Please
 reach out if you are interested in extending Vortex from C++ so we can prioritize these features.

docs/api/java/index.rst

@@ -39,21 +39,17 @@ They support any Linux distribution with a GLIBC version >= 2.31. This includes
 * Ubuntu 20.04 or newer
 
 
-Usage Example
--------------
-
-Here's a basic example of using the Vortex Java API to read a Vortex file:
-
-.. code-block:: java
-
-    import dev.vortex.api.File;
-    import dev.vortex.api.Array;
+Core API
+--------
 
-    // Open a Vortex file
-    File vortexFile = File.open("path/to/file.vortex");
+The main entry points of the Vortex Java API are:
 
-    // Read arrays from the file
-    Array array = vortexFile.readArray();
+* ``Session`` — the top-level handle that owns data sources and writers.
+* ``DataSource`` — opens a Vortex file, glob, or table via ``DataSource.open(session, uri)`` and
+  exposes its schema and row count.
+* ``Scan`` / ``Partition`` — a scan over a ``DataSource`` iterates its partitions, yielding
+  Arrow-compatible batches.
+* ``VortexWriter`` — writes arrays to a Vortex file.
 
-    // Work with the array data
-    System.out.println("Array length: " + array.getLength());
+See the `Vortex JNI API <../../_static/vortex-jni/index.html>`_ above for the full,
+method-level reference.

docs/api/python/arrays.rst

@@ -22,8 +22,9 @@ Base Class
 Canonical Encodings
 -------------------
 
-Each :class:`~vortex.DType` has a corresponding canonical encoding. These encodings represent the uncompressed version
-of the array, and are also zero-copy to Apache Arrow.
+Each :class:`~vortex.DType` has a corresponding canonical encoding, with the sole exception of ``Union`` (not yet
+canonicalized). These encodings represent the uncompressed version of the array, and are also zero-copy to
+Apache Arrow. The classes below are the canonical encodings currently exposed in Python.
 
 .. autoclass:: vortex.NullArray
      :members:
@@ -37,19 +38,15 @@ of the array, and are also zero-copy to Apache Arrow.
      :members:
 
 
-.. autoclass:: vortex.VarBinArray
-    :members:
-
-
 .. autoclass:: vortex.VarBinViewArray
     :members:
 
 
-.. autoclass:: vortex.StructArray
+.. autoclass:: vortex.FixedSizeListArray
     :members:
 
 
-.. autoclass:: vortex.ListArray
+.. autoclass:: vortex.StructArray
     :members:
 
 
@@ -72,6 +69,14 @@ Utility Encodings
      :members:
 
 
+.. autoclass:: vortex.VarBinArray
+    :members:
+
+
+.. autoclass:: vortex.ListArray
+    :members:
+
+
 Compressed Encodings
 --------------------
 

docs/api/python/index.rst

@@ -27,6 +27,7 @@ The Python bindings require Python 3.11 or newer. Pre-built wheels are available
 * x86_64 Linux
 * ARM64 Linux
 * Apple Silicon macOS
+* Intel macOS
 
 They support any Linux distribution with a GLIBC version >= 2.17. This includes
 
@@ -41,14 +42,17 @@ Here's a basic example of using the Vortex Python API to write and read a Vortex
 
 .. code-block:: python
 
+    import pyarrow as pa
     import vortex
 
     # Write a Vortex file from a PyArrow table
-    vortex.io.write_path(my_table, "data.vortex")
+    my_table = pa.table({"id": [1, 2, 3], "name": ["a", "b", "c"]})
+    vortex.io.write(my_table, "data.vortex")
 
     # Read a Vortex file
-    dataset = vortex.dataset("data.vortex")
-    table = dataset.to_arrow()
+    vf = vortex.open("data.vortex")
+    reader = vf.to_arrow()  # a pyarrow.RecordBatchReader
+    table = reader.read_all()
 
 
 API Reference

docs/concepts/arrays.md

@@ -41,19 +41,22 @@ define their own.
 
 ### Canonical Arrays
 
-In order to avoid having to implement logic for an exponential combination of encodings, Vortex defines one canonical
-encoding per logical data type. All arrays can eventually be decompressed one of these canonical encodings.
+In order to avoid having to implement logic for an exponential combination of encodings, Vortex defines a canonical
+encoding for each logical data type — every type except `Union`, which is not yet canonicalized. Arrays of those types
+can eventually be decompressed to one of the canonical encodings listed below.
 
 | Data Type              | Canonical Encoding   |
 |------------------------|----------------------|
 | `DType::Null`          | `NullArray`          |
 | `DType::Bool`          | `BoolArray`          |
 | `DType::Primitive`     | `PrimitiveArray`     |
-| `DType::UTF8`          | `VarBinViewArray`    |
+| `DType::Decimal`       | `DecimalArray`       |
+| `DType::Utf8`          | `VarBinViewArray`    |
 | `DType::Binary`        | `VarBinViewArray`    |
 | `DType::Struct`        | `StructArray`        |
 | `DType::List`          | `ListViewArray`      |
 | `DType::FixedSizeList` | `FixedSizeListArray` |
+| `DType::Variant`       | `VariantArray`       |
 | `DType::Extension`     | `ExtensionArray`     | 
 
 ### Builtin Arrays
@@ -100,15 +103,15 @@ These can be found in the `encodings/` directory of the Vortex repository.
 Arrays carry their own statistics with them, allowing many compute functions to short-circuit or optimize their
 implementations. Currently, the available statistics are:
 
+* `is_constant`: Whether the array holds only a single unique value (treating nulls as equal).
+* `is_sorted`: Whether the array's values are in ascending order.
+* `is_strict_sorted`: Whether the array's values are in strictly ascending order, with no duplicates.
+* `max`: The maximum value in the array (ignoring nulls).
+* `min`: The minimum value in the array (ignoring nulls).
+* `sum`: The sum of the non-null values in the array.
 * `null_count`: The number of null values in the array.
-* `true_count`: The number of `true` values in a boolean array.
-* `run_count`: The number of consecutive runs in an array.
-* `is_constant`: Whether the array only holds a single unique value
-* `is_sorted`: Whether the array values are sorted.
-* `is_strict_sorted`: Whether the array values are sorted and unique.
-* `min`: The minimum value in the array.
-* `max`: The maximum value in the array.
-* `uncompressed_size`: The size of the array in memory before any compression.
+* `uncompressed_size_in_bytes`: The size of the array in memory before any compression.
+* `nan_count`: The number of NaN values in the array.
 
 ## Execution