zig-arrow

Extremely WIP Apache Arrow library for Zig. Essentially nothing works. Currently focused on IPC files; Tensor and SparseTensor support is a priority.

First, initialize the flatcc submodule

git submodule update --init

Then run the test parsing an example .arrow file

$ zig build test
test
└─ run test stderr
[default] (warn): footer: Footer[V5]
[default] (warn): schema fields (3)
[default] (warn): - id	u32
[default] (warn): - incoming_degree	u32
[default] (warn): - outgoing_degree	u32
[default] (warn): record batchs (1):
[default] (warn):   - RecordBatch[BUFFER, LZ4_FRAME]
[default] (warn):   - nodes (3)
[default] (warn):     - len 997, 0 nulls
[default] (warn):     - len 997, 0 nulls
[default] (warn):     - len 997, 0 nulls
[default] (warn):   - buffers (6)
[default] (warn):     - .{ .offset = 0, .length = 0 }
[default] (warn):     - .{ .offset = 0, .length = 3988 }
[default] (warn):     - .{ .offset = 3992, .length = 0 }
[default] (warn):     - .{ .offset = 3992, .length = 3988 }
[default] (warn):     - .{ .offset = 7984, .length = 0 }
[default] (warn):     - .{ .offset = 7984, .length = 3988 }
[default] (warn):   - variadic_buffer_counts (0)
[default] (warn): footer dictionaries (0):

Due to this bug in Zig's translate-c module we are not able to use the flatcc-generated reader/builder directly. As a temporary workaround, a manually patched flatbuf/flatbuf.zig is currently checked into the repo.

The pre-patch flatbuf.zig file can be generated with this script:

mkdir -p flatbuf/include
flatcc -wvc -o flatbuf/include flatbuf/*.fbs

clang -E -P -dD -Iflatcc/include -Iflatbuf/include flatbuf/flatbuf.h \
  | grep -v '^#define *__' \
  > flatbuf/flatbuf.c

zig translate-c -IIflatbuf/include -Iflatcc/include flatbuf/flatbuf.c > flatbuf/flatbuf.zig

... and can be patched by adding @alignCast(...) until the project compiles successfully.