Skip to content

Commit 31342b0

Browse files
tejasbadadareclaude
tejasbadadare
and
claude
authored
tejas/pfg 1554 tob tracer bullet okx recorder crate bbo tbt okx book ticker (#3975)
* feat(okx-recorder): ToB tracer bullet — bbo-tbt -> okx_book_ticker with local Tilt stack (PFG-1554) New standalone apps/okx-recorder crate modeled on binance-recorder: one websocket connection to the OKX public endpoint multiplexing a bbo-tbt subscription per config-driven instrument (seeded with OPENAI-USDT-SWAP, ANTHROPIC-USDT-SWAP), client-side received_at stamped next to the exchange ts, and batched inserts into okx_book_ticker (ReplacingMergeTree(ingested_at), monthly partitions, 90-day TTL). No in-memory dedupe: ClickHouse owns row identity via the ORDER BY keys. /ready gates on ClickHouse reachability plus per-instrument ToB freshness; metrics live under okx_recorder_*. LaneRow enum and the subscribe/dispatch seams are ready for the trades and funding lanes. Local verification loop included: Tiltfile + docker-compose stack (ClickHouse with auto-loaded migration, Prometheus, Grafana) and scripts/local_e2e_check.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(okx-recorder): address PR review feedback on metrics init, dedupe docs, placeholder levels - Pre-initialize okx_recorder_tob_last_seen_unix_seconds{inst_id} for every configured instrument at runtime start, seeded with the process start time, so staleness alerts can fire for an instrument that never streams. - Correct the overstated dedupe claim in the README and recorder comments: ReplacingMergeTree with received_at in the ORDER BY collapses byte-identical insert retries only; a genuine exchange re-send arrives with a new received_at and is persisted as a distinct row by design. - Treat a placeholder book level with an empty-string price (e.g. ["","","",""]) as an absent side instead of a parse error, so one-sided updates on channels that emit placeholder levels are not dropped. Addresses PR review feedback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent a1da9bc commit 31342b0

33 files changed

Lines changed: 5826 additions & 0 deletions

‎.github/greenlight.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ auto_approve_paths:
2222
# research; not on the price-delivery wire)
2323
- "apps/binance-recorder/**"
2424
- "apps/hyperliquid-recorder/**"
25+
- "apps/okx-recorder/**"
2526
- "apps/ondo-recorder/**"
2627
# Presentational UI packages (not imported by any critical-path tool)
2728
- "packages/component-library/**"

‎.github/workflows/docker-okx-recorder.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Build and Push OKX Recorder Image
2+
on:
3+
push:
4+
tags:
5+
- okx-recorder-v*
6+
pull_request:
7+
paths:
8+
- "apps/okx-recorder/**"
9+
- ".github/workflows/docker-okx-recorder.yml"
10+
workflow_dispatch:
11+
inputs:
12+
dispatch_description:
13+
description: "Dispatch description"
14+
required: true
15+
type: string
16+
permissions:
17+
contents: read
18+
id-token: write
19+
packages: write
20+
env:
21+
REGISTRY: ghcr.io
22+
IMAGE_NAME: pyth-network/okx-recorder
23+
jobs:
24+
okx-recorder-image:
25+
runs-on: ubuntu-latest
26+
steps:
27+
- uses: actions/checkout@v4
28+
- name: Set image tag to version of the git tag
29+
if: ${{ startsWith(github.ref, 'refs/tags/okx-recorder-v') }}
30+
run: |
31+
PREFIX="refs/tags/okx-recorder-"
32+
VERSION="${GITHUB_REF:${#PREFIX}}"
33+
echo "IMAGE_TAG=${VERSION}" >> "${GITHUB_ENV}"
34+
- name: Set image tag to the git commit hash
35+
if: ${{ !startsWith(github.ref, 'refs/tags/okx-recorder-v') }}
36+
run: |
37+
SHORT_SHA=$(echo "${GITHUB_SHA}" | cut -c1-7)
38+
echo "IMAGE_TAG=sha-${SHORT_SHA}" >> "${GITHUB_ENV}"
39+
- name: Log in to the Container registry
40+
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
41+
with:
42+
registry: ${{ env.REGISTRY }}
43+
username: ${{ github.actor }}
44+
password: ${{ secrets.GITHUB_TOKEN }}
45+
- name: Extract metadata (tags, labels) for Docker
46+
id: metadata_okx_recorder
47+
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
48+
with:
49+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
50+
tags: |
51+
type=raw,value=${{ env.IMAGE_TAG }}
52+
- name: Build and push docker image
53+
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
54+
with:
55+
context: ./apps/okx-recorder
56+
file: "./apps/okx-recorder/Dockerfile"
57+
push: ${{ github.event_name != 'pull_request' }}
58+
tags: ${{ steps.metadata_okx_recorder.outputs.tags }}
59+
labels: ${{ steps.metadata_okx_recorder.outputs.labels }}

‎apps/okx-recorder/.dockerignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
target/
2+
.vendor/
3+
config.yml
4+
.env

‎apps/okx-recorder/.env.sample‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# okx-recorder consumes OKX's public websocket channels, so no API key is
2+
# required. This file is only for optional runtime overrides.
3+
4+
# Log verbosity (read by the tracing EnvFilter). The tungstenite crates are
5+
# pinned to warn to silence per-frame chatter; raise them (e.g.
6+
# tokio_tungstenite=debug) to debug the websocket client.
7+
RUST_LOG=info,tungstenite=warn,tokio_tungstenite=warn
8+
9+
# Any AppConfig field can be overridden here with the OKX_RECORDER__ prefix
10+
# and a `__` separator, e.g.:
11+
# OKX_RECORDER__CLICKHOUSE__URL=http://clickhouse-local:8123
12+
# OKX_RECORDER__INSTRUMENTS=OPENAI-USDT-SWAP,ANTHROPIC-USDT-SWAP

‎apps/okx-recorder/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
config.yml
2+
.env
3+
target/

0 commit comments

Comments
 (0)