Skip to content

Commit e1dd0bf

Browse files
acmeguyclaude
andauthored
feat(011): Model Management API — validate, refresh, delete, single-cube, diff, rollback (#45)
* fix(cubejs): match checkSqlAuth callback signature to Cube.js v1.6 Cube.js v1.6 invokes checkSqlAuth as (request, user, password) — three positional args — see @cubejs-backend/api-gateway/dist/src/sql-server.js:291,105. Our implementation declared (_, user) and did: password = typeof user === "string" ? user : user?.password username = typeof user === "string" ? _ : user?.username With the v1.6 wire server, user arrives as a plain string (the Postgres username), so the code took the username as the password AND used the request metadata object as the username. findSqlCredentials then received the {protocol, method, apiType} object, and Hasura rejected the query with: parsing Text failed, expected String, but encountered Object path: $.selectionSet.sql_credentials.args.where.username._eq Every SQL API login failed before any password comparison ran (reproduced via `psql -U <valid> -h <cubejs>` → 28P01). Fix: match the documented v1.6 signature and keep a defensive branch for the legacy object-shape call. Also reject non-string username early so the Hasura GraphQL layer cannot receive a non-string variable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cubejs): propagate teamProperties/memberProperties for SQL API logins queryRewrite's rule-based row filtering relies on `securityContext.userScope.teamProperties` and `.memberProperties` to look up per-rule property values (e.g. `partition` from team settings). defineUserScope populates both from the member's team settings and member properties. buildSqlSecurityContext (the SQL API path) never did, so userScope for SQL logins had no team/member properties. Every rule whose property lookup returned undefined blocked the whole query and queryRewrite replaced query.filters with: [{ member: allMembers[0], operator: "equals", values: ["__blocked_by_access_control__"] }] When the first member was a numeric measure (e.g. `count`), ClickHouse tried to cast the sentinel to Float64: Cannot parse string '__blocked_by_access_control__' as Float64 Fix: buildSqlSecurityContext now resolves the member for the datasource's team and passes the team settings + member properties into the scope (matching defineUserScope). Team settings also flow into buildSecurityContext so the content hash includes them, keeping cache isolation consistent between REST and SQL paths. Reproduced via `SELECT count(*) FROM stockout_event` over the Postgres wire — rule `semantic_events.partition` couldn't resolve team.partition, blocked fired, sentinel filter crashed ClickHouse. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cubejs): include team.settings and member.properties in membersFragment Follow-up to #42. buildSqlSecurityContext now resolves teamProperties and memberProperties from sqlCredentials.user.members, but the GraphQL query used to load sql_credentials (sqlCredentialsQuery → membersFragment) never selected those fields. At runtime teamMember was found but team and properties were undefined, so teamProperties stayed empty and queryRewrite rules that look up teamProperties.<key> still blocked every query. Observed via: SELECT count(*) FROM stockout_event → still rewritten to: HAVING count(*) = toFloat64('__blocked_by_access_control__') Add team { id settings } and properties to membersFragment so the SQL API path has the same shape defineUserScope consumes on the REST path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(cubejs): bump @cubejs-backend/* from 1.6.19 to 1.6.37 Brings in 16 patch releases of upstream Cube.js fixes and features, including cubesql improvements (Tableau format codes, Talend compat, MEASURE function panic fix, LAG/LEAD pushdown, TO_TIMESTAMP formats, SET TIMEZONE, FETCH directions, CASE/LIKE planning, pg_catalog.pg_collation, SAVEPOINT/ROLLBACK TO/RELEASE, SET ROLE auth context, and more). Does not close the DataGrip introspection gap (regclass in functions, pg_get_userbyid coercion, OPERATOR(schema.~), CHAR[] arrays, SHOW server_version, pg_description.objoid mapping, empty pg_index/ pg_constraint — none addressed upstream between 1.6.21 and 1.6.37), but keeps us current before we build or wait on a JetBrains-friendly fix. yarn.lock will regenerate on CI build (Dockerfile runs `yarn --network-timeout 100000`). No breaking changes relevant to us (server-core access-policy row filtering breaking change doesn't affect our usage — we don't use cube's access_policy feature; row filtering is via queryRewrite.js). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(cubejs): bump CUBESTORE_VERSION in .env.example to v1.6.37 Keeps local docker-compose in sync with the Cube.js backend bump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(011): Model Management API — validate, refresh, delete, single-cube, diff, rollback Adds six authenticated REST endpoints that let an agent own the full author-to-publish lifecycle of a cube model without operator assistance: POST /api/v1/validate-in-branch (US1) POST /api/v1/internal/refresh-compiler (US2) DELETE /api/v1/dataschema/:dataschemaId (US3) GET /api/v1/meta/cube/:cubeName (US4) POST /api/v1/version/diff (US5) POST /api/v1/version/rollback (US5) Delete and rollback emit durable audit rows via Hasura event triggers (`audit_dataschema_delete`, `audit_version_rollback`) into a new `audit_logs` table with 90-day retention via a daily cron trigger. Refresh is cache-only and emits a non-durable structured log line only. The Hasura migration (`1713600000000_dataschemas_delete_permission`) adds `versions.origin` + `versions.is_current` with a statement-level trigger that uses a NEW TABLE transition table so multi-row inserts cannot break the invariant. A transaction-scoped advisory lock keyed on affected branches serialises concurrent inserts on the same branch while different branches still proceed in parallel. Backfill runs in 1 000-row batches. `/meta-all` now enriches every cube summary with `dataschema_id` + `file_name` by parsing each schema once per call (cube-name keyed, since Cube.js v1.6 `metaConfig` omits `fileName`). All five mutating handlers write a durable audit row on every failure path (partition mismatch, insufficient role, historical version, blocking references, Hasura rejection). Success is captured by the event triggers. New utilities: `compilerCacheInvalidator`, `referenceScanner` (FR-008 seven kinds), `directVerifyAuth`, `requireOwnerOrAdmin`, `mapHasuraErrorCode`, `auditWriter`, `metaForBranch`, `versionDiff`, `errorCodes` (FR-017 single-source-of-truth enum). `graphql.js` gains a `preserveErrors` option so handlers can surface Hasura extension codes as stable FR-017 codes. Spec + runbook: `specs/011-model-mgmt-api/` (spec.md, plan.md, tasks.md, research.md, data-model.md, contracts/, quickstart.md, DEPLOYMENT.md, migration README). `scripts/lint-error-codes.mjs` fails the build if the error-code enum drifts across `errorCodes.js` and any of the six contracts. Tests: 49 Vitest-style `node:test` unit tests for the new utilities + `summarizeCube` + versionDiff adapter + SC-003 fixture corpus; 5 integration tests for the Actions RPC handlers; 8 StepCI workflows under `tests/workflows/model-management/` including an end-to-end flow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 9bfc8bd commit e1dd0bf

72 files changed

Lines changed: 7570 additions & 29 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -96,15 +96,32 @@ Docker Compose files per environment: `docker-compose.dev.yml`, `docker-compose.
9696

9797
## Key File Locations
9898

99-
- `services/actions/src/rpc/`22 RPC handlers (one file per action)
99+
- `services/actions/src/rpc/`25 RPC handlers (one file per action), includes `auditDataschemaDelete`, `auditVersionRollback`, `auditLogsRetention`
100100
- `services/cubejs/src/utils/driverFactory.js` — Database driver creation (25+ drivers)
101101
- `services/cubejs/src/utils/checkAuth.js` — JWT verification and security context setup
102102
- `services/cubejs/src/utils/defineUserScope.js` — Branch/version/access resolution
103103
- `services/cubejs/src/utils/buildSecurityContext.js` — Content-hashed context for cache isolation
104-
- `services/cubejs/src/routes/` — 7 REST API endpoints (run-sql, test, get-schema, generate-models, etc.)
104+
- `services/cubejs/src/utils/compilerCacheInvalidator.js` — Branch-scoped compiler-cache eviction (011-model-mgmt-api)
105+
- `services/cubejs/src/utils/referenceScanner.js` — Seven-kind cross-cube reference detector (FR-008)
106+
- `services/cubejs/src/utils/directVerifyAuth.js` — Shared direct-verify helper for branch-scoped routes
107+
- `services/cubejs/src/utils/metaForBranch.js` — Helper that returns raw visibility-filtered metaConfig
108+
- `services/cubejs/src/utils/auditWriter.js` — Best-effort audit-log writer with retry
109+
- `services/cubejs/src/utils/errorCodes.js` — Canonical Model Management API error codes (FR-017)
110+
- `services/cubejs/src/utils/requireOwnerOrAdmin.js` — Owner/admin team-role check
111+
- `services/cubejs/src/utils/mapHasuraErrorCode.js` — Hasura extensions.code → stable error code
112+
- `services/cubejs/src/utils/versionDiff.js` — Per-cube diff adapter over smart-generation/diffModels
113+
- `services/cubejs/src/routes/` — 13 REST API endpoints, now including:
114+
- `validateInBranch.js` (POST /api/v1/validate-in-branch, US1)
115+
- `refreshCompiler.js` (POST /api/v1/internal/refresh-compiler, US2)
116+
- `deleteDataschema.js` (DELETE /api/v1/dataschema/:id, US3)
117+
- `metaSingleCube.js` (GET /api/v1/meta/cube/:cubeName, US4)
118+
- `versionDiff.js` + `versionRollback.js` (POST /api/v1/version/{diff,rollback}, US5)
105119
- `services/hasura/metadata/actions.yaml` — GraphQL action definitions (maps to Actions RPC)
106120
- `services/hasura/metadata/tables.yaml` — Table definitions, relationships, and permissions
107-
- `services/hasura/migrations/` — 96+ SQL migration directories
121+
- `services/hasura/metadata/cron_triggers.yaml` — Cron triggers (now includes `audit_logs_retention_90d`)
122+
- `services/hasura/migrations/` — 97+ SQL migration directories
123+
- `scripts/lint-error-codes.mjs` — Fails CI when FR-017 error-code enum drifts across contracts
124+
- `tests/workflows/model-management/` — StepCI workflows + SC-003 fixtures for all six endpoints
108125

109126
## Release Process
110127

@@ -273,6 +290,8 @@ These are the target patterns for adapting Synmetrix and client-v2:
273290
- N/A — no database schema changes. Query results are transient. (009-query-output)
274291
- JavaScript (ES modules), Node.js 22+ + Cube.js v1.6.x (CubeJS service), Express 4.18.2, `openai` npm v6.x (NEW), React 18 + Vite + Ant Design 5 (client-v2), URQL (GraphQL client) (010-dynamic-models-ii)
275292
- PostgreSQL via Hasura (versions, dataschemas), ClickHouse (profiling target — read-only) (010-dynamic-models-ii)
293+
- JavaScript (ES modules), Node.js 22.x (already current in cubejs service after 003-update-deps) + `@cubejs-backend/schema-compiler` ^1.6.19 (existing; `prepareCompiler` powers validation), `@cubejs-backend/server-core` ^1.6.19 (existing; exposes `cubejs.compilerCache` LRU-cache), `@cubejs-backend/api-gateway` ^1.6.19 (existing; `getCompilerApi` + `filterVisibleItemsInMeta`), `jose` (existing; FraiOS/WorkOS JWT verification), Express 4.x (existing router). No new dependencies. (011-model-mgmt-api)
294+
- PostgreSQL via Hasura (existing `dataschemas`, `versions`, `branches` tables — one new Hasura delete-permission migration on `dataschemas`). In-memory LRU compiler cache inside the cubejs process (existing). No new tables. (011-model-mgmt-api)
276295

277296
## Recent Changes
278297
- 001-dev-environment: Added TypeScript (ES2022, Node16 modules) — matches + oclif (CLI framework), zx (shell execution)

scripts/lint-error-codes.mjs

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
#!/usr/bin/env node
2+
3+
/**
4+
* lint-error-codes — FR-017 guard.
5+
*
6+
* Fails with a non-zero exit code (and a human-readable diff) whenever the
7+
* `ErrorCode` enum drifts across:
8+
* - services/cubejs/src/utils/errorCodes.js (single source of truth)
9+
* - specs/011-model-mgmt-api/contracts/*.yaml (every OpenAPI contract)
10+
*
11+
* Every contract under `contracts/` must contain a top-level `ErrorCode`
12+
* schema with an explicit `enum` array. Each enum must match the exhaustive
13+
* list of values in `errorCodes.js`, so clients generating bindings from any
14+
* single contract receive the complete enum.
15+
*/
16+
17+
import { readFile, readdir } from "node:fs/promises";
18+
import { fileURLToPath } from "node:url";
19+
import { dirname, join } from "node:path";
20+
21+
const __dirname = dirname(fileURLToPath(import.meta.url));
22+
const repoRoot = join(__dirname, "..");
23+
24+
const ERROR_CODES_JS = join(
25+
repoRoot,
26+
"services/cubejs/src/utils/errorCodes.js"
27+
);
28+
const CONTRACTS_DIR = join(
29+
repoRoot,
30+
"specs/011-model-mgmt-api/contracts"
31+
);
32+
33+
async function readErrorCodesJs() {
34+
const text = await readFile(ERROR_CODES_JS, "utf8");
35+
const re = /["']([a-z][a-z0-9_]+)["']/g;
36+
const known = new Set();
37+
let m;
38+
while ((m = re.exec(text))) {
39+
const v = m[1];
40+
if (
41+
/_(?:invalid|not_found|unresolved|not_visible|unauthorized|by_references|historical_version|authorization|cross_branch|not_on_branch|columns_missing)_?/.test(
42+
v
43+
) ||
44+
/_reference$/.test(v) ||
45+
v === "cube_not_found" ||
46+
v === "validate_invalid_mode" ||
47+
v === "validate_target_not_found" ||
48+
v === "validate_unresolved_reference" ||
49+
v === "refresh_branch_not_visible" ||
50+
v === "refresh_unauthorized" ||
51+
v === "delete_blocked_by_references" ||
52+
v === "delete_blocked_historical_version" ||
53+
v === "delete_blocked_authorization" ||
54+
v === "diff_cross_branch" ||
55+
v === "diff_invalid_request" ||
56+
v === "rollback_version_not_on_branch" ||
57+
v === "rollback_blocked_authorization" ||
58+
v === "rollback_invalid_request" ||
59+
v === "rollback_source_columns_missing"
60+
) {
61+
known.add(v);
62+
}
63+
}
64+
return known;
65+
}
66+
67+
async function readContractEnum(path) {
68+
const text = await readFile(path, "utf8");
69+
// Locate the ErrorCode schema's enum list. Permissive regex: we want every
70+
// value inside the first `enum:` block that follows `ErrorCode:`.
71+
const anchor = text.indexOf("ErrorCode:");
72+
if (anchor === -1) {
73+
throw new Error(`${path} has no ErrorCode schema`);
74+
}
75+
const afterAnchor = text.slice(anchor);
76+
// Find the enum block: `enum:` → dashed list → end when we hit a line that
77+
// starts with non-dash non-whitespace content at the same or lower indent.
78+
const enumAnchor = afterAnchor.search(/\n\s*enum:\s*\n/);
79+
if (enumAnchor === -1) {
80+
throw new Error(`${path} ErrorCode.enum block not found`);
81+
}
82+
const afterEnum = afterAnchor.slice(enumAnchor).split("\n");
83+
// Skip the `enum:` line itself.
84+
const values = [];
85+
let inEnum = false;
86+
for (const raw of afterEnum) {
87+
if (!inEnum) {
88+
if (/^\s*enum:\s*$/.test(raw)) inEnum = true;
89+
continue;
90+
}
91+
const trimmed = raw.trim();
92+
if (trimmed === "") continue;
93+
if (trimmed.startsWith("- ")) {
94+
const v = trimmed.slice(2).replace(/['"]/g, "").trim();
95+
if (v) values.push(v);
96+
continue;
97+
}
98+
// First non-dash non-empty line after the list ends the enum block.
99+
break;
100+
}
101+
return new Set(values);
102+
}
103+
104+
function setDiff(a, b) {
105+
const out = [];
106+
for (const v of a) if (!b.has(v)) out.push(v);
107+
return out;
108+
}
109+
110+
async function main() {
111+
const sourceOfTruth = await readErrorCodesJs();
112+
if (sourceOfTruth.size === 0) {
113+
console.error("lint-error-codes: errorCodes.js produced an empty set");
114+
process.exit(1);
115+
}
116+
117+
const entries = (await readdir(CONTRACTS_DIR)).filter((f) =>
118+
f.endsWith(".yaml")
119+
);
120+
if (entries.length === 0) {
121+
console.error(`lint-error-codes: no contracts found in ${CONTRACTS_DIR}`);
122+
process.exit(1);
123+
}
124+
125+
let failed = false;
126+
for (const entry of entries) {
127+
const path = join(CONTRACTS_DIR, entry);
128+
let contractSet;
129+
try {
130+
contractSet = await readContractEnum(path);
131+
} catch (err) {
132+
console.error(`lint-error-codes: ${entry}${err.message}`);
133+
failed = true;
134+
continue;
135+
}
136+
const missing = setDiff(sourceOfTruth, contractSet);
137+
const extra = setDiff(contractSet, sourceOfTruth);
138+
if (missing.length || extra.length) {
139+
failed = true;
140+
console.error(`lint-error-codes: ${entry} drift detected`);
141+
if (missing.length) {
142+
console.error(` MISSING (in errorCodes.js, not in ${entry}):`);
143+
for (const v of missing) console.error(` - ${v}`);
144+
}
145+
if (extra.length) {
146+
console.error(` EXTRA (in ${entry}, not in errorCodes.js):`);
147+
for (const v of extra) console.error(` - ${v}`);
148+
}
149+
}
150+
}
151+
152+
if (failed) process.exit(1);
153+
console.log(
154+
`lint-error-codes: OK (${sourceOfTruth.size} codes, ${entries.length} contracts)`
155+
);
156+
}
157+
158+
main().catch((err) => {
159+
console.error(err);
160+
process.exit(1);
161+
});
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
import { describe, it, beforeEach, afterEach } from "node:test";
2+
import assert from "node:assert/strict";
3+
4+
let handler;
5+
6+
describe("auditDataschemaDelete RPC", () => {
7+
let originalError;
8+
9+
beforeEach(async () => {
10+
originalError = console.error;
11+
console.error = () => {};
12+
({ default: handler } = await import("../auditDataschemaDelete.js"));
13+
});
14+
15+
afterEach(() => {
16+
console.error = originalError;
17+
});
18+
19+
it("rejects payloads missing event.data.old.id", async () => {
20+
const res = await handler({}, { event: { data: { old: null } } });
21+
assert.equal(res.ok, false);
22+
assert.match(res.error, /no event.data.old payload/);
23+
});
24+
25+
it("falls through to fetchGraphQL and returns a structured error when HASURA is unreachable", async () => {
26+
const prev = process.env.HASURA_ENDPOINT;
27+
process.env.HASURA_ENDPOINT =
28+
"http://127.0.0.1:1/unreachable-audit-test";
29+
try {
30+
const res = await handler(
31+
{ "x-hasura-user-id": "user-1" },
32+
{
33+
event: {
34+
data: {
35+
old: {
36+
id: "00000000-0000-4000-8000-000000000001",
37+
datasource_id: "00000000-0000-4000-8000-000000000010",
38+
version_id: "00000000-0000-4000-8000-000000000020",
39+
user_id: "user-1",
40+
},
41+
},
42+
session_variables: { "x-hasura-user-id": "user-1" },
43+
},
44+
}
45+
);
46+
assert.equal(res.ok, false);
47+
assert.ok(typeof res.error === "string" && res.error.length > 0);
48+
} finally {
49+
if (prev == null) delete process.env.HASURA_ENDPOINT;
50+
else process.env.HASURA_ENDPOINT = prev;
51+
}
52+
});
53+
});
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
import { describe, it, beforeEach, afterEach } from "node:test";
2+
import assert from "node:assert/strict";
3+
4+
let handler;
5+
6+
describe("auditLogsRetention RPC", () => {
7+
let originalError;
8+
9+
beforeEach(async () => {
10+
originalError = console.error;
11+
console.error = () => {};
12+
({ default: handler } = await import("../auditLogsRetention.js"));
13+
});
14+
15+
afterEach(() => {
16+
console.error = originalError;
17+
});
18+
19+
it("returns a structured error when Hasura is unreachable", async () => {
20+
const prev = process.env.HASURA_ENDPOINT;
21+
process.env.HASURA_ENDPOINT =
22+
"http://127.0.0.1:1/unreachable-retention-test";
23+
try {
24+
const res = await handler();
25+
assert.ok(res);
26+
assert.ok(
27+
typeof res.error === "string" && res.error.length > 0,
28+
"expected structured error on unreachable Hasura"
29+
);
30+
} finally {
31+
if (prev == null) delete process.env.HASURA_ENDPOINT;
32+
else process.env.HASURA_ENDPOINT = prev;
33+
}
34+
});
35+
});
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
import { describe, it, beforeEach, afterEach } from "node:test";
2+
import assert from "node:assert/strict";
3+
4+
let handler;
5+
6+
describe("auditVersionRollback RPC", () => {
7+
let originalError;
8+
9+
beforeEach(async () => {
10+
originalError = console.error;
11+
console.error = () => {};
12+
({ default: handler } = await import("../auditVersionRollback.js"));
13+
});
14+
15+
afterEach(() => {
16+
console.error = originalError;
17+
});
18+
19+
it("skips non-rollback inserts", async () => {
20+
const res = await handler(
21+
{},
22+
{
23+
event: {
24+
data: {
25+
new: { id: "v-1", origin: "user" },
26+
},
27+
},
28+
}
29+
);
30+
assert.deepEqual(res, { ok: true, skipped: true });
31+
});
32+
33+
it("processes rollback inserts and reports Hasura errors gracefully", async () => {
34+
const prev = process.env.HASURA_ENDPOINT;
35+
process.env.HASURA_ENDPOINT =
36+
"http://127.0.0.1:1/unreachable-rollback-audit";
37+
try {
38+
const res = await handler(
39+
{},
40+
{
41+
event: {
42+
data: {
43+
new: {
44+
id: "00000000-0000-4000-8000-000000000099",
45+
origin: "rollback",
46+
branch_id: "00000000-0000-4000-8000-000000000050",
47+
user_id: "00000000-0000-4000-8000-000000000005",
48+
checksum: "abc",
49+
},
50+
},
51+
session_variables: {
52+
"x-hasura-user-id": "00000000-0000-4000-8000-000000000005",
53+
},
54+
},
55+
}
56+
);
57+
assert.equal(res.ok, false);
58+
assert.ok(typeof res.error === "string");
59+
} finally {
60+
if (prev == null) delete process.env.HASURA_ENDPOINT;
61+
else process.env.HASURA_ENDPOINT = prev;
62+
}
63+
});
64+
});

0 commit comments

Comments
 (0)