feat: major refactoring and effect improvements and fixes (#21)

- Refactor core modules across the codebase (flight, wings, ack, vcr)
- Add record batch with metadata support (flight)
- Improve testing and standards (wings-testing)
- Update connectors and templates
- ack skill
- Miscellaneous file changes

Author: fracek

.agents/skills/airfoil-kit/README.md

@@ -8,7 +8,8 @@ Airfoil producer connector end-to-end.
 - Confirms no existing implementation is being copied.
 - Copies `templates/producer-template/` into `connectors/producer-<name>/`.
 - Helps you research the target API and derive schemas from recorded traffic.
-- Wires Effect v4 `Config`, API clients, `WebhookRoute`, and streams.
+- Wires current Effect v4 `Config`, API client layers, `Webhook.route(...)`,
+  connector layers, and streams.
 - Guides deterministic replay testing (VCR for REST/GraphQL, fixtures/mocks for gRPC).
 - Enforces a Definition of Done before declaring the task complete.
 
@@ -29,6 +30,42 @@ Canonical process docs:
 
 Example-oriented docs are optional aids, not normative contracts.
 
+## Public package surfaces you should know
+
+Current root surfaces used most often by connector work:
+
+- `@useairfoil/connector-kit`
+  - core exports flattened at root
+  - `Ingestion`
+  - `Publisher`
+  - `Streams`
+  - `Webhook`
+  - flat root errors
+- `@useairfoil/effect-vcr`
+  - `CassetteStore`
+  - `FileSystemCassetteStore`
+  - `VcrHttpClient`
+  - flat root VCR types
+  - focused subpath exports for cassette store, file-system cassette store,
+    types, and VCR HTTP client
+- `@useairfoil/wings`
+  - `Cluster`
+  - `ClusterClient`
+  - `WingsClient`
+  - `Arrow`
+  - `Partition`
+  - `Schema`
+  - `Topic`
+  - flat root errors
+- `@useairfoil/flight`
+  - `ArrowFlightClient`
+  - `ArrowFlightSqlClient`
+  - `FlightClientError`
+  - root encoder/proto exports and typed client options
+
+When writing examples or guidance, prefer the actual current package surface
+over historical helper names or internal file-level imports.
+
 ## Files
 
 ```
@@ -40,9 +77,9 @@ references/
 ├── api-mode-graphql.md                # GraphQL implementation contract
 ├── api-mode-grpc.md                   # gRPC implementation contract
 ├── connector-kit-api.md               # exhaustive @useairfoil/connector-kit docs
-├── effect-vcr-api.md                  # exhaustive @useairfoil/effect-vcr docs
+├── effect-vcr-api.md                  # current @useairfoil/effect-vcr docs and wiring
 ├── effect-v4-essentials.md            # Effect v4 idioms relevant to connectors
-├── patterns.md                        # shared patterns (cursor, cutoff, streams)
+├── patterns.md                        # shared naming, layer, cursor, cutoff, and stream patterns
 ├── webhooks.md                        # WebhookRoute + signature verification
 ├── vcr-workflow.md                    # record/replay + ACK_DISABLE_VCR
 ├── api-research.md                    # how to learn a real API's shape

.agents/skills/airfoil-kit/SKILL.md

@@ -10,6 +10,10 @@ inside this monorepo. Work in small, verified steps. Use the template as your
 starting point, never guess API shapes, and keep changes aligned with the
 existing patterns in `connectors/producer-polar/`.
 
+The current repo shape matters more than historical examples. When in doubt,
+follow the current source packages and the refreshed reference docs in this
+skill.
+
 ---
 
 ## Hard rules (do not violate)
@@ -25,7 +29,9 @@ existing patterns in `connectors/producer-polar/`.
 3. **Use Effect v4 only** (`effect@4.x`, `@effect/vitest@4.x`, `@effect/platform-*@4.x`).
    No legacy `@effect/platform`, `@effect/schema`, or Effect v2/v3 patterns.
    Read [`references/effect-v4-essentials.md`](./references/effect-v4-essentials.md)
-   whenever you reach for a new Effect module.
+   whenever you reach for a new Effect module. For Effect guidance, consult
+   `effect-smol` only. Do not use the older official Effect docs as source of
+   truth for this repo right now.
 4. **No `process.env` reads in connector code or tests.** Use
    `Config`/`ConfigProvider` everywhere. Sandbox/runtime layers attach
    `ConfigProvider.fromEnv()`; tests attach `ConfigProvider.fromUnknown({ ... })`
@@ -68,6 +74,16 @@ existing patterns in `connectors/producer-polar/`.
     item in [`references/definition-of-done.md`](./references/definition-of-done.md)
     passes (lint, typecheck, build, test:ci, and mode-appropriate deterministic
     replay: VCR for REST/GraphQL, fixtures or mock servers for gRPC).
+14. **Use current names.** Prefer `make`, `layer(config)`,
+    `layerConfig(Config.Wrap<...>)`, namespace entrypoint exports,
+    `Ingestion.runConnector(...)`, `Ingestion.layerMemory`,
+    `Publisher.Publisher`, and `Webhook.route(...)`.
+15. **Use correct layer semantics.** `Layer.mergeAll(...)` is for independent
+    layers. If a layer needs another to build, satisfy that dependency with
+    `Layer.provide(...)` before merging.
+16. **Do not hide dependency graph mistakes behind casts.** If a runtime or
+    test entrypoint seems to need `as Effect.Effect<...>`, inspect the layer
+    graph first.
 
 ---
 
@@ -108,14 +124,15 @@ existing patterns in `connectors/producer-polar/`.
    [`references/patterns.md`](./references/patterns.md),
    [`references/webhooks.md`](./references/webhooks.md)
 10. **Update the sandbox runner** — rename config names and port, keep the
-    telemetry + console publisher boilerplate.
+    telemetry + console publisher shape, and preserve the dependency graph.
 11. **Write tests** —
     - REST/GraphQL: `api.vcr.test.ts` replays the backfill path.
     - gRPC: deterministic fixture/mock-server tests cover equivalent paths.
     - `webhook.test.ts` exercises webhook endpoint behavior in-memory.
       Switch to replay mode (or fixture-only deterministic mode) before
       committing.
-12. **Run the CI gate locally** — `pnpm run lint && pnpm run typecheck && pnpm run build && pnpm run test:ci`.
+12. **Run local verification in order** — `pnpm install`, then the relevant
+    `build`, `typecheck`, `test:ci`, format, and lint checks.
     Every one must pass. → [`references/definition-of-done.md`](./references/definition-of-done.md)
 
 A detailed, numbered version of this flow lives at

.agents/skills/airfoil-kit/assets/rename-checklist.md

@@ -30,15 +30,15 @@ rg -l "template" connectors/producer-<service> --glob '!**/__cassettes__' --glob
 | `@useairfoil/producer-template`                   | `@useairfoil/producer-<service>`                    |
 | `TEMPLATE_` (env prefix)                          | `<SERVICE>_`                                        |
 | `TemplateApiClient`                               | `<Service>ApiClient`                                |
-| `TemplateApiClientConfig`                         | `<Service>ApiClientConfig`                          |
+| API raw-config layer                              | `layer`                                             |
 | `TemplateApiClientService`                        | `<Service>ApiClientService`                         |
 | `TemplateListPage`                                | `<Service>ListPage`                                 |
 | `TemplateConfig` (type)                           | `<Service>Config`                                   |
 | `TemplateConfigConfig` (Config value)             | `<Service>ConfigConfig`                             |
 | `TemplateConnector` (service tag)                 | `<Service>Connector`                                |
-| `TemplateConnectorConfig` (layer factory)         | `<Service>ConnectorConfig`                          |
+| Config-decoded layers                             | `layerConfig(config)`                               |
 | `TemplateConnectorRuntime`                        | `<Service>ConnectorRuntime`                         |
-| `makeTemplateConnector`                           | `make<Service>Connector`                            |
+| Connector constructor                             | `make`                                              |
 | `Template` (any other identifier prefix)          | `<Service>`                                         |
 | `template` (lowercase in strings / URNs)          | `<service>`                                         |
 | `@useairfoil/producer-template/TemplateApiClient` | `@useairfoil/producer-<service>/<Service>ApiClient` |
@@ -92,7 +92,8 @@ recreate it.
 Rewrite `connectors/producer-<service>/README.md`:
 
 - Drop every JSONPlaceholder reference.
-- Document the real API entities, auth, base URLs, env vars.
+- Document the current public exports, real API entities, auth, runtime
+  wiring, base URLs, and env vars.
 - List known limitations specific to the target (rate limits, missing
   historical data, sandbox quirks).
 

.agents/skills/airfoil-kit/references/connector-kit-api.md

@@ -11,16 +11,10 @@ import {
   defineConnector,
   defineEntity,
   defineEvent,
-  makePullStream,
-  makeWebhookQueue,
+  Ingestion,
   Publisher,
-  runConnector,
-  StateStore,
-  StateStoreInMemory,
-  WingsPublisherLayer,
-  ConnectorRuntimeContext,
-  ConnectorRuntimeContextLayer,
-  buildWebhookRouter,
+  Streams,
+  Webhook,
 } from "@useairfoil/connector-kit";
 
 import type {
@@ -37,11 +31,8 @@ import type {
   IngestionState,
   LiveSource,
   LiveStream,
-  RunConnectorOptions,
   StreamState,
   Transform,
-  WebhookRoute,
-  WebhookStream,
 } from "@useairfoil/connector-kit";
 ```
 
@@ -232,12 +223,12 @@ runConnector(
 ): Effect.Effect<void, ConnectorError, StateStore | Publisher>;
 
 // With webhook: also requires HttpServer
-runConnector<TPayload>(
+runConnector(
   connector,
   options: {
     initialCutoff?: Cursor;
     webhook: {
-      routes: ReadonlyArray<WebhookRoute<TPayload>>;
+      routes: ReadonlyArray<Webhook.WebhookRoute>;
       healthPath?: HttpRouter.PathInput;  // default "/health"
       disableHttpLogger?: boolean;         // default true
     };
@@ -247,15 +238,35 @@ runConnector<TPayload>(
 
 Internally:
 
-- Provides `ConnectorRuntimeContextLayer(connector)` so downstream spans can
-  tag metrics with `connector.name`.
+- Provides an internal connector runtime context so downstream spans can tag
+  metrics with `connector.name`.
 - Wraps the whole run in an `Effect.withSpan("connector.run", ...)`.
 - Emits `connector_batches_total`, `connector_rows_total`, and
   `connector_batch_size` via `effect/Metric`.
 - For webhooks, composes `buildWebhookRouter(routes)` with a `/health`
   route and serves it via `HttpRouter.serve(app, { disableLogger })`.
 
-### `RunConnectorOptions<TWebhookPayload>`
+Current runtime composition pattern around `runConnector(...)`:
+
+```ts
+const ConnectorLayer = layerConfig.pipe(Layer.provide(EnvLayer));
+
+const program = Effect.gen(function* () {
+  const { connector, routes } = yield* MyConnector;
+  const serverLayer = NodeHttpServer.layer(createServer, { port: 8080 });
+
+  return yield* Ingestion.runConnector(connector, {
+    initialCutoff: new Date(),
+    webhook: {
+      routes,
+      healthPath: "/health",
+      disableHttpLogger: true,
+    },
+  }).pipe(Effect.provide(serverLayer));
+});
+```
+
+### `Ingestion.RunConnectorOptions`
 
 Exposed type for callers who build options programmatically.
 
@@ -282,7 +293,7 @@ class StateStore extends Context.Service<
 
 Keyed by entity/event name. One row per stream.
 
-### `StateStoreInMemory`
+### `Ingestion.layerMemory`
 
 In-process `Map<string, IngestionState>` backed `StateStore` layer. Use for
 the sandbox runner and tests. Production deployments provide a durable
@@ -310,10 +321,10 @@ class Publisher extends Context.Service<
 `PublishAck = { readonly success: boolean }`. The engine fails the stream
 if `publish` fails.
 
-### `WingsPublisherLayer(config)`
+### `Publisher.layerWings(config)`
 
 ```ts
-WingsPublisherLayer({
+Publisher.layerWings({
   connector,
   topics: { customers: customerTopic, orders: orderTopic },
   partitionValues: { customers: "account_id" },
@@ -323,24 +334,27 @@ WingsPublisherLayer({
 Production-grade publisher that fans each entity into a Wings topic. For
 the sandbox / tests, use a hand-written console publisher instead.
 
+Current tag access pattern in this repo is `Publisher.Publisher` from the root
+module namespace.
+
 ---
 
 ## Streams
 
-### `makeWebhookQueue<T>(options?)`
+### `Streams.makeWebhookQueue<T>(options?)`
 
 ```ts
-makeWebhookQueue<T>({ capacity?: number }): Effect.Effect<WebhookStream<T>>;
+Streams.makeWebhookQueue<T>({ capacity?: number }): Effect.Effect<WebhookStream<T>>;
 ```
 
 Creates a bounded `Queue` (default capacity 1024) and its `Stream.fromQueue`
 view. Always keep the queue bounded — unbounded queues can let a noisy
 webhook drown the publisher.
 
-### `makePullStream<T, R>(options)`
+### `Streams.makePullStream<T, R>(options)`
 
 ```ts
-makePullStream({
+Streams.makePullStream({
   initialCursor?: Cursor,
   fetchPage: (cursor: Cursor | undefined) => Effect.Effect<PullPage<T>, ConnectorError, R>,
 }): Stream.Stream<Batch<T>, ConnectorError, R>;
@@ -360,14 +374,14 @@ list endpoint.
 
 ## Webhooks
 
-### `WebhookRoute<TPayload>`
+### `Webhook.WebhookRoute<S>`
 
 ```ts
-type WebhookRoute<TPayload> = {
+type WebhookRoute<S extends Schema.Schema<any>> = {
   readonly path: HttpRouter.PathInput;
-  readonly schema: Schema.Schema<TPayload>;
+  readonly schema: S;
   readonly handle: (
-    payload: TPayload,
+    payload: Schema.Schema.Type<S>,
     request: HttpServerRequest.HttpServerRequest,
     rawBody?: Uint8Array,
   ) => Effect.Effect<void, ConnectorError>;
@@ -378,7 +392,7 @@ The framework decodes the request body, validates against `schema`, and
 invokes `handle(payload, request, rawBody)`. Use `rawBody` for HMAC
 verification; use `payload` for dispatch.
 
-### `buildWebhookRouter(routes)`
+### `Webhook.buildWebhookRouter(routes)`
 
 Low-level helper that turns an array of routes into an `HttpRouter` Layer.
 `runConnector(...)` uses this internally; you rarely call it directly.
@@ -387,19 +401,6 @@ Low-level helper that turns an array of routes into an `HttpRouter` Layer.
 
 ## Runtime context
 
-### `ConnectorRuntimeContext`
-
-Service tag exposing `{ connector: ConnectorDefinition }`. The engine sets
-this via `ConnectorRuntimeContextLayer(connector)`. Metrics attributes use
-it to tag batches with `connector.name`.
-
-### `ConnectorRuntimeContextLayer(connector)`
-
-Returns a `Layer.succeed(ConnectorRuntimeContext)({ connector })`. Call this
-in custom test harnesses if you bypass `runConnector`.
-
----
-
 ## Observability (provided by the engine)
 
 ### Spans
@@ -429,27 +430,36 @@ sandbox uses `Observability.Otlp.layerJson({ baseUrl, resource })` from
 ## Typical composition recipe
 
 ```ts
-const runtimeLayer = Layer.mergeAll(
-  StateStoreInMemory,
-  ConsolePublisherLayer, // or WingsPublisherLayer(...)
-  MyConnectorConfig(), // Layer<MyConnector, ConnectorError, HttpClient>
+const EnvLayer = Layer.mergeAll(
+  FetchHttpClient.layer,
+  Layer.succeed(ConfigProvider.ConfigProvider, ConfigProvider.fromEnv()),
+)
+
+const ConnectorLayer = layerConfig.pipe(Layer.provide(EnvLayer))
+
+const TelemetryLayer = Layer.unwrap(...).pipe(Layer.provide(EnvLayer))
+
+const RuntimeLayer = Layer.mergeAll(
+  Ingestion.layerMemory,
+  ConsolePublisherLayer, // or Publisher.layerWings(...)
+  ConnectorLayer,
   Logger.layer([Logger.consolePretty()]),
-  TelemetryLayer, // optional
-  Layer.mergeAll(
-    FetchHttpClient.layer,
-    Layer.succeed(ConfigProvider.ConfigProvider, ConfigProvider.fromEnv()),
-  ),
+  TelemetryLayer,
 );
 
 const program = Effect.gen(function* () {
   const { connector, routes } = yield* MyConnector;
-  return yield* runConnector(connector, {
+  return yield* Ingestion.runConnector(connector, {
     initialCutoff: new Date(),
-    webhook: { routes },
+    webhook: {
+      routes,
+      healthPath: "/health",
+      disableHttpLogger: true,
+    },
   }).pipe(Effect.provide(NodeHttpServer.layer(createServer, { port: 8080 })));
-});
+}).pipe(Effect.annotateLogs({ component: "producer-foo" }));
 
-Effect.runPromise(Effect.scoped(program).pipe(Effect.provide(runtimeLayer)));
+Effect.runPromise(Effect.scoped(program).pipe(Effect.provide(RuntimeLayer)));
 ```
 
 See `connectors/producer-polar/src/sandbox.ts` for the live reference.