diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..3511fe555 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,252 @@ +# LaunchQL Agent Navigation Guide + +This guide helps AI agents quickly navigate and understand the LaunchQL codebase without having to read everything. LaunchQL is a comprehensive full-stack framework for building secure, role-aware GraphQL APIs backed by PostgreSQL databases. + +## ๐ŸŽฏ Quick Start for Agents + +**Most Important Packages to Know First:** +1. **`packages/core`** - Main orchestration and migration engine ([detailed guide](packages/core/AGENTS.md)) +2. **`packages/cli`** - Command-line interface and user workflows ([detailed guide](packages/cli/AGENTS.md)) +3. **`packages/pgsql-test`** - Testing infrastructure ([detailed guide](packages/pgsql-test/AGENTS.md)) +4. **`packages/server`** - GraphQL API server +5. **`packages/types`** - TypeScript type definitions + +**Key Classes to Understand:** +- **`LaunchQLPackage`** (`packages/core/src/core/class/launchql.ts`) - Workspace and module management +- **`LaunchQLMigrate`** (`packages/core/src/migrate/client.ts`) - Database migration operations + +## ๐Ÿ“ฆ Package Categories + +### ๐Ÿ—๏ธ Core Framework +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`core`** | Main orchestration, migrations, dependency resolution | `src/core/class/launchql.ts`, `src/migrate/client.ts` | +| **`cli`** | Command-line interface (`lql` command) | `src/commands.ts`, `src/commands/deploy.ts` | +| **`types`** | TypeScript type definitions | `src/launchql.ts` | +| **`env`** | Environment and configuration management | - | + +### ๐Ÿš€ API & Server +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`server`** | Express + PostGraphile GraphQL API server | `src/middleware/api.ts` | +| **`explorer`** | GraphiQL API browser/debugger | - | +| **`graphile-settings`** | PostGraphile configuration utilities | - | +| **`graphile-cache`** | Caching for GraphQL operations | - | +| **`graphile-test`** | GraphQL testing utilities | - | +| **`graphile-query`** | GraphQL query building | - | + +### ๐Ÿงช Testing Infrastructure +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`pgsql-test`** | Isolated PostgreSQL test environments | `src/test-client.ts` | +| **`pg-query-context`** | Session context injection for tests | - | + +### ๐Ÿ”„ Database & Migration +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`pg-env`** | PostgreSQL environment configuration | - | +| **`pg-cache`** | PostgreSQL connection pooling | - | +| **`pg-codegen`** | Code generation from PostgreSQL schemas | - | +| **`introspectron`** | PostgreSQL schema introspection | - | + +### ๐Ÿง  Parsing & AST +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`pg-ast`** | PostgreSQL AST tools and transformations | - | +| **`gql-ast`** | GraphQL AST utilities | - | + +### ๐Ÿ” Streaming & File Handling +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`s3-streamer`** | Direct S3 streaming for large files | - | +| **`s3-utils`** | S3 utilities and helpers | - | +| **`etag-hash`** | S3-compatible ETag generation | - | +| **`etag-stream`** | ETag computation via streams | - | +| **`stream-to-etag`** | Stream-to-ETag transformation | - | +| **`uuid-hash`** | Deterministic UUID generation | - | +| **`uuid-stream`** | Streaming UUID generation | - | +| **`upload-names`** | Collision-resistant filename utilities | - | +| **`content-type-stream`** | Content type detection in streams | - | +| **`mime-bytes`** | MIME type utilities | - | + +### ๐Ÿ—๏ธ Query Building & Code Generation +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`query-builder`** | TypeScript SQL query builder | - | +| **`query`** | Fluent GraphQL query builder | - | +| **`launchql-gen`** | Auto-generated GraphQL mutations/queries | - | +| **`client`** | Client-side utilities | - | +| **`react`** | React integration components | - | + +### ๐Ÿ› ๏ธ Development Tools +| Package | Purpose | Key Files | +|---------|---------|-----------| +| **`templatizer`** | Template rendering system | - | +| **`logger`** | Logging utilities | - | +| **`url-domains`** | URL and domain utilities | - | +| **`server-utils`** | Server utility functions | - | +| **`orm`** | Object-relational mapping utilities | - | + +## ๐Ÿ”‘ Key Classes and Entry Points + +### LaunchQLPackage Class +**Location:** `packages/core/src/core/class/launchql.ts` + +**Purpose:** High-level orchestration for workspace and module management + +**Key Methods:** +- `deploy(opts, target?, recursive?)` - Deploy modules to database +- `revert(opts, target?)` - Revert database changes +- `verify(opts, target?)` - Verify database state +- `getModuleMap()` - Get all available modules +- `initModule(options)` - Initialize new module +- `installModules(...packages)` - Install module dependencies +- `addTag(tagName, changeName?, comment?)` - Tag changes for versioning + +**Context Detection:** +- `getContext()` - Determine if in workspace, module, or outside +- `isInWorkspace()` / `isInModule()` - Context checks + +### LaunchQLMigrate Class +**Location:** `packages/core/src/migrate/client.ts` + +**Purpose:** Low-level database migration operations + +**Key Methods:** +- `deploy(options)` - Deploy changes to database +- `revert(options)` - Revert changes from database +- `verify(options)` - Verify deployed changes +- `status(packageName?)` - Get deployment status +- `isDeployed(packageName, changeName)` - Check if change is deployed +- `initialize()` - Set up migration schema + +**Configuration:** +- Supports `content` or `ast` hash methods for SQL files +- Configurable via `LaunchQLMigrateOptions` + +### CLI Command Structure +**Location:** `packages/cli/src/commands.ts` + +**Main Commands:** +- `deploy` - Deploy database changes +- `revert` - Revert database changes +- `verify` - Verify database state +- `init` - Initialize workspace or module +- `server` - Start development server +- `explorer` - Launch GraphiQL explorer +- `migrate` - Migration management +- `install` - Install module dependencies +- `tag` - Version management + +**Command Pattern:** All commands follow the pattern: +```typescript +export default async (argv: ParsedArgs, prompter: Inquirerer, options: CLIOptions) => { + // Command implementation +} +``` + +## ๐Ÿš€ Common Workflows + +### 1. Module Development Workflow +```typescript +// 1. Initialize workspace +const pkg = new LaunchQLPackage(cwd); +pkg.initWorkspace(); + +// 2. Create module +pkg.initModule({ + name: 'my-module', + description: 'My module', + author: 'Developer', + extensions: ['uuid-ossp'] +}); + +// 3. Deploy changes +await pkg.deploy(options); +``` + +### 2. Testing Workflow +```typescript +// Set up isolated test database +import { getConnections } from 'pgsql-test'; + +const { db, teardown } = await getConnections(); +await db.query('SELECT 1'); // Ready for testing +``` + +### 3. Migration Workflow +```typescript +// Direct migration operations +const migrate = new LaunchQLMigrate(pgConfig); +await migrate.deploy({ modulePath: './my-module' }); +await migrate.verify({ modulePath: './my-module' }); +``` + +## ๐Ÿ“ File Structure Patterns + +### Module Structure +``` +my-module/ +โ”œโ”€โ”€ launchql.plan # Migration plan +โ”œโ”€โ”€ my-module.control # Extension metadata +โ”œโ”€โ”€ Makefile # Build configuration +โ”œโ”€โ”€ deploy/ # Deploy scripts +โ”‚ โ”œโ”€โ”€ init.sql +โ”‚ โ””โ”€โ”€ tables.sql +โ”œโ”€โ”€ revert/ # Revert scripts +โ”‚ โ”œโ”€โ”€ tables.sql +โ”‚ โ””โ”€โ”€ init.sql +โ””โ”€โ”€ verify/ # Verification scripts + โ”œโ”€โ”€ init.sql + โ””โ”€โ”€ tables.sql +``` + +### Workspace Structure +``` +workspace/ +โ”œโ”€โ”€ launchql.config.js # Workspace configuration +โ”œโ”€โ”€ packages/ # Module packages +โ”‚ โ”œโ”€โ”€ module-a/ +โ”‚ โ””โ”€โ”€ module-b/ +โ””โ”€โ”€ extensions/ # Installed extensions +``` + +## ๐Ÿ” Finding Specific Functionality + +### For Database Operations +- **Migrations:** `packages/core/src/migrate/` +- **Schema introspection:** `packages/introspectron/` +- **Connection management:** `packages/pg-cache/`, `packages/pg-env/` + +### For API Development +- **GraphQL server:** `packages/server/` +- **Query building:** `packages/query/`, `packages/query-builder/` +- **Code generation:** `packages/launchql-gen/` + +### For Testing +- **Test infrastructure:** `packages/pgsql-test/` +- **GraphQL testing:** `packages/graphile-test/` +- **Context injection:** `packages/pg-query-context/` + +### For File Operations +- **S3 streaming:** `packages/s3-streamer/` +- **File hashing:** `packages/etag-hash/`, `packages/uuid-hash/` +- **Upload handling:** `packages/upload-names/` + +## ๐Ÿ“š Additional Resources + +- **Package-specific guides:** See `packages/*/AGENTS.md` for detailed documentation +- **Type definitions:** `packages/types/src/` for comprehensive TypeScript interfaces +- **Examples:** `sandbox/` directory contains example projects +- **Tests:** Each package's `__tests__/` directory shows usage patterns + +## ๐ŸŽฏ Agent Tips + +1. **Start with `packages/core/AGENTS.md`** for deep understanding of the main classes +2. **Check `packages/cli/AGENTS.md`** to understand user-facing workflows +3. **Use `packages/pgsql-test/AGENTS.md`** for testing patterns +4. **Look at existing tests** in `__tests__/` directories for usage examples +5. **Check `package.json` files** to understand dependencies between packages +6. **Use the type definitions** in `packages/types/` to understand interfaces + +This navigation guide should help you quickly find the right code without reading the entire codebase. For detailed information about specific packages, refer to their individual AGENTS.md files. diff --git a/packages/cli/AGENTS.md b/packages/cli/AGENTS.md new file mode 100644 index 000000000..ce54b38d3 --- /dev/null +++ b/packages/cli/AGENTS.md @@ -0,0 +1,478 @@ +# LaunchQL CLI Package - Agent Guide + +The `@launchql/cli` package provides the command-line interface for LaunchQL, exposing all framework functionality through the `lql` command. This guide helps agents understand the CLI structure and command patterns. + +## ๐ŸŽฏ Overview + +**Main Purpose:** User-facing command-line toolkit for managing LaunchQL projects, supporting database scaffolding, migrations, seeding, code generation, and automation. + +**Binary:** `lql` (LaunchQL command-line tool) + +**Key Features:** +- Database-first development workflow +- Module system with dependency management +- Smart migrations with automated deployment +- Development server with hot-reload +- Production-ready deployment plans + +## ๐Ÿ—๏ธ CLI Architecture + +### Main Entry Point +**File:** `src/commands.ts` + +**Command Structure:** +```typescript +export const commands = async ( + argv: Partial, + prompter: Inquirerer, + options: CLIOptions & { skipPgTeardown?: boolean } +) => { + // Command routing and execution +} +``` + +**Connection Management:** +All commands (except `server` and `explorer`) are wrapped with `withPgTeardown()` to ensure proper PostgreSQL connection cleanup. + +### Command Map +**Available Commands:** +```typescript +const commandMap = { + 'admin-users': adminUsers, // User management + 'clear': clear, // Clear/cleanup operations + 'deploy': deploy, // Deploy database changes + 'verify': verify, // Verify database state + 'revert': revert, // Revert database changes + 'remove': remove, // Remove changes from plan + 'init': init, // Initialize workspace/module + 'extension': extension, // Manage module dependencies + 'plan': plan, // Generate deployment plans + 'export': _export, // Export migrations + 'package': _package, // Package modules + 'tag': tag, // Version management + 'kill': kill, // Cleanup connections/databases + 'install': install, // Install module dependencies + 'migrate': migrate, // Migration management + 'analyze': analyze, // Analyze modules + 'rename': rename, // Rename modules + 'server': server, // Development server + 'explorer': explorer // GraphiQL explorer +}; +``` + +## ๐Ÿš€ Core Commands + +### 1. Deploy Command +**File:** `src/commands/deploy.ts` +**Purpose:** Deploy database changes and migrations to target database + +**Usage:** +```bash +lql deploy [OPTIONS] +``` + +**Key Options:** +- `--createdb` - Create database if it doesn't exist +- `--recursive` - Deploy recursively through dependencies (default) +- `--package ` - Target specific package +- `--to ` - Deploy to specific change or tag +- `--tx` - Use transactions (default: true) +- `--fast` - Use fast deployment strategy +- `--logOnly` - Log-only mode, skip script execution +- `--usePlan` - Use deployment plan +- `--cache` - Enable caching + +**Implementation Pattern:** +```typescript +export default async ( + argv: Partial, + prompter: Inquirerer, + _options: CLIOptions +) => { + // 1. Show usage if requested + if (argv.help || argv.h) { + console.log(deployUsageText); + process.exit(0); + } + + // 2. Get target database + const database = await getTargetDatabase(argv, prompter, { + message: 'Select database' + }); + + // 3. Prompt for confirmation and options + const { yes, tx, fast, logOnly } = await prompter.prompt(argv, questions); + + // 4. Create database if requested + if (createdb) { + execSync(`createdb ${database}`, { + env: getSpawnEnvWithPg(pgEnv) + }); + } + + // 5. Execute deployment + const project = new LaunchQLPackage(cwd); + await project.deploy(opts, target, recursive); +} +``` + +### 2. Init Command +**File:** `src/commands/init.ts` +**Purpose:** Initialize new LaunchQL workspace or module + +**Subcommands:** +- **Workspace Init:** `src/commands/init/workspace.ts` +- **Module Init:** `src/commands/init/module.ts` + +**Usage:** +```bash +lql init --workspace # Initialize workspace +lql init # Initialize module (in workspace) +``` + +### 3. Server Command +**File:** `src/commands/server.ts` +**Purpose:** Start GraphQL development server with hot-reload + +**Features:** +- PostGraphile-powered GraphQL API +- Automatic schema generation from database +- Hot-reload on database changes +- GraphiQL interface +- CORS configuration + +### 4. Migrate Command +**File:** `src/commands/migrate.ts` +**Purpose:** Comprehensive migration management + +**Subcommands:** +- `init` - Initialize migration tracking +- `status` - Check migration status +- `list` - List all changes +- `deps` - Show change dependencies + +**Subcommand Files:** +- `src/commands/migrate/status.ts` +- `src/commands/migrate/list.ts` +- `src/commands/migrate/deps.ts` + +### 5. Extension Command +**File:** `src/commands/extension.ts` +**Purpose:** Interactively manage module dependencies + +**Features:** +- Interactive dependency selection +- Automatic .control file updates +- Dependency validation + +## ๐ŸŽฎ Command Patterns + +### Standard Command Structure +All CLI commands follow this pattern: + +```typescript +export default async ( + argv: Partial, // Command-line arguments + prompter: Inquirerer, // Interactive prompting + options: CLIOptions // CLI options +) => { + // 1. Help handling + if (argv.help || argv.h) { + console.log(usageText); + process.exit(0); + } + + // 2. Argument processing and prompting + const answers = await prompter.prompt(argv, questions); + + // 3. Validation and confirmation + if (!answers.confirmed) { + log.info('Operation cancelled.'); + return; + } + + // 4. Core logic execution + const project = new LaunchQLPackage(cwd); + await project.someOperation(options); + + // 5. Success reporting + log.success('Operation complete.'); + return argv; +}; +``` + +### Interactive Prompting +Commands use the `Inquirerer` for interactive prompts: + +```typescript +const questions: Question[] = [ + { + type: 'confirm', + name: 'confirmed', + message: 'Are you sure you want to proceed?', + required: true + }, + { + type: 'text', + name: 'database', + message: 'Database name', + required: true + }, + { + type: 'autocomplete', + name: 'package', + message: 'Select package', + options: availablePackages + } +]; + +const answers = await prompter.prompt(argv, questions); +``` + +### Database Selection +**File:** `src/utils/database.ts` + +**Key Function:** +```typescript +export async function getTargetDatabase( + argv: Partial, + prompter: Inquirerer, + options: { message: string } +): Promise +``` + +**Features:** +- Lists available databases +- Supports database creation +- Handles PostgreSQL connection errors + +### Module Selection +**File:** `src/utils/module-utils.ts` + +**Key Function:** +```typescript +export async function selectPackage( + argv: Partial, + prompter: Inquirerer, + cwd: string, + operation: string, + log: Logger +): Promise +``` + +## ๐Ÿ”ง Utility Systems + +### Environment Integration +Commands integrate with environment configuration: + +```typescript +import { getEnvOptions } from '@launchql/env'; +import { getPgEnvOptions, getSpawnEnvWithPg } from 'pg-env'; + +// Get PostgreSQL environment +const pgEnv = getPgEnvOptions(); + +// Override with CLI options +const cliOverrides = { + pg: getPgEnvOptions({ database }), + deployment: { + useTx: tx !== false, + fast: fast !== false, + usePlan: argv.usePlan !== false + } +}; + +const opts = getEnvOptions(cliOverrides); +``` + +### Connection Management +**PostgreSQL Cleanup:** +```typescript +const withPgTeardown = (fn: Function, skipTeardown: boolean = false) => + async (...args: any[]) => { + try { + await fn(...args); + } finally { + if (!skipTeardown) { + await teardownPgPools(); + } + } + }; +``` + +### Error Handling +Commands provide comprehensive error handling: + +```typescript +try { + await project.deploy(opts, target, recursive); + log.success('Deployment complete.'); +} catch (error) { + log.error('Deployment failed:', error); + process.exit(1); +} +``` + +## ๐Ÿ“‹ Command Reference + +### Development Commands +| Command | Purpose | Key Options | +|---------|---------|-------------| +| `init` | Initialize workspace/module | `--workspace` | +| `server` | Start development server | `--port`, `--no-postgis` | +| `explorer` | Launch GraphiQL explorer | `--origin` | + +### Database Commands +| Command | Purpose | Key Options | +|---------|---------|-------------| +| `deploy` | Deploy changes | `--createdb`, `--fast`, `--package`, `--to` | +| `verify` | Verify database state | `--package` | +| `revert` | Revert changes | `--to` | + +### Migration Commands +| Command | Purpose | Key Options | +|---------|---------|-------------| +| `migrate init` | Initialize migration tracking | - | +| `migrate status` | Check migration status | - | +| `migrate list` | List all changes | - | +| `migrate deps` | Show dependencies | - | + +### Module Commands +| Command | Purpose | Key Options | +|---------|---------|-------------| +| `install` | Install module dependencies | - | +| `extension` | Manage dependencies interactively | - | +| `tag` | Version changes | `--comment`, `--changeName` | +| `analyze` | Analyze module issues | - | +| `rename` | Rename module | `--to` | + +### Packaging Commands +| Command | Purpose | Key Options | +|---------|---------|-------------| +| `plan` | Generate deployment plans | - | +| `package` | Package module for distribution | `--no-plan` | +| `export` | Export migrations | - | + +### Utility Commands +| Command | Purpose | Key Options | +|---------|---------|-------------| +| `kill` | Clean up connections/databases | `--no-drop` | +| `clear` | Clear/cleanup operations | - | +| `admin-users` | User management | - | +| `remove` | Remove changes from plan | - | + +## ๐ŸŽฏ Common Usage Patterns + +### 1. New Project Setup +```bash +# Initialize workspace +lql init --workspace +cd my-project + +# Create first module +lql init + +# Deploy to database +lql deploy --createdb + +# Start development server +lql server +``` + +### 2. Module Development +```bash +# Create new module +lql init + +# Add dependencies +lql extension + +# Deploy changes +lql deploy + +# Verify deployment +lql verify +``` + +### 3. Production Deployment +```bash +# Generate deployment plan +lql plan + +# Package module +lql package + +# Deploy to production +lql deploy --package myapp --to @production + +# Verify deployment +lql verify --package myapp +``` + +### 4. Migration Management +```bash +# Check migration status +lql migrate status + +# List all changes +lql migrate list + +# Show dependencies +lql migrate deps + +# Revert to specific point +lql revert --to @v1.0.0 +``` + +## ๐Ÿ” Integration with Core Classes + +### LaunchQLPackage Integration +Commands primarily use `LaunchQLPackage` for operations: + +```typescript +import { LaunchQLPackage } from '@launchql/core'; + +const project = new LaunchQLPackage(cwd); + +// Context-aware operations +if (project.isInWorkspace()) { + // Workspace operations +} else if (project.isInModule()) { + // Module operations +} + +// Deploy with options +await project.deploy(opts, target, recursive); +``` + +### LaunchQLMigrate Integration +Some commands use `LaunchQLMigrate` directly: + +```typescript +import { LaunchQLMigrate } from '@launchql/core'; + +const migrate = new LaunchQLMigrate(pgConfig); +const result = await migrate.status(); +``` + +## ๐Ÿ“ Key Files to Understand + +1. **`src/commands.ts`** - Main command router and setup +2. **`src/commands/deploy.ts`** - Primary deployment command +3. **`src/commands/init.ts`** - Workspace/module initialization +4. **`src/commands/server.ts`** - Development server +5. **`src/commands/migrate.ts`** - Migration management +6. **`src/utils/database.ts`** - Database selection utilities +7. **`src/utils/module-utils.ts`** - Module selection utilities +8. **`package.json`** - CLI binary configuration + +## ๐ŸŽฏ Agent Tips + +1. **Command Pattern:** All commands follow the same structure - study `deploy.ts` as a template +2. **Interactive Prompts:** Use `prompter.prompt()` for user input with validation +3. **Environment Integration:** Always use `getEnvOptions()` for configuration +4. **Error Handling:** Provide clear error messages and exit codes +5. **Help Text:** Include comprehensive usage text for each command +6. **Connection Cleanup:** Use `withPgTeardown()` wrapper for database commands + +This guide covers the essential aspects of the CLI package. The CLI serves as the primary interface between users and the LaunchQL core functionality, so understanding these patterns is crucial for extending or debugging the system. diff --git a/packages/core/AGENTS.md b/packages/core/AGENTS.md new file mode 100644 index 000000000..ece7cba54 --- /dev/null +++ b/packages/core/AGENTS.md @@ -0,0 +1,381 @@ +# LaunchQL Core Package - Agent Guide + +The `@launchql/core` package is the heart of the LaunchQL framework, providing the main orchestration classes and migration engine. This guide helps agents understand the key classes and their methods. + +## ๐ŸŽฏ Overview + +**Main Purpose:** Database migrations, package management, and package scaffolding for PostgreSQL-backed applications. + +**Key Responsibilities:** +- Managing PostgreSQL extensions and modules +- Deploying, reverting, and verifying migrations +- Parsing and generating migration plans +- Reading and writing SQL scripts +- Resolving dependencies between migrations + +## ๐Ÿ—๏ธ Core Architecture + +### Main Entry Point +**File:** `src/index.ts` + +**Key Exports:** +```typescript +// Main classes +export { LaunchQLPackage } from './core/class/launchql'; +export { LaunchQLMigrate } from './migrate/client'; +export { LaunchQLInit } from './init/client'; + +// Migration types +export { DeployOptions, DeployResult, RevertOptions, RevertResult, StatusResult, VerifyOptions, VerifyResult } from './migrate/types'; + +// Utilities +export { hashFile, hashString } from './migrate/utils/hash'; +export { executeQuery, TransactionContext, TransactionOptions, withTransaction } from './migrate/utils/transaction'; +``` + +## ๐ŸŽฎ LaunchQLPackage Class + +**Location:** `src/core/class/launchql.ts` +**Purpose:** High-level orchestration for workspace and module management + +### Context Management + +```typescript +enum PackageContext { + Outside = 'outside', + Workspace = 'workspace-root', + Module = 'module', + ModuleInsideWorkspace = 'module-in-workspace' +} +``` + +**Key Context Methods:** +- `getContext(): PackageContext` - Determine current context +- `isInWorkspace(): boolean` - Check if in workspace +- `isInModule(): boolean` - Check if in module +- `ensureWorkspace()` - Throw error if not in workspace +- `ensureModule()` - Throw error if not in module + +### Workspace Operations + +**Module Discovery:** +- `getModules(): Promise` - Get all workspace modules +- `listModules(): ModuleMap` - Parse .control files to find modules +- `getModuleMap(): ModuleMap` - Get cached module map +- `getAvailableModules(): string[]` - Get list of available module names +- `getModuleProject(name: string): LaunchQLPackage` - Get specific module project + +**Module Creation:** +- `initModule(options: InitModuleOptions): void` - Initialize new module +- `createModuleDirectory(modName: string): string` - Create module directory structure + +```typescript +interface InitModuleOptions { + name: string; + description: string; + author: string; + extensions: string[]; +} +``` + +### Module Operations + +**Module Information:** +- `getModuleInfo(): ExtensionInfo` - Get current module info +- `getModuleName(): string` - Get current module name +- `getRequiredModules(): string[]` - Get module dependencies + +**Dependency Management:** +- `setModuleDependencies(modules: string[]): void` - Set module dependencies +- `validateModuleDependencies(modules: string[]): void` - Check for circular dependencies +- `getModuleExtensions(): { resolved: string[]; external: string[] }` - Get extension dependencies +- `getModuleDependencies(moduleName: string): { native: string[]; modules: string[] }` - Get dependencies +- `getModuleDependencyChanges(moduleName: string)` - Get dependency changes with versions + +### Plan Management + +**Plan Operations:** +- `getModulePlan(): string` - Read launchql.plan file +- `generateModulePlan(options): string` - Generate plan from dependencies +- `writeModulePlan(options): void` - Write generated plan to file +- `addTag(tagName: string, changeName?: string, comment?: string): void` - Add version tag + +**Plan Generation Options:** +```typescript +interface PlanOptions { + uri?: string; + includePackages?: boolean; + includeTags?: boolean; +} +``` + +### Deployment Operations + +**Main Deployment Method:** +```typescript +async deploy( + opts: LaunchQLOptions, + target?: string, // Package:change or package:@tag format + recursive: boolean = true +): Promise +``` + +**Deployment Features:** +- **Fast Deployment:** Uses `packageModule()` for consolidated SQL +- **Standard Deployment:** Uses `LaunchQLMigrate` for change-by-change deployment +- **Dependency Resolution:** Automatically resolves and deploys dependencies +- **Caching:** Optional caching for fast deployments +- **Transaction Control:** Configurable transaction usage + +**Revert Operations:** +```typescript +async revert(opts: LaunchQLOptions, target?: string): Promise +``` + +**Verification:** +```typescript +async verify(opts: LaunchQLOptions, target?: string): Promise +``` + +### Package Management + +**Installation:** +- `installModules(...pkgstrs: string[]): Promise` - Install npm packages as modules +- `publishToDist(distFolder?: string): void` - Package module for distribution + +**Analysis:** +- `analyzeModule(): PackageAnalysisResult` - Analyze module for issues +- `renameModule(to: string, options?: RenameOptions): void` - Rename module + +**Utility Methods:** +- `removeFromPlan(changeName: string): void` - Remove change from plan +- `resolveWorkspaceExtensionDependencies()` - Get all workspace dependencies +- `parsePackageTarget(target?: string)` - Parse deployment target strings + +## ๐Ÿ”„ LaunchQLMigrate Class + +**Location:** `src/migrate/client.ts` +**Purpose:** Low-level database migration execution + +### Configuration + +```typescript +interface LaunchQLMigrateOptions { + hashMethod?: 'content' | 'ast'; // How to hash SQL files +} +``` + +**Hash Methods:** +- `content` - Hash raw file content (fast, sensitive to formatting) +- `ast` - Hash parsed AST structure (robust, ignores formatting) + +### Core Migration Operations + +**Deploy Changes:** +```typescript +async deploy(options: DeployOptions): Promise +``` + +**DeployOptions:** +```typescript +interface DeployOptions { + modulePath: string; + toChange?: string; // Stop at specific change + useTransaction?: boolean; // Default: true + debug?: boolean; // Show full SQL on errors + logOnly?: boolean; // Log without executing + usePlan?: boolean; // Use plan vs SQL files +} +``` + +**Revert Changes:** +```typescript +async revert(options: RevertOptions): Promise +``` + +**Verify Changes:** +```typescript +async verify(options: VerifyOptions): Promise +``` + +### Status and Introspection + +**Status Checking:** +- `status(packageName?: string): Promise` - Get deployment status +- `isDeployed(packageName: string, changeName: string): Promise` - Check if change is deployed +- `getRecentChanges(targetDatabase: string, limit?: number)` - Get recent deployments +- `getPendingChanges(planPath: string, targetDatabase: string)` - Get undeployed changes +- `getDeployedChanges(targetDatabase: string, packageName: string)` - Get all deployed changes +- `getDependencies(packageName: string, changeName: string)` - Get change dependencies + +**Migration Schema:** +- `initialize(): Promise` - Create launchql_migrate schema +- `hasSqitchTables(): Promise` - Check for existing Sqitch tables +- `importFromSqitch(): Promise` - Import from existing Sqitch deployment + +### Error Handling and Debugging + +**Error Context:** The deploy method provides comprehensive error information: +- Change name and package +- Script hash and dependencies +- SQL script preview (first 10 lines by default, full script in debug mode) +- PostgreSQL error codes with hints +- Debugging suggestions based on error type + +**Common Error Codes:** +- `25P02` - Previous command in transaction failed +- `42P01` - Table/view does not exist +- `42883` - Function does not exist + +## ๐Ÿ”ง Supporting Systems + +### Dependency Resolution +**Location:** `src/resolution/deps.ts` + +**Key Functions:** +- `resolveDependencies(cwd, moduleName, options)` - Resolve module dependencies +- `resolveExtensionDependencies(moduleName, moduleMap)` - Resolve extension dependencies + +### File Operations +**Location:** `src/files/` + +**Key Functions:** +- `parsePlanFile(planPath)` - Parse launchql.plan files +- `generatePlan(options)` - Generate plan content +- `writePlan(planPath, content)` - Write plan files +- `readScript(baseDir, type, changeName)` - Read SQL scripts + +### Module Management +**Location:** `src/modules/modules.ts` + +**Key Functions:** +- `getExtensionsAndModules(moduleName, modules)` - Get module dependencies +- `latestChange(moduleName, modules, workspacePath)` - Get latest change +- `latestChangeAndVersion(moduleName, modules, workspacePath)` - Get latest with version + +## ๐ŸŽฏ Common Usage Patterns + +### 1. Workspace Setup +```typescript +const pkg = new LaunchQLPackage('/path/to/workspace'); +if (!pkg.isInWorkspace()) { + // Initialize workspace +} + +// Create new module +pkg.initModule({ + name: 'my-module', + description: 'My module description', + author: 'Developer Name', + extensions: ['uuid-ossp', 'postgis'] +}); +``` + +### 2. Module Deployment +```typescript +const pkg = new LaunchQLPackage('/path/to/module'); +const options = { + pg: { database: 'mydb', host: 'localhost' }, + deployment: { useTx: true, fast: false } +}; + +// Deploy single module +await pkg.deploy(options); + +// Deploy to specific change +await pkg.deploy(options, 'my-module:my-change'); + +// Deploy to tag +await pkg.deploy(options, 'my-module:@v1.0.0'); +``` + +### 3. Direct Migration Operations +```typescript +const migrate = new LaunchQLMigrate(pgConfig, { hashMethod: 'ast' }); + +// Deploy changes +const result = await migrate.deploy({ + modulePath: '/path/to/module', + useTransaction: true, + debug: true +}); + +if (result.failed) { + console.error(`Deployment failed at: ${result.failed}`); +} +``` + +### 4. Dependency Management +```typescript +const pkg = new LaunchQLPackage('/path/to/module'); + +// Get module dependencies +const deps = pkg.getModuleDependencies('my-module'); +console.log('Native extensions:', deps.native); +console.log('Module dependencies:', deps.modules); + +// Install new dependencies +await pkg.installModules('@launchql/auth', '@launchql/utils'); +``` + +### 5. Plan Management +```typescript +const pkg = new LaunchQLPackage('/path/to/module'); + +// Generate plan with external dependencies +const plan = pkg.generateModulePlan({ + includePackages: true, + includeTags: true +}); + +// Write plan to file +pkg.writeModulePlan({ includePackages: true }); + +// Add version tag +pkg.addTag('v1.0.0', 'latest-change', 'Initial release'); +``` + +## ๐Ÿ” Debugging Tips + +### 1. Context Issues +```typescript +const pkg = new LaunchQLPackage(cwd); +console.log('Context:', pkg.getContext()); +console.log('Workspace path:', pkg.getWorkspacePath()); +console.log('Module path:', pkg.getModulePath()); +``` + +### 2. Dependency Resolution +```typescript +// Check module map +const modules = pkg.getModuleMap(); +console.log('Available modules:', Object.keys(modules)); + +// Check specific module dependencies +const deps = pkg.getModuleDependencyChanges('my-module'); +console.log('Dependencies:', deps); +``` + +### 3. Migration Debugging +```typescript +const migrate = new LaunchQLMigrate(pgConfig, { hashMethod: 'content' }); + +// Use debug mode for detailed error information +await migrate.deploy({ + modulePath: '/path/to/module', + debug: true, // Shows full SQL scripts on errors + logOnly: true // Test without executing +}); +``` + +## ๐Ÿ“ Key Files to Understand + +1. **`src/core/class/launchql.ts`** - Main LaunchQLPackage class (1472 lines) +2. **`src/migrate/client.ts`** - LaunchQLMigrate class (661 lines) +3. **`src/resolution/deps.ts`** - Dependency resolution algorithms +4. **`src/files/plan/parser.ts`** - Plan file parsing +5. **`src/modules/modules.ts`** - Module discovery and management +6. **`src/migrate/sql/schema.sql`** - Migration schema definition +7. **`src/migrate/utils/transaction.ts`** - Transaction management utilities + +This guide covers the essential aspects of the core package. For specific implementation details, refer to the source files and their comprehensive inline documentation. diff --git a/packages/pgsql-test/AGENTS.md b/packages/pgsql-test/AGENTS.md new file mode 100644 index 000000000..91425e49e --- /dev/null +++ b/packages/pgsql-test/AGENTS.md @@ -0,0 +1,467 @@ +# LaunchQL pgsql-test Package - Agent Guide + +The `pgsql-test` package provides instant, isolated PostgreSQL databases for testing with automatic transaction rollbacks, context switching, and flexible seeding strategies. This guide helps agents understand the testing infrastructure. + +## ๐ŸŽฏ Overview + +**Main Purpose:** Isolated testing environments with per-test transaction rollbacks, ideal for integration tests, complex migrations, and RLS simulation. + +**Key Features:** +- โšก Instant test databases (UUID-named, seeded, isolated) +- ๐Ÿ”„ Per-test rollback (transaction/savepoint-based) +- ๐Ÿ›ก๏ธ RLS-friendly testing with role-based auth +- ๐ŸŒฑ Flexible seeding (SQL files, programmatic, CSV, JSON) +- ๐Ÿงช Compatible with any async test runner +- ๐Ÿงน Automatic teardown + +## ๐Ÿ—๏ธ Core Architecture + +### Main Entry Point +**Function:** `getConnections()` + +**Basic Usage:** +```typescript +import { getConnections } from 'pgsql-test'; + +let db, teardown; + +beforeAll(async () => { + ({ db, teardown } = await getConnections()); +}); + +afterAll(() => teardown()); +beforeEach(() => db.beforeEach()); +afterEach(() => db.afterEach()); +``` + +**Complete Object Destructuring:** +```typescript +const { pg, db, admin, teardown, manager } = await getConnections(); +``` + +**Returned Objects:** +- `pg` - `PgTestClient` connected as superuser (administrative setup) +- `db` - `PgTestClient` connected as app-level user (main testing) +- `admin` - `DbAdmin` utility for managing database state +- `teardown()` - Function to shut down test environment +- `manager` - Shared connection pool manager (`PgTestConnector`) + +## ๐ŸŽฎ PgTestClient API + +**Location:** `src/test-client.ts` +**Purpose:** Enhanced PostgreSQL client wrapper with test-specific features + +### Core Methods + +**Query Execution:** +- `query(sql, values?)` - Run raw SQL query, get `QueryResult` +- `any(sql, values?)` - Expect any number of rows +- `one(sql, values?)` - Expect exactly one row +- `oneOrNone(sql, values?)` - Expect zero or one row +- `many(sql, values?)` - Expect one or more rows +- `manyOrNone(sql, values?)` - Expect zero or more rows +- `none(sql, values?)` - Expect no rows +- `result(sql, values?)` - Get full QueryResult object + +**Test Isolation:** +- `beforeEach()` - Begin transaction and set savepoint +- `afterEach()` - Rollback to savepoint and commit outer transaction + +**Context Management:** +- `setContext({ key: value })` - Set PostgreSQL config variables for RLS testing + +### Context Switching for RLS Testing + +**Setting Authentication Context:** +```typescript +// Simulate authenticated user +await db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': '123', + 'jwt.claims.org_id': 'acme' +}); +``` + +**Role-Based Testing Pattern:** +```typescript +describe('authenticated role', () => { + beforeEach(async () => { + await db.setContext({ role: 'authenticated' }); + await db.beforeEach(); + }); + + afterEach(() => db.afterEach()); + + it('runs as authenticated user', async () => { + const res = await db.query(`SELECT current_setting('role', true) AS role`); + expect(res.rows[0].role).toBe('authenticated'); + }); +}); +``` + +## ๐ŸŒฑ Seeding System + +### Seeding Architecture +**Function Signature:** +```typescript +const { db, teardown } = await getConnections(connectionOptions?, seedAdapters?); +``` + +**Default Behavior:** If no `seedAdapters` provided, LaunchQL seeding is used automatically. + +### Available Seed Adapters + +#### 1. SQL File Seeding +**Purpose:** Execute raw `.sql` files from disk + +```typescript +import { getConnections, seed } from 'pgsql-test'; +import path from 'path'; + +const sql = (f: string) => path.join(__dirname, 'sql', f); + +const { db, teardown } = await getConnections({}, [ + seed.sqlfile([ + sql('schema.sql'), + sql('fixtures.sql') + ]) +]); +``` + +#### 2. Programmatic Seeding +**Purpose:** Run JavaScript/TypeScript logic for data insertion + +```typescript +const { db, teardown } = await getConnections({}, [ + seed.fn(async ({ pg }) => { + await pg.query(` + INSERT INTO users (name) VALUES ('Seeded User'); + `); + }) +]); +``` + +#### 3. CSV Seeding +**Purpose:** Load tabular data from CSV files + +```typescript +const csv = (file: string) => path.resolve(__dirname, '../csv', file); + +const { db, teardown } = await getConnections({}, [ + // Create schema first + seed.fn(async ({ pg }) => { + await pg.query(` + CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT NOT NULL); + CREATE TABLE posts (id SERIAL PRIMARY KEY, user_id INT REFERENCES users(id), content TEXT NOT NULL); + `); + }), + // Load from CSV (headers must match column names) + seed.csv({ + users: csv('users.csv'), + posts: csv('posts.csv') + }), + // Fix SERIAL sequences + seed.fn(async ({ pg }) => { + await pg.query(`SELECT setval(pg_get_serial_sequence('users', 'id'), (SELECT MAX(id) FROM users));`); + }) +]); +``` + +#### 4. JSON Seeding +**Purpose:** Use in-memory objects as seed data + +```typescript +const { db, teardown } = await getConnections({}, [ + seed.fn(async ({ pg }) => { + await pg.query(` + CREATE SCHEMA custom; + CREATE TABLE custom.users (id SERIAL PRIMARY KEY, name TEXT NOT NULL); + `); + }), + seed.json({ + 'custom.users': [ + { id: 1, name: 'Alice' }, + { id: 2, name: 'Bob' } + ] + }) +]); +``` + +#### 5. Sqitch Seeding +**Purpose:** Deploy Sqitch-compatible projects with LaunchQL's high-performance engine + +```typescript +const cwd = path.resolve(__dirname, '../path/to/sqitch'); + +const { db, teardown } = await getConnections({}, [ + seed.sqitch(cwd) // Uses LaunchQL's TypeScript engine (up to 10x faster) +]); +``` + +#### 6. LaunchQL Seeding (Default) +**Purpose:** Apply LaunchQL modules using `deployFast()` + +```typescript +// Zero configuration - uses current working directory +const { db, teardown } = await getConnections(); + +// Or specify custom path +const cwd = path.resolve(__dirname, '../path/to/launchql'); +const { db, teardown } = await getConnections({}, [ + seed.launchql(cwd) // Uses deployFast() - up to 10x faster than traditional Sqitch +]); +``` + +### Composable Seeding +**Multiple Strategies:** You can combine different seeding approaches: + +```typescript +const { db, teardown } = await getConnections({}, [ + seed.launchql('./my-module'), // Deploy LaunchQL module + seed.sqlfile(['./fixtures/data.sql']), // Add fixture data + seed.fn(async ({ pg }) => { // Programmatic setup + await pg.query(`SELECT setval('users_id_seq', 1000);`); + }), + seed.csv({ users: './test-data.csv' }) // Load test data +]); +``` + +## โš™๏ธ Configuration Options + +### Connection Options +**Interface:** `PgTestConnectionOptions` + +```typescript +interface PgTestConnectionOptions { + extensions?: string[]; // PostgreSQL extensions to include + cwd?: string; // Working directory for LaunchQL/Sqitch projects + // ... additional pg connection options +} +``` + +**Usage:** +```typescript +const { db, teardown } = await getConnections({ + extensions: ['uuid-ossp', 'postgis'], + cwd: '/path/to/project' +}); +``` + +### Database Connection Options +**Standard PostgreSQL options supported:** +- `host`, `port`, `database`, `user`, `password` +- `ssl`, `connectionTimeoutMillis`, `idleTimeoutMillis` +- All standard `pg` client options + +## ๐ŸŽฏ Common Testing Patterns + +### 1. Basic Integration Test +```typescript +import { getConnections } from 'pgsql-test'; + +let db, teardown; + +beforeAll(async () => { + ({ db, teardown } = await getConnections()); + + // Setup schema + await db.query(` + CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT); + INSERT INTO users (name) VALUES ('Alice'), ('Bob'); + `); +}); + +afterAll(() => teardown()); +beforeEach(() => db.beforeEach()); +afterEach(() => db.afterEach()); + +test('user count starts at 2', async () => { + const res = await db.query('SELECT COUNT(*) FROM users'); + expect(res.rows[0].count).toBe('2'); +}); + +test('can insert user', async () => { + await db.query(`INSERT INTO users (name) VALUES ('Charlie')`); + const res = await db.query('SELECT COUNT(*) FROM users'); + expect(res.rows[0].count).toBe('3'); // Isolated from previous test +}); +``` + +### 2. RLS Testing Pattern +```typescript +describe('Row Level Security', () => { + let db, teardown; + + beforeAll(async () => { + ({ db, teardown } = await getConnections()); + + await db.query(` + CREATE TABLE posts ( + id SERIAL PRIMARY KEY, + user_id INT, + content TEXT + ); + + ALTER TABLE posts ENABLE ROW LEVEL SECURITY; + + CREATE POLICY user_posts ON posts + FOR ALL TO authenticated + USING (user_id = current_setting('jwt.claims.user_id')::INT); + `); + }); + + afterAll(() => teardown()); + + describe('as user 1', () => { + beforeEach(async () => { + await db.setContext({ + role: 'authenticated', + 'jwt.claims.user_id': '1' + }); + await db.beforeEach(); + }); + + afterEach(() => db.afterEach()); + + it('can only see own posts', async () => { + await db.query(`INSERT INTO posts (user_id, content) VALUES (1, 'My post'), (2, 'Other post')`); + const res = await db.query('SELECT * FROM posts'); + expect(res.rows).toHaveLength(1); + expect(res.rows[0].content).toBe('My post'); + }); + }); +}); +``` + +### 3. LaunchQL Module Testing +```typescript +import { getConnections } from 'pgsql-test'; + +let db, teardown; + +beforeAll(async () => { + // Automatically deploys LaunchQL module from current directory + ({ db, teardown } = await getConnections()); +}); + +afterAll(() => teardown()); +beforeEach(() => db.beforeEach()); +afterEach(() => db.afterEach()); + +test('module functions work', async () => { + const result = await db.one('SELECT my_module_function($1) as result', ['test']); + expect(result.result).toBe('expected_value'); +}); +``` + +### 4. Multi-Database Testing +```typescript +let userDb, adminDb, teardown; + +beforeAll(async () => { + ({ db: userDb, pg: adminDb, teardown } = await getConnections()); +}); + +afterAll(() => teardown()); + +beforeEach(async () => { + await userDb.beforeEach(); + await adminDb.beforeEach(); +}); + +afterEach(async () => { + await userDb.afterEach(); + await adminDb.afterEach(); +}); + +test('admin can see all data', async () => { + // Setup as admin + await adminDb.query(`INSERT INTO sensitive_data (value) VALUES ('secret')`); + + // Test as regular user + await userDb.setContext({ role: 'authenticated' }); + const userResult = await userDb.query('SELECT * FROM sensitive_data'); + expect(userResult.rows).toHaveLength(0); + + // Test as admin + const adminResult = await adminDb.query('SELECT * FROM sensitive_data'); + expect(adminResult.rows).toHaveLength(1); +}); +``` + +## ๐Ÿ”ง Advanced Features + +### SeedAdapter Interface +**Location:** `src/seed/types.ts` + +```typescript +interface SeedAdapter { + seed(context: SeedContext): Promise; +} + +interface SeedContext { + pg: PgTestClient; // Superuser client + db: PgTestClient; // App-level client + admin: DbAdmin; // Database admin utilities +} +``` + +### Custom Seed Adapters +```typescript +const customSeed: SeedAdapter = { + async seed({ pg, db, admin }) { + // Custom seeding logic + await pg.query('CREATE EXTENSION IF NOT EXISTS "uuid-ossp"'); + await db.query('INSERT INTO custom_table VALUES (uuid_generate_v4())'); + } +}; + +const { db, teardown } = await getConnections({}, [customSeed]); +``` + +### DbAdmin Utilities +**Purpose:** Database administration and introspection + +**Key Methods:** +- Database state management +- Extension installation +- Role management +- Template operations + +## ๐ŸŽฏ Performance Considerations + +### LaunchQL vs Traditional Sqitch +**LaunchQL Advantages:** +- **10x faster** schema deployments via TypeScript engine +- **Sqitch compatibility** - keep existing migration syntax +- **Optimized for CI** - dramatically reduced test suite run times +- **Developer experience** - near-instant schema setup for tests + +### Test Isolation Strategy +**Transaction-based Isolation:** +- Each test runs in its own transaction +- `beforeEach()` starts transaction and sets savepoint +- `afterEach()` rolls back to savepoint +- No data persists between tests +- Fast and reliable isolation + +## ๐Ÿ“ Key Files to Understand + +1. **`src/test-client.ts`** - PgTestClient implementation +2. **`src/seed/types.ts`** - Seeding interfaces and types +3. **`src/seed/adapters/`** - Built-in seed adapter implementations +4. **`src/connections.ts`** - Main getConnections() function +5. **`src/admin.ts`** - DbAdmin utilities +6. **`__tests__/`** - Example usage patterns + +## ๐ŸŽฏ Agent Tips + +1. **Default Seeding:** `getConnections()` with no arguments uses LaunchQL seeding automatically +2. **Test Isolation:** Always use `beforeEach()/afterEach()` pattern for proper isolation +3. **RLS Testing:** Use `setContext()` to simulate different user roles and JWT claims +4. **Seeding Composition:** Combine multiple seed adapters for complex test scenarios +5. **Performance:** LaunchQL seeding is significantly faster than traditional Sqitch +6. **Error Handling:** Test database errors are isolated and don't affect other tests +7. **Connection Management:** `teardown()` handles all cleanup automatically + +This testing infrastructure is designed to make PostgreSQL integration testing fast, reliable, and easy to set up. The combination of transaction-based isolation and flexible seeding makes it ideal for testing complex database-driven applications.