diff --git a/packages/core/DEPLOY.md b/packages/core/DEPLOY.md new file mode 100644 index 000000000..4d8a06e6a --- /dev/null +++ b/packages/core/DEPLOY.md @@ -0,0 +1,274 @@ +# LaunchQL Deployment System + +## Overview + +The LaunchQL deployment system provides a comprehensive solution for managing database schema changes across modules and projects. It supports both individual module deployments and recursive project-wide operations with sophisticated dependency resolution. + +## Current Behavior + +### Module-Level Deployment (`deployModule`) + +The `deployModule` function handles deployment of individual modules: + +- **Location**: `src/modules/deploy.ts` +- **Purpose**: Deploy database changes for a single module based on its plan file +- **Behavior**: + - Checks for existence of `launchql.plan` file + - Initializes `LaunchQLMigrate` client with database configuration + - Executes deployment with optional transaction control + - Supports deploying up to a specific change via `toChange` parameter + - Logs deployment results including deployed and skipped changes + +### Project-Level Deployment (`deployProject`) + +The `deployProject` function orchestrates deployment across multiple modules: + +- **Location**: `src/projects/deploy.ts` +- **Purpose**: Deploy all modules within a project in dependency order +- **Behavior**: + - Resolves module dependencies using `LaunchQLProject.getModuleExtensions()` + - Iterates through resolved extensions in topological order + - Handles external extensions via PostgreSQL `CREATE EXTENSION` + - Delegates to `deployModule` for local modules + - Supports both Sqitch compatibility mode and native LaunchQL migration + - Provides transaction control and change targeting + +### Orchestration-Level Deployment (`deployModules`) + +The `deployModules` function provides the highest-level deployment interface: + +- **Location**: `src/migrate/migration.ts` +- **Purpose**: Handle both recursive (project-level) and non-recursive (module-level) operations +- **Behavior**: + - Routes to `deployProject` when `recursive: true` + - Routes to `deployModule` when `recursive: false` + - Validates required parameters based on operation mode + - Provides unified interface for CLI and programmatic usage + +### Missing Capabilities Analysis + +#### Support recursive deploy via tag (across projects/modules) +- **Status**: Not implemented +- **Reason**: While individual modules and projects support tag-based deployment via `toChange`, there's no mechanism to recursively deploy across multiple projects using a single tag reference +- **Internal methods involved**: Would need `resolveTagToChangeName`, `resolveDependencies` with cross-project awareness +- **Implementation complexity**: **Non-trivial** - requires extending dependency resolution to work across project boundaries and coordinating tag resolution across multiple plan files + +## Tag Support Capabilities + +### Current Tag Support Architecture + +The LaunchQL system provides comprehensive tag support through several key components: + +#### Tag Definition and Storage +- **Plan Files**: Tags are defined in `launchql.plan` files with format: `%tag_name change_name` +- **Tag Structure**: Each tag maps to a specific change name within a project +- **Project Scoping**: Tags are scoped to individual projects by default + +#### Tag Resolution Function (`resolveTagToChangeName`) +Located in `src/resolution/resolve.ts` (lines 82-126), this function provides: + +**Supported Tag Formats:** +- **Simple Format**: `@tagName` - resolves within current project context +- **Qualified Format**: `project:@tagName` - explicitly specifies target project +- **Fallback Handling**: Non-tag references pass through unchanged + +**Resolution Process:** +1. **Format Detection**: Checks for `@` symbol to identify tag references +2. **Project Resolution**: Adds current project prefix for simple format tags +3. **Plan File Parsing**: Loads and parses target project's plan file +4. **Tag Lookup**: Searches plan file tags array for matching tag name +5. **Change Return**: Returns the change name that the tag points to + +#### Tag Integration Points + +**Module-Level Operations:** +- `deployModule()`: Accepts `toChange` parameter supporting tag references +- `revertModule()`: Accepts `toChange` parameter supporting tag references +- `verifyModule()`: Currently no tag support (missing feature) + +**Project-Level Operations:** +- `deployProject()`: Passes `toChange` through to individual modules +- `revertProject()`: Passes `toChange` through to individual modules +- `verifyProject()`: Currently no tag support (missing feature) + +**Database Operations:** +- `LaunchQLMigrate.deploy()`: Calls `resolveTagToChangeName` when `toChange` contains `@` +- `LaunchQLMigrate.revert()`: Calls `resolveTagToChangeName` when `toChange` contains `@` +- `LaunchQLMigrate.verify()`: No tag resolution integration + +#### Dependency Resolution Tag Modes + +The `resolveDependencies()` function supports three tag resolution modes: + +**'preserve' Mode (Default):** +- Keeps tag references as-is in dependency graph +- Tags remain unresolved until deployment execution +- Maintains original tag syntax in dependency tracking + +**'resolve' Mode:** +- Fully resolves tags to their target changes during dependency resolution +- Replaces tag references with actual change names +- Enables dependency analysis on resolved changes + +**'internal' Mode:** +- Resolves tags internally but preserves original references in output +- Maintains tag mappings for later resolution +- Balances resolution needs with tag preservation + +### Tag Support Limitations + +#### Cross-Project Tag Operations +- **Current State**: Tags work within single project scope +- **Limitation**: No mechanism for workspace-wide tag-based operations +- **Impact**: Cannot deploy/revert multiple projects to same tag simultaneously + +#### Verification Tag Support +- **Current State**: Verification functions don't support tag parameters +- **Limitation**: Cannot verify deployment state at specific tag +- **Impact**: No way to validate that database matches tagged state + +#### Tag Dependency Resolution +- **Current State**: Tags in dependency statements are resolved per project +- **Limitation**: Cross-project tag dependencies not fully supported +- **Impact**: Complex multi-project tag scenarios may not resolve correctly + +### Implementation Complexity for Missing Features + +#### Workspace-Wide Tag Operations +- **Complexity**: **Non-trivial** +- **Requirements**: + - Workspace-level tag coordination + - Cross-project dependency resolution + - Unified tag namespace or conflict resolution +- **Internal Methods**: Would extend `LaunchQLProject`, `resolveDependencies` + +#### Verification Tag Support +- **Complexity**: **Easy** +- **Requirements**: + - Add `toChange` parameter to verification functions + - Integrate `resolveTagToChangeName` in verification flow +- **Internal Methods**: Modify `verifyModule`, `verifyProject`, `verifyModules` + +#### Advanced Tag Features +- **Tag Inheritance**: **Non-trivial** - would need hierarchical tag resolution +- **Tag Validation**: **Easy** - could validate tag existence before operations +- **Tag History**: **Non-trivial** - would need tag change tracking over time + +#### Support cross-project dependency resolution +- **Status**: Partially supported +- **Reason**: `resolveDependencies` can handle external dependencies but doesn't recursively resolve them across project boundaries +- **Internal methods involved**: `resolveDependencies`, `LaunchQLProject.getModuleMap` +- **Implementation complexity**: **Non-trivial** - requires workspace-aware dependency resolution and cross-project plan file coordination + +#### Support workspace-wide recursive deployment +- **Status**: Not implemented +- **Reason**: No mechanism exists to deploy all projects in a workspace in dependency order +- **Internal methods involved**: Would need workspace-level orchestration, possibly extending `LaunchQLProject` +- **Implementation complexity**: **Easy** - could be implemented by extending existing project-level logic to workspace level + +#### Support parallel deployment of independent modules +- **Status**: Not implemented +- **Reason**: Current implementation processes modules sequentially +- **Internal methods involved**: Would need async coordination in `deployProject` and `deployModules` +- **Implementation complexity**: **Non-trivial** - requires careful handling of database connections and transaction coordination + +## Internal Methods + +### Core Deployment Functions +- `deployModule(config, database, cwd, options)` - Single module deployment +- `deployProject(opts, name, database, dir, options)` - Project-wide deployment +- `deployModules(options)` - Orchestration layer + +### Dependency Resolution +- `resolveDependencies(packageDir, extname, options)` - Main dependency resolver +- `LaunchQLProject.getModuleExtensions()` - Module dependency resolution +- `LaunchQLProject.getModuleMap()` - Module discovery and mapping + +### Tag Resolution +- `resolveTagToChangeName(planPath, tagReference, currentProject)` - Tag to change resolution +- Supports formats: `@tagName` and `project:@tagName` +- Integrates with `LaunchQLMigrate.deploy()` for tag-based deployments + +### Database Operations +- `LaunchQLMigrate.deploy(options)` - Core database deployment logic +- `LaunchQLMigrate.initializeSchema()` - Migration schema setup +- Plan file parsing and change tracking + +## Architecture Notes + +### Recursive Behavior +The system implements recursion at the project level: +1. **Project Discovery**: `LaunchQLProject` scans workspace for modules +2. **Dependency Resolution**: `resolveDependencies` builds dependency graph +3. **Topological Sort**: Dependencies processed in correct order +4. **Module Processing**: Each module deployed via `deployModule` + +### Transaction Control +- Module-level: Individual module deployments can be wrapped in transactions +- Project-level: Each module deployment is independently transacted +- No cross-module transaction support (by design for safety) + +### Tag Support +- Tags defined in `launchql.plan` files map to specific changes +- `resolveTagToChangeName` handles both simple (`@v1.0`) and qualified (`project:@v1.0`) formats +- Tag resolution occurs before deployment execution +- Limited to single-project scope currently + +## Recursive Behaviors + +### Multi-Module Operations via LaunchQLProject + +The `LaunchQLProject.getModuleExtensions()` method (lines 219-225 in `src/core/class/launchql.ts`) enables multi-module operations by: + +1. **Module Discovery**: Scans workspace for all available modules using `getModuleMap()` +2. **Dependency Resolution**: Calls `resolveDependencies()` to build complete dependency graph +3. **Extension Filtering**: Returns both resolved modules and external dependencies +4. **Topological Ordering**: Ensures modules are processed in correct dependency order + +### Dependency Resolution System + +The `resolveDependencies()` function (lines 223-533 in `src/resolution/deps.ts`) provides sophisticated dependency management: + +#### Tag Resolution Modes +- **'preserve'**: Keep tags as-is in dependency graph (default behavior) +- **'resolve'**: Fully resolve tags to their target changes before processing +- **'internal'**: Resolve tags internally but preserve original references in output + +#### Recursive Processing +1. **SQL File Scanning**: Parses all `deploy/*.sql` files for `-- requires:` statements +2. **Dependency Graph Building**: Creates complete dependency map with circular detection +3. **External Dependency Handling**: Identifies and tracks cross-project dependencies +4. **Topological Sort**: Orders modules to respect all dependencies + +#### Cross-Project Awareness +- Handles `project:module` syntax for external references +- Loads plan files from other projects when needed +- Maintains separation between internal and external dependencies +- Supports workspace-level module discovery via `LaunchQLProject` + +### Deployment Recursion Flow + +1. **Entry Point**: `deployModules()` with `recursive: true` +2. **Project Resolution**: Routes to `deployProject()` +3. **Module Discovery**: `LaunchQLProject.getModuleExtensions()` finds all modules +4. **Dependency Order**: Processes modules in topological order +5. **Individual Deployment**: Each module deployed via `deployModule()` +6. **External Extensions**: PostgreSQL extensions created via `CREATE EXTENSION` + +### Revert Recursion Flow (Reverse Order) + +1. **Entry Point**: `revertModules()` with `recursive: true` +2. **Project Resolution**: Routes to `revertProject()` +3. **Module Discovery**: Same as deployment - `LaunchQLProject.getModuleExtensions()` +4. **Reverse Order**: `[...extensions.resolved].reverse()` for safe rollback +5. **Individual Revert**: Each module reverted via `revertModule()` +6. **External Cleanup**: PostgreSQL extensions dropped with CASCADE + +### Verification Recursion Flow (Forward Order) + +1. **Entry Point**: `verifyModules()` with `recursive: true` +2. **Project Resolution**: Routes to `verifyProject()` +3. **Module Discovery**: Same dependency resolution as deployment +4. **Forward Order**: Processes in deployment order for consistency +5. **Individual Verification**: Each module verified via `verifyModule()` +6. **External Validation**: Checks extension availability via `pg_available_extensions` diff --git a/packages/core/ISSUES.md b/packages/core/ISSUES.md new file mode 100644 index 000000000..0ac2166d3 --- /dev/null +++ b/packages/core/ISSUES.md @@ -0,0 +1,56 @@ +# TODO and test cases to cover + +- [ ] Ensure the system rolls back to the last successful state (e.g. @v1.0.0) if a deployment fails +- [ ] Rollback on Failed Deployment to a Tagged State +- [ ] Test scenario where deployment fails and we need to revert from the last that was successful +- [ ] Rollback to a point and fork/modify a previously depended on change & plan, then redeploy with recursive +- [ ] Validate should also validate that it’s NOT any SQL errors in change files +- [ ] projects ref'ing each other multiple times — but not in a circular way — can't we just treat projects like files? + +## Revert and Rollback + +- [x] Support reverting individual modules +- [x] Support reverting individual projects +- [x] Support reverting via tag (using `toChange` parameter) +- [ ] Support recursive revert via tag (across projects/modules) +- [x] Support transaction control for reverts +- [x] Support reverse dependency order processing +- [x] Support external extension cleanup with CASCADE +- [ ] Check if extenral extensions cleanup doesn't impact cross-projects +- [x] Support Sqitch compatibility mode +- [x] Support change targeting (revert to specific change) +- [ ] Support cross-project dependency-aware revert +- [ ] Support workspace-wide recursive revert +- [ ] Support dry-run revert operations +- [ ] Support revert impact analysis + +## Deploy + +- [x] Support deploying individual modules +- [x] Support deploying individual projects +- [x] Support deploying via tag (using `toChange` parameter) +- [ ] Support recursive deploy via tag (across projects/modules) +- [x] Support transaction control for deployments +- [x] Support dependency resolution within projects +- [x] Support external extension management +- [x] Support Sqitch compatibility mode +- [x] Support change targeting (deploy up to specific change) +- [ ] Support cross-project dependency resolution +- [ ] Support workspace-wide recursive deployment +- [ ] Support parallel deployment of independent modules + +## Verify issues + +- [x] Support verifying individual modules +- [x] Support verifying individual projects +- [ ] Support verifying via tag (tag-based verification) +- [ ] Support recursive verify across project graph +- [x] Support external extension availability checking +- [x] Support Sqitch compatibility mode +- [x] Support change state validation +- [x] Support dependency order verification +- [ ] Support workspace-wide recursive verification +- [ ] Support verification reporting and summaries +- [ ] Support verification with change targeting +- [ ] Support verification impact analysis +- [ ] Support verification dry-run mode \ No newline at end of file diff --git a/packages/core/REVERT.md b/packages/core/REVERT.md new file mode 100644 index 000000000..5744ba41e --- /dev/null +++ b/packages/core/REVERT.md @@ -0,0 +1,269 @@ +# LaunchQL Revert System + +## Overview + +The LaunchQL revert system provides comprehensive functionality for safely rolling back database schema changes across modules and projects. It implements reverse dependency processing to ensure safe rollback operations and supports both individual module reverts and recursive project-wide operations. + +## Current Behavior + +### Module-Level Revert (`revertModule`) + +The `revertModule` function handles reversion of individual modules: + +- **Location**: `src/modules/revert.ts` +- **Purpose**: Revert database changes for a single module based on its plan file +- **Behavior**: + - Checks for existence of `launchql.plan` file + - Initializes `LaunchQLMigrate` client with database configuration + - Executes revert with optional transaction control + - Supports reverting to a specific change via `toChange` parameter (exclusive - the target change is NOT reverted) + - Logs revert results including reverted and skipped changes + - Mimics Sqitch revert behavior for compatibility + +### Project-Level Revert (`revertProject`) + +The `revertProject` function orchestrates reversion across multiple modules: + +- **Location**: `src/projects/revert.ts` +- **Purpose**: Revert all modules within a project in reverse dependency order +- **Behavior**: + - Resolves module dependencies using `LaunchQLProject.getModuleExtensions()` + - **Reverses the dependency order** for safe rollback (`[...extensions.resolved].reverse()`) + - Handles external extensions via PostgreSQL `DROP EXTENSION IF EXISTS ... CASCADE` + - Delegates to `revertModule` for local modules + - Supports both Sqitch compatibility mode and native LaunchQL migration + - Provides transaction control and change targeting + - Uses CASCADE for external extension drops to handle dependencies + +### Orchestration-Level Revert (`revertModules`) + +The `revertModules` function provides the highest-level revert interface: + +- **Location**: `src/migrate/migration.ts` +- **Purpose**: Handle both recursive (project-level) and non-recursive (module-level) operations +- **Behavior**: + - Routes to `revertProject` when `recursive: true` + - Routes to `revertModule` when `recursive: false` + - Validates required parameters based on operation mode + - Provides unified interface for CLI and programmatic usage + +### Missing Capabilities Analysis + +#### Support recursive revert via tag (across projects/modules) +- **Status**: Not implemented +- **Reason**: While individual modules and projects support tag-based revert via `toChange`, there's no mechanism to recursively revert across multiple projects using a single tag reference +- **Internal methods involved**: Would need `resolveTagToChangeName`, cross-project dependency resolution +- **Implementation complexity**: **Non-trivial** - requires coordinating tag resolution across multiple plan files and ensuring safe cross-project rollback order + +## Tag Support in Revert Operations + +### Current Tag Integration + +The revert system fully supports tag-based operations through the same infrastructure as deployment: + +#### Tag Resolution in Revert Context +- **Same Function**: Uses `resolveTagToChangeName()` identically to deployment +- **Format Support**: Supports both `@tagName` and `project:@tagName` formats +- **Resolution Timing**: Tag resolution occurs before reverse dependency processing +- **Exclusive Behavior**: `toChange` parameter is exclusive - the target change is NOT reverted + +#### Revert-Specific Tag Behavior + +**Module-Level Revert with Tags:** +```typescript +await revertModule(opts.pg, database, modulePath, { + useTransaction: options?.useTransaction, + toChange: options?.toChange // Can be tag reference like '@v1.0.0' +}); +``` + +**Project-Level Revert with Tags:** +- Tags resolved once at project level +- Same tag reference passed to all modules in project +- Each module reverts to the resolved change name +- Reverse dependency order maintained regardless of tag resolution + +#### Tag Resolution Modes in Revert + +The `resolveDependencies()` function's tag resolution modes affect revert operations: + +**'preserve' Mode:** +- Tags kept as-is during dependency resolution +- Resolution happens at execution time in `LaunchQLMigrate.revert()` +- Maintains flexibility for different tag interpretations per module + +**'resolve' Mode:** +- Tags resolved to change names during dependency analysis +- Enables dependency validation against resolved changes +- May be more efficient for complex dependency graphs + +**'internal' Mode:** +- Tags resolved internally but preserved in dependency tracking +- Balances resolution needs with tag reference preservation +- Useful for debugging and audit trails + +### Tag Safety in Reverse Operations + +#### Critical Considerations for Tag-Based Revert + +**Dependency Order Preservation:** +- Tag resolution must complete before reverse order processing +- `[...extensions.resolved].reverse()` applies after tag resolution +- Ensures safe rollback regardless of tag complexity + +**Cross-Module Tag Consistency:** +- Same tag reference should resolve consistently across modules +- Plan file changes between modules could cause inconsistent resolution +- Current implementation assumes stable plan files during operation + +**External Dependency Handling:** +- External dependencies dropped with CASCADE regardless of tags +- Tag resolution doesn't affect external extension cleanup +- External extensions don't have tag-based targeting + +### Missing Tag Features in Revert + +#### Cross-Project Tag Revert +- **Current Limitation**: Cannot revert multiple projects to same tag +- **Safety Concern**: Cross-project reverse order must be maintained +- **Complexity**: Requires workspace-level dependency analysis with tag resolution + +#### Tag Range Revert +- **Current Limitation**: Can only revert to single tag, not between tags +- **Use Case**: Revert changes between `@v1.0.0` and `@v2.0.0` +- **Implementation**: Would need tag range resolution and change filtering + +#### Tag Validation Before Revert +- **Current Limitation**: No pre-revert validation that tag exists/is reachable +- **Safety Concern**: Revert could fail mid-operation if tag is invalid +- **Implementation**: Could add tag validation phase before execution + +### Tag-Related Error Scenarios + +#### Tag Resolution Failures +- **Invalid Format**: Malformed tag references cause immediate failure +- **Missing Tag**: Tag not found in plan file causes operation failure +- **Plan File Issues**: Corrupted or missing plan files prevent tag resolution + +#### Partial Revert Scenarios +- **Module Failure**: If one module fails tag-based revert, others continue +- **Tag Inconsistency**: Different modules might resolve same tag differently +- **External Dependencies**: External extension cleanup proceeds regardless of tag issues + +#### Support cross-project dependency-aware revert +- **Status**: Not implemented +- **Reason**: Current revert operations are scoped to individual projects and don't consider dependencies between projects +- **Internal methods involved**: Would need workspace-level dependency resolution, extending `resolveDependencies` +- **Implementation complexity**: **Non-trivial** - requires understanding cross-project dependencies and coordinating rollback order across project boundaries + +#### Support workspace-wide recursive revert +- **Status**: Not implemented +- **Reason**: No mechanism exists to revert all projects in a workspace in reverse dependency order +- **Internal methods involved**: Would need workspace-level orchestration, possibly extending `LaunchQLProject` +- **Implementation complexity**: **Easy** - could be implemented by extending existing project-level logic to workspace level with proper reverse ordering + +#### Support dry-run revert operations +- **Status**: Not implemented +- **Reason**: No mechanism to preview what would be reverted without actually executing the revert +- **Internal methods involved**: Would need to extend `LaunchQLMigrate.revert()` with dry-run mode +- **Implementation complexity**: **Easy** - could be implemented by adding a flag to skip actual SQL execution while logging planned operations + +#### Support revert impact analysis +- **Status**: Not implemented +- **Reason**: No tooling to analyze the impact of a revert operation before execution +- **Internal methods involved**: Would need dependency analysis and change impact assessment +- **Implementation complexity**: **Non-trivial** - requires analyzing SQL changes and their potential impact on dependent modules + +## Internal Methods + +### Core Revert Functions +- `revertModule(config, database, cwd, options)` - Single module revert +- `revertProject(opts, name, database, dir, options)` - Project-wide revert +- `revertModules(options)` - Orchestration layer + +### Dependency Resolution (Reverse Order) +- `LaunchQLProject.getModuleExtensions()` - Module dependency resolution +- **Reverse processing**: `[...extensions.resolved].reverse()` ensures safe rollback order +- External extension handling with CASCADE cleanup + +### Tag Resolution +- `resolveTagToChangeName(planPath, tagReference, currentProject)` - Tag to change resolution +- Supports formats: `@tagName` and `project:@tagName` +- Integrates with `LaunchQLMigrate.revert()` for tag-based reverts + +### Database Operations +- `LaunchQLMigrate.revert(options)` - Core database revert logic +- External extension cleanup: `DROP EXTENSION IF EXISTS "${extension}" CASCADE` +- Plan file parsing and change tracking in reverse order + +## Architecture Notes + +### Reverse Dependency Processing +The revert system implements critical safety measures: +1. **Dependency Resolution**: Same as deployment - `resolveDependencies` builds dependency graph +2. **Reverse Order Processing**: `[...extensions.resolved].reverse()` ensures dependencies are removed before dependents +3. **CASCADE Cleanup**: External extensions dropped with CASCADE to handle dependent objects +4. **Module Processing**: Each module reverted via `revertModule` in reverse order + +### Transaction Control +- Module-level: Individual module reverts can be wrapped in transactions +- Project-level: Each module revert is independently transacted +- No cross-module transaction support (by design for safety and partial rollback capability) + +### Tag Support +- Tags defined in `launchql.plan` files map to specific changes +- `resolveTagToChangeName` handles both simple (`@v1.0`) and qualified (`project:@v1.0`) formats +- Tag resolution occurs before revert execution +- `toChange` parameter is exclusive - the specified change is NOT reverted +- Limited to single-project scope currently + +### Safety Mechanisms +- **Reverse dependency order**: Prevents orphaned dependencies +- **CASCADE cleanup**: Handles PostgreSQL extension dependencies +- **Transaction isolation**: Each module revert is atomic +- **Error handling**: Failed reverts don't affect subsequent modules +- **Confirmation support**: Sqitch mode supports confirmation prompts + +## Recursive Behaviors in Revert Operations + +### Critical Reverse Processing + +The revert system implements sophisticated reverse dependency processing to ensure safe rollbacks: + +#### Dependency Resolution (Same as Deploy) +- Uses `LaunchQLProject.getModuleExtensions()` to discover all modules +- Calls `resolveDependencies()` with same logic as deployment +- Builds complete dependency graph including external references + +#### Reverse Order Enforcement +```typescript +const reversedExtensions = [...extensions.resolved].reverse(); +``` + +This critical line ensures that: +1. **Dependents First**: Modules that depend on others are reverted first +2. **Dependencies Last**: Core dependencies are reverted after their dependents +3. **Safe Rollback**: Prevents orphaned database objects and constraint violations + +#### Tag Resolution in Reverse Context +- Same tag resolution modes as deployment: 'preserve', 'resolve', 'internal' +- `resolveTagToChangeName()` works identically for revert operations +- `toChange` parameter is exclusive - target change is NOT reverted +- Tag resolution occurs before reverse processing begins + +### Multi-Module Revert Flow + +1. **Dependency Discovery**: `LaunchQLProject.getModuleExtensions()` finds all modules +2. **Forward Resolution**: `resolveDependencies()` builds dependency graph in deployment order +3. **Reverse Processing**: Array is reversed to create safe rollback order +4. **External Extensions**: Dropped first with CASCADE to handle dependencies +5. **Local Modules**: Reverted in reverse dependency order +6. **Error Isolation**: Each module revert is independent - failures don't cascade + +### Cross-Project Considerations + +While current implementation is project-scoped, the architecture supports cross-project revert: +- External dependencies are tracked but not recursively reverted +- `resolveDependencies()` can load plan files from other projects +- Workspace-level revert would need to coordinate across project boundaries +- Reverse order must be maintained across entire workspace dependency graph diff --git a/packages/core/VERIFY.md b/packages/core/VERIFY.md new file mode 100644 index 000000000..b6f23a356 --- /dev/null +++ b/packages/core/VERIFY.md @@ -0,0 +1,336 @@ +# LaunchQL Verification System + +## Overview + +The LaunchQL verification system provides comprehensive functionality for validating the current state of deployed database schema changes across modules and projects. It ensures that deployed changes match their expected state and can detect inconsistencies or missing deployments without making any modifications to the database. + +## Current Behavior + +### Module-Level Verification (`verifyModule`) + +The `verifyModule` function handles verification of individual modules: + +- **Location**: `src/modules/verify.ts` +- **Purpose**: Verify database changes for a single module against its plan file +- **Behavior**: + - Checks for existence of `launchql.plan` file + - Initializes `LaunchQLMigrate` client with database configuration + - Executes verification without modifying the database + - Validates that all deployed changes match their expected state + - Throws error if any changes fail verification + - Logs number of successfully verified changes + - Mimics Sqitch verify behavior for compatibility + +### Project-Level Verification (`verifyProject`) + +The `verifyProject` function orchestrates verification across multiple modules: + +- **Location**: `src/projects/verify.ts` +- **Purpose**: Verify all modules within a project in dependency order +- **Behavior**: + - Resolves module dependencies using `LaunchQLProject.getModuleExtensions()` + - Processes modules in forward dependency order (same as deployment) + - Handles external extensions via PostgreSQL availability check (`pg_available_extensions`) + - Delegates to `verifyModule` for local modules + - Supports both Sqitch compatibility mode and native LaunchQL migration + - Uses query `SELECT 1/count(*) FROM pg_available_extensions WHERE name = $1` for external extension verification + - Continues verification even if individual modules fail (logs errors) + +### Orchestration-Level Verification (`verifyModules`) + +The `verifyModules` function provides the highest-level verification interface: + +- **Location**: `src/migrate/migration.ts` +- **Purpose**: Handle both recursive (project-level) and non-recursive (module-level) operations +- **Behavior**: + - Routes to `verifyProject` when `recursive: true` + - Routes to `verifyModule` when `recursive: false` + - Validates required parameters based on operation mode + - Provides unified interface for CLI and programmatic usage + +### Database Validation (`LaunchQLMigrate.verify`) + +The core database verification logic: + +- **Location**: `src/migrate/client.ts` (lines 281-337) +- **Purpose**: Perform actual database state validation +- **Behavior**: + - Queries migration tracking tables to get deployed changes + - Compares deployed state against plan file expectations + - Validates change checksums and dependencies + - Returns detailed verification results + - Does not modify database state during verification + +### Missing Capabilities Analysis + +#### Support verifying via tag (tag-based verification) +- **Status**: Not implemented +- **Reason**: While deployment and revert support tag-based operations via `toChange`, verification doesn't have equivalent tag-based targeting +- **Internal methods involved**: Would need `resolveTagToChangeName` integration with `verifyModule` and `verifyProject` +- **Implementation complexity**: **Easy** - could be implemented by adding `toChange` parameter to verification functions and integrating existing tag resolution + +## Tag Support in Verification Operations + +### Current Tag Support Status + +The verification system currently lacks tag support, representing a significant gap compared to deploy and revert operations: + +#### Missing Tag Integration Points + +**Module-Level Verification:** +- `verifyModule()` function has no `toChange` parameter +- Cannot verify deployment state up to a specific tag +- No integration with `resolveTagToChangeName()` function + +**Project-Level Verification:** +- `verifyProject()` function has no tag support +- Cannot verify project state at tagged releases +- No mechanism to pass tag references to individual modules + +**Database Verification:** +- `LaunchQLMigrate.verify()` has no tag resolution logic +- Verifies entire deployed state, not state at specific tag +- No filtering of changes based on tag boundaries + +### Tag Support Infrastructure Available + +Despite missing integration, the verification system could leverage existing tag infrastructure: + +#### Available Tag Resolution Components +- **`resolveTagToChangeName()`**: Fully functional tag resolution +- **Tag Formats**: Support for `@tagName` and `project:@tagName` already exists +- **Plan File Parsing**: Tag definitions already parsed from plan files +- **Dependency Resolution**: `resolveDependencies()` supports tag resolution modes + +#### Verification-Specific Tag Requirements + +**Tag-Based State Validation:** +- Verify only changes deployed up to specified tag +- Validate that tag-referenced changes are properly deployed +- Ensure no changes beyond tag are present in database + +**Tag Boundary Enforcement:** +- Filter verification scope to changes within tag boundary +- Validate tag exists and is reachable in deployment history +- Handle tag resolution errors gracefully + +### Implementation Path for Tag Support + +#### Easy Implementation Steps + +**1. Add `toChange` Parameter:** +```typescript +// Current signature +verifyModule(config, database, cwd, options?) + +// Proposed signature +verifyModule(config, database, cwd, options?: { toChange?: string }) +``` + +**2. Integrate Tag Resolution:** +```typescript +// In verifyModule function +const resolvedToChange = toChange && toChange.includes('@') + ? resolveTagToChangeName(planPath, toChange, project) + : toChange; +``` + +**3. Pass to Database Layer:** +```typescript +// Pass resolved tag to LaunchQLMigrate +await client.verify({ + project, + targetDatabase: database, + planPath, + toChange: resolvedToChange +}); +``` + +#### Database Layer Changes Required + +**Modify `LaunchQLMigrate.verify()`:** +- Add `toChange` parameter to `VerifyOptions` interface +- Filter verification scope to changes up to resolved tag +- Validate tag boundary in deployed changes + +**Change Filtering Logic:** +- Query deployed changes up to tag boundary +- Verify only changes within scope +- Report on changes beyond tag (if any) + +### Tag Verification Use Cases + +#### Release Validation +- **Use Case**: Verify production database matches tagged release +- **Command**: `launchql verify --to @v1.2.0` +- **Benefit**: Ensures deployment matches expected release state + +#### Rollback Validation +- **Use Case**: Verify database state after tag-based revert +- **Command**: `launchql verify --to @v1.1.0` (after reverting from v1.2.0) +- **Benefit**: Confirms revert operation completed successfully + +#### Development Validation +- **Use Case**: Verify development database matches feature branch tag +- **Command**: `launchql verify --to feature:@milestone-1` +- **Benefit**: Validates development environment consistency + +### Missing Advanced Tag Features + +#### Tag Range Verification +- **Feature**: Verify changes between two tags +- **Use Case**: Validate specific release changes +- **Implementation Complexity**: **Non-trivial** - requires tag range resolution + +#### Cross-Project Tag Verification +- **Feature**: Verify multiple projects at same tag +- **Use Case**: Validate workspace-wide release state +- **Implementation Complexity**: **Non-trivial** - requires cross-project coordination + +#### Tag History Verification +- **Feature**: Verify deployment history matches tag progression +- **Use Case**: Validate deployment audit trail +- **Implementation Complexity**: **Non-trivial** - requires historical state analysis + +#### Support recursive verify across project graph +- **Status**: Not implemented +- **Reason**: Current verification is scoped to individual projects and doesn't traverse cross-project dependencies +- **Internal methods involved**: Would need workspace-level dependency resolution, extending `resolveDependencies` +- **Implementation complexity**: **Non-trivial** - requires understanding cross-project dependencies and coordinating verification across project boundaries + +#### Support workspace-wide recursive verification +- **Status**: Not implemented +- **Reason**: No mechanism exists to verify all projects in a workspace in dependency order +- **Internal methods involved**: Would need workspace-level orchestration, possibly extending `LaunchQLProject` +- **Implementation complexity**: **Easy** - could be implemented by extending existing project-level logic to workspace level + +#### Support verification reporting and summaries +- **Status**: Partially supported +- **Reason**: Basic logging exists but no comprehensive reporting or summary generation +- **Internal methods involved**: Would need to extend verification result handling and add reporting utilities +- **Implementation complexity**: **Easy** - could be implemented by enhancing existing result logging and adding summary generation + +#### Support verification with change targeting +- **Status**: Not implemented +- **Reason**: No mechanism to verify only up to a specific change (unlike deploy/revert `toChange`) +- **Internal methods involved**: Would need to add `toChange` parameter support to verification functions +- **Implementation complexity**: **Easy** - could be implemented by adding change targeting logic similar to deploy/revert + +#### Support verification impact analysis +- **Status**: Not implemented +- **Reason**: No tooling to analyze what verification would check before execution +- **Internal methods involved**: Would need dependency analysis and change impact assessment +- **Implementation complexity**: **Non-trivial** - requires analyzing planned verification scope and potential issues + +#### Support verification dry-run mode +- **Status**: Unknown +- **Reason**: Verification is inherently read-only, but no explicit dry-run mode for preview +- **Internal methods involved**: Would need to extend verification logging to show what would be verified +- **Implementation complexity**: **Easy** - verification is already non-destructive, just needs enhanced preview logging + +## Internal Methods + +### Core Verification Functions +- `verifyModule(config, database, cwd, options)` - Single module verification +- `verifyProject(opts, name, database, dir, options)` - Project-wide verification +- `verifyModules(options)` - Orchestration layer + +### Dependency Resolution (Forward Order) +- `LaunchQLProject.getModuleExtensions()` - Module dependency resolution +- **Forward processing**: Uses same order as deployment for consistency +- External extension availability checking via `pg_available_extensions` + +### Database Validation +- `LaunchQLMigrate.verify(options)` - Core database verification logic +- Migration state queries and validation +- Change checksum verification +- Dependency validation + +### External Extension Verification +- Query: `SELECT 1/count(*) FROM pg_available_extensions WHERE name = $1` +- Validates that required external extensions are available +- Does not check if extensions are actually installed, only available + +## Architecture Notes + +### Forward Dependency Processing +The verification system processes modules in deployment order: +1. **Dependency Resolution**: Same as deployment - `resolveDependencies` builds dependency graph +2. **Forward Order Processing**: Processes modules in same order as deployment +3. **External Extension Checks**: Validates availability before checking local modules +4. **Module Processing**: Each module verified via `verifyModule` in dependency order + +### Read-Only Operations +- **No Database Modifications**: Verification never changes database state +- **State Validation**: Compares current state against expected state +- **Error Reporting**: Reports inconsistencies without attempting fixes +- **Continuation**: Verification continues even if individual modules fail + +### External Extension Handling +- **Availability Check**: Uses `pg_available_extensions` system view +- **Non-Installation Check**: Only verifies extensions are available, not installed +- **Error Handling**: Treats unavailable extensions as verification failures + +### Sqitch Compatibility +- **Plan File Support**: Works with standard `launchql.plan` files +- **Command Compatibility**: Supports Sqitch-style verification workflows +- **Exit Code Handling**: Proper exit code handling for CI/CD integration + +### Verification Scope +- **Module Level**: Verifies individual module state +- **Project Level**: Verifies all modules in a project +- **No Cross-Project**: Limited to single project scope currently +- **No Tag Support**: Cannot verify state at specific tags currently + +## Recursive Behaviors in Verification Operations + +### Forward Dependency Processing + +The verification system processes modules in the same order as deployment to maintain consistency: + +#### Dependency Resolution (Same as Deploy/Revert) +- Uses `LaunchQLProject.getModuleExtensions()` for module discovery +- Calls `resolveDependencies()` with identical logic to deployment +- Builds complete dependency graph including external references +- Maintains forward processing order (no reversal like revert) + +#### Tag Resolution Support (Currently Limited) +- Same `resolveDependencies()` infrastructure supports tag resolution modes +- Could support 'preserve', 'resolve', 'internal' modes like deploy/revert +- Currently no `toChange` parameter support in verification functions +- Tag resolution would occur before verification execution + +### Multi-Module Verification Flow + +1. **Dependency Discovery**: `LaunchQLProject.getModuleExtensions()` finds all modules +2. **Forward Resolution**: `resolveDependencies()` builds dependency graph +3. **Forward Processing**: Modules verified in deployment order +4. **External Extensions**: Availability checked via `pg_available_extensions` +5. **Local Modules**: Each verified via `verifyModule()` in dependency order +6. **Continuation**: Verification continues even if individual modules fail + +### Database State Validation + +The verification system uses `LaunchQLMigrate.verify()` (lines 281-337 in `src/migrate/client.ts`) for: +- **Migration State Queries**: Checks deployed changes against tracking tables +- **Checksum Validation**: Ensures deployed changes match expected checksums +- **Dependency Validation**: Verifies all required dependencies are present +- **Read-Only Operations**: Never modifies database state during verification + +### Cross-Project Verification Potential + +While currently project-scoped, the architecture could support cross-project verification: +- `resolveDependencies()` can load plan files from other projects +- External dependencies are tracked but not recursively verified +- Workspace-level verification would need to coordinate across project boundaries +- Forward order must be maintained across entire workspace dependency graph + +### Verification vs Deploy/Revert Recursion + +| Aspect | Deploy | Revert | Verify | +|--------|--------|--------|--------| +| **Order** | Forward (dependencies first) | Reverse (dependents first) | Forward (same as deploy) | +| **External Handling** | CREATE EXTENSION | DROP EXTENSION CASCADE | Check availability | +| **Tag Support** | Full support via `toChange` | Full support via `toChange` | Not implemented | +| **Cross-Project** | Limited to external refs | Limited to external refs | Limited to external refs | +| **Database Changes** | Modifies state | Modifies state | Read-only validation |