Skip to content

bug(memory-migration): residual Source check crashes on payloads stored before the source envelope #1934

Closed

Description

@happy-v587

Describe the bug

atomic-memory-migrate --action apply aborts with KeyError: 'value' when the database contains content Sources that were stored before the powercontext-source-v1 payload envelope existed.

residual_issues() narrows candidate rows with a free-text match, CAST(payload AS CHAR) LIKE '%memory%', and then unconditionally reads json.loads(payload)["value"]. Sources written by an older release store the source value directly ({"content": ..., "description": ..., "materialization": ..., "metadata": ..., "name": ...}) and have no value key, so any captured user prompt or Source whose text merely contains the substring memory crashes the migration.

The runtime read path already tolerates both layouts through _decode_legacy_payload(), so the offline checks are the only places that assumed the envelope unconditionally.

Correction after review: the conversion path is affected too. _work_sources() and _dream_inputs_change() read the same field, so a legacy structured Work Source aborts the conversion step or makes an unfinished Dream run look unaffected. _work_sources() narrows with LIKE '%"memory%', which does match an escaped \"kind\": \"memory\" in a stored payload, so the earlier claim that only the residual check is affected was wrong.

Steps to reproduce

  1. Run an older PowerContext release with prompt capture enabled and let it store content Sources (for example opencode-user-prompt:<hash>) whose text contains the word memory.
  2. Upgrade to a build that includes feat(memory): implement independent atomic memory (feat(memory): implement independent atomic memory #1857) and start the Server against that database.
  3. Startup fails with AtomicMemoryMigrationError reporting that legacy Memory collections remain in public Artifact tables.
  4. Back up the database, stop all writers and run powercontext server atomic-memory-migrate --action apply --env-file .env --maintenance-confirmed.
  5. Apply imports every entry and then aborts before removing any collection:
File "src/powercontext/builtin/persistence/migrations/atomic_memory_v1.py", line 1307, in apply_atomic_memory_migration
    report = await _verify_atomic_memory_migration(
File "src/powercontext/builtin/persistence/migrations/atomic_memory_v1.py", line 1380, in _verify_atomic_memory_migration
    errors.extend(await residual_issues(connection, tables, thorough=thorough, collections_removed=collections_removed))
File "src/powercontext/builtin/persistence/migrations/atomic_memory_references.py", line 1298, in residual_issues
    value = json.loads(bytes(row["payload"]))["value"]
KeyError: 'value'

The offline task is documented as resumable after an interruption, and it is, but every rerun fails at the same pre-removal acceptance step, so the deployment cannot complete the migration or start the new Server.

Expected behavior

residual_issues() judges parsed fields rather than assuming a payload shape, and does so for both stored layouts. A Source without the stored envelope is read through its top-level object, so it is either converted like any other Work Source or judged and reported; a Source that holds no Work payload still cannot hold a legacy Memory citation. Apply then completes, the collections leave the public tables, and verify returns ready: true.

Actual behavior

Apply aborts with KeyError: 'value' after importing all entries. The legacy collections stay in the public Artifact tables, so Server startup keeps failing the residual check.

Environment

Are you willing to submit a PR to fix this bug?

  • Yes, I would like to submit a PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions