Skip to content

fix: preserve full document chunks - #32

Merged
suguanYang merged 1 commit into
stagingfrom
fix/wangbinqi/preserve-document-chunks
May 12, 2026
Merged

fix: preserve full document chunks#32
suguanYang merged 1 commit into
stagingfrom
fix/wangbinqi/preserve-document-chunks

Conversation

@suguanYang

@suguanYang suguanYang commented May 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • publish the full parsed chunk list to document/retrieval state during job finalization
  • remove the redundant duplicate media GC and overlap-stat collection path entirely
  • keep temp file ownership with the worker task workspace cleanup
  • update the parse contract so result ZIP, job_chunks, and document_chunks preserve all parsed chunks and no overlap metadata is written

Closes #31

Tests

  • uv run pytest apps/worker/tests/contract/test_parse_task_contract.py::test_should_export_full_result_when_publication_deduplicates_existing_chunks
  • uv run pytest apps/worker/tests/contract/test_parse_task_contract.py
  • uv run --all-packages --group lint ruff check apps/worker/app/core/tasks/kb_tasks.py packages/shared-python/shared/services/retrieval/publication_service.py packages/shared-python/shared/services/job_lifecycle_sync.py apps/worker/tests/contract/test_parse_task_contract.py
  • uv run --all-packages --group typecheck pyright --project pyproject.toml apps/worker/app packages/shared-python/shared

@suguanYang
suguanYang force-pushed the fix/wangbinqi/preserve-document-chunks branch from 955a678 to 29cc0ba Compare May 12, 2026 06:30
@suguanYang
suguanYang force-pushed the fix/wangbinqi/preserve-document-chunks branch from 29cc0ba to c46d08c Compare May 12, 2026 06:41
@suguanYang
suguanYang merged commit 77e5331 into staging May 12, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant