Skip to content

Add CC Index URL list tutorial - #6

Draft
VibhuJawa wants to merge 5 commits into
mainfrom
vjawa/cc-index-url-list-tutorial
Draft

Add CC Index URL list tutorial#6
VibhuJawa wants to merge 5 commits into
mainfrom
vjawa/cc-index-url-list-tutorial

Conversation

@VibhuJawa

@VibhuJawa VibhuJawa commented Jun 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a YAML-driven tutorial for building a global unique URL list from selected Common Crawl Index snapshots on PBSS
  • run Curator ExactDeduplicationWorkflow(text_field="url", assign_id=True) directly on the configured CC Index parquet inputs
  • run TextDuplicatesRemovalWorkflow(input_fields=["url", "warc_filename"], output_fields=["url", "warc_filename"]) on the same inputs to write global_unique_urls/
  • link the tutorial from the text tutorials index

Validation

  • git diff --check
  • uv run --group linting ruff check tutorials/text/cc-index-url-list/create_cc_index_url_list.py
  • python -m py_compile tutorials/text/cc-index-url-list/create_cc_index_url_list.py
  • PBSS dry-run/listing smoke with --max-files-per-crawl 1 --dry-run

Notes

  • The full PBSS exact-dedup run is documented but not executed here because this login node has no usable NVIDIA driver and no cudf install.
  • included_crawls is authoritative; excluded crawls are omitted from the YAML rather than represented as a separate config field.

@VibhuJawa
VibhuJawa force-pushed the vjawa/cc-index-url-list-tutorial branch from 5f8bb30 to c65b119 Compare June 29, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant