[codex] docs: clarify deduplication input discovery

[lbliii]

What changed

• document the Parquet and JSONL extension defaults shared by exact, fuzzy, semantic, and duplicate-removal workflows
• explain None, empty-list, and explicit extension override behavior
• document recursive discovery for a directory string and top-level discovery for listed directories
• add runnable JSONL-default and custom Parquet-suffix examples
• require matching discovery and partition settings between identification and removal when IDs are generated
• correct obsolete duplicate-removal argument names in the touched examples

Why

PR #2045 changed deduplication workflows to derive input_file_extensions from input_filetype, but the published Fern pages did not describe the defaults or override semantics. Users could also copy removal examples that referenced constructor arguments no longer present in TextDuplicatesRemovalWorkflow.

User impact

Users can now predict which files each workflow discovers, safely use custom suffixes such as .pq, and preserve generated-ID consistency between identification and removal.

Validation

• npm run check from fern/: 0 errors
• fern docs broken-links: no errors in changed pages; 22 existing errors remain in older API-reference pages
• git diff --check

Closes #2127
Parent tracking issue: #2118

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR clarifies how the exact, fuzzy, semantic, and duplicate-removal deduplication workflows discover input files, and corrects stale parameter names (input_id_field → id_field, ids_to_remove_duplicate_id_field → duplicate_id_field) in code examples across four documentation pages. All renamed parameters were verified against the actual TextDuplicatesRemovalWorkflow dataclass, and the documented recursive-vs-top-level discovery behaviour was confirmed against FilePartitioningStage._get_file_list_with_sizes.

• Adds a new Input File Discovery section to the deduplication index explaining None/[]/explicit override semantics, recursive vs. top-level scoping, and the single-filetype constraint.
• Corrects obsolete TextDuplicatesRemovalWorkflow constructor argument names in every touched example and adds input_file_extensions to the parameter tables for exact, fuzzy, and semantic workflows.
• Adds runnable JSONL-default and custom Parquet-suffix examples and cross-links each sub-page to the new discovery section.

Confidence Score: 5/5

Documentation-only change; all renamed parameters and discovery behaviour were verified against the live Python source.

Every parameter rename was confirmed correct against TextDuplicatesRemovalWorkflow and the recursive/top-level discovery semantics were confirmed against FilePartitioningStage. No executable code is modified.

index.mdx — the abbreviated removal snippet in the ID Field Configuration accordion omits duplicate_id_field, which defaults to id rather than _curator_dedup_id; worth a second look before merge.

Important Files Changed

Filename	Overview
fern/versions/main/pages/curate-text/process-data/deduplication/index.mdx	Adds the new Input File Discovery section with accurate semantics (verified against FilePartitioningStage); fixes duplicate_id_field rename in the common removal example; adds an accordion code snippet demonstrating custom extension override.
fern/versions/main/pages/curate-text/process-data/deduplication/exact.mdx	Renames input_id_field to id_field and ids_to_remove_duplicate_id_field to duplicate_id_field in all examples; adds input_file_extensions row to the parameter table and a cross-link to the new discovery section.
fern/versions/main/pages/curate-text/process-data/deduplication/fuzzy.mdx	Same parameter renames as exact.mdx; adds input_file_extensions to the table and a cross-link; adds a comment hint in the removal code block about matching extension overrides.
fern/versions/main/pages/curate-text/process-data/deduplication/semdedup.mdx	Adds input_path, input_filetype, and input_file_extensions rows to the parameter table and a discovery-rules note linking to the index; adds input_filetype=jsonl to the advanced configuration example.
fern/versions/main/pages/about/concepts/text/data-processing-concepts.mdx	Single-line fix: renames ids_to_remove_duplicate_id_field to duplicate_id_field in the fuzzy duplicate removal example.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[input_file_extensions provided?] -->|None or empty list| B[Use defaults from input_filetype]
    A -->|Non-empty list| C[Use provided extensions as-is]
    B --> D{input_filetype}
    D -->|parquet| E[Extension: .parquet]
    D -->|jsonl| F[Extensions: .jsonl and .json]
    E --> G[input_path type?]
    F --> G
    C --> G
    G -->|Single string path| H[Recursive discovery - all nested subdirectories]
    G -->|List of paths| I[Top-level only per entry - non-recursive]
    H --> J[Filter files by extension]
    I --> J
    J --> K[input_filetype selects reader - not extensions]
    K --> L[Read files with parquet or jsonl reader]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[input_file_extensions provided?] -->|None or empty list| B[Use defaults from input_filetype]
    A -->|Non-empty list| C[Use provided extensions as-is]
    B --> D{input_filetype}
    D -->|parquet| E[Extension: .parquet]
    D -->|jsonl| F[Extensions: .jsonl and .json]
    E --> G[input_path type?]
    F --> G
    C --> G
    G -->|Single string path| H[Recursive discovery - all nested subdirectories]
    G -->|List of paths| I[Top-level only per entry - non-recursive]
    H --> J[Filter files by extension]
    I --> J
    J --> K[input_filetype selects reader - not extensions]
    K --> L[Read files with parquet or jsonl reader]

Loading

_{Reviews (5): Last reviewed commit: "Merge branch 'main' into codex/docs-dedu..." | Re-trigger Greptile}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-codex-docs-dedup-input-defaults.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/about/concepts/text/data/processing
• nemo/curator/curate-text/process-data/deduplication/exact
• nemo/curator/curate-text/process-data/deduplication/fuzzy
• nemo/curator/curate-text/process-data/deduplication
• nemo/curator/curate-text/process-data/deduplication/semdedup

[lbliii]

/ok to test d9406a8