Files

.github
config
dist
flagdata
- all2txt
- analysis
- cleaner
- data_gen
- data_operator
  - formatter
  - pruner
  - samplefilter
  - Operator.md
  - Operator_ZH.md
  - __init__.py
  - base_operator.py
  - make_data.py
  - new_data.csv
  - new_data.json
  - new_data.parquet
  - new_data.tsv
  - test.py
  - test_data.csv
  - test_data.json
  - test_data.parquet
  - test_data.tsv
- deduplication
- language_identification
- quality_assessment
- __init__.py
pic
quickstart
tests
.gitignore
FlagOpen.png
LICENSE
README.md
README_zh.md
contact_me.png
dedup.png
flagdata_logo.png
pipeline.png
pipeline_zh.png
pyproject.toml
requirements.txt

test_data.parquet

wuchengwei

Jun 13, 2024

84a3d7e · Jun 13, 2024