icon	description	cover	coverY
1	다양한 종류의 데이터셋에 대한 소개와 이를 수집하는 방식을 알아봅니다!	https://images.unsplash.com/photo-1451187580459-43490279c0fa?crop=entropy&cs=srgb&fm=jpg&ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwxMHx8dGFidWxhciUyMGRhdGF8ZW58MHx8fHwxNzMwNjE5NTk1fDA&ixlib=rb-4.0.3&q=85	0

Dataset

Ⅰ. Dataset

하나의 완전한 데이터셋을 소개합니다.

AwesomeKorean Data
- https://github.com/songys/AwesomeKorean_Data
KorQuad : 한국어 Machine Reading Comprehension 데이터셋
- https://github.com/korquad/korquad.github.io
CounselGPT : 한국어 심리 상담 데이터셋
- https://github.com/MrBananaHuman/CounselGPT
Welfare-QA : 한국어 복지제도 관련 QA 데이터셋
- https://github.com/ash-hun/Welfare-QA
KLUE
- https://github.com/KLUE-benchmark/KLUE
Korpora
- https://github.com/ko-nlp/Korpora
국립국어원 사전
- https://github.com/spellcheck-ko/korean-dict-nikl
한국어 챗봇데이터
- https://github.com/songys/Chatbot_data
Korean HateSpeech Data
- https://github.com/kocohub/korean-hate-speech
Ko-Wiki-Data
- https://huggingface.co/datasets/maywell/ko_wikidata_QA
Kor-Wiki-Hop
- https://huggingface.co/datasets/KETI-AIR/kor_wiki_hop
Korean Human Judgement
- https://huggingface.co/datasets/HAERAE-HUB/Korean-Human-Judgements
Ko-Calibration
- https://huggingface.co/datasets/maywell/ko-calibration

주어진 문서에서 내용을 추출하거나 외부에서 데이터를 수집/크롤링하는 레퍼런스를 소개합니다.

MegaParse
- https://github.com/QuivrHQ/MegaParse
FireCrawl
- https://github.com/mendableai/firecrawl
Playwright
- https://github.com/microsoft/playwright
MinerU
- https://github.com/opendatalab/MinerU
PDFMathTranslate
- https://github.com/Byaidu/PDFMathTranslate
OpenParse : Easy PDF Chunking open-source library
- https://github.com/Filimoa/open-parse