How is the repository built?

• external_sources folder:

  • contains all external datasets we had utilized for training
  • prior to running scripts, download all relevant datasets to external_sources by running the .sh scripts

• format.py:

  • unified dataset creation from various sources we had implemented
  • datasets are formatted according to global format we had optimized for training (after removing columns that hindered training, etc.)
  • some datasets are automatically downloaded from Kaggle source (toggleable via commandline argument when running script)
  • allows to build dynamic dataset by sampling from each source by percentages, see sample_by_percentages()

• fetch_data.py:

  • is utilized by format.py to download Kaggle datasets and parse external_sources folder tree

• finetuning.py:

  • performs finetuning by creating dataset using format.py, saving dataset to pickle file in ./data folder, and then finetuning on that dataset
  • final model is saved to ./models, naming scheme consists of:
    • Hugging Face name of base model, e.g. "distilbert-base-uncased"
    • versioning of dataset, which can be modified in a global argument inside format.py whenever dataset makeup is modified
    • size of finetuning dataset (e.g 5,000 examples)
    • sources used to make up the dataset

🧠 A note about format.py

Some of the datasets we had considered for this task only had LLM generated or human generated essays, without the actual prompt given to the subject ("Write an essay about..."). However, we had found by playing around with small training datasets that training is much better when the input does consist of the prompt given to the subject (and also, that's how the Kaggle dataset is built...) We thus put some work into implementing logic that given an essay with no prompt, generates the prompt via LLM.

This happens whenever you run format_all_datasets in format.py, during which add_prompts() is run. If add_prompts encounters a dataframe in which the prompt_text column is empty, it fills it in via a flan-t5 huggingface pipeline.

We had chosen not to go forward with this direction because it hindered performance, but you may try out this pipeline with testing_automatic_promptname_generation.py.

🧠 How to use finetuning.py?

This script fine-tunes a transformer-based classifier to distinguish between essays written by humans and those generated by language models. It supports dataset preparation, saving/loading, fine-tuning, and inference.

🚀 How to Use (run from command line or interpreter like so)

python finetuning.py \
    --sources outfox, fpe, daigt, persuade \ 
    --base_model distilbert-base-uncased \
    --save_dataset \
    --sample_size 10000
   

Note that you can use any subset of the sources, and that the base_model is distilbert by default (=if you don't enter this parameter). distilbert can be changed to any HuggingFace model name but probably better to leave it like that now and just play around with the dataset makeup. sample_size controls the size of the dataset we finetune on.

🧩 Arguments

Argument	Type	Description
--sources	list[str]	One or more .pkl files that contain lists of generated and human essays.
--base_model	str	Hugging Face model name to fine-tune (default: distilbert-base-uncased).
--save_dataset	flag	If set, saves the merged training data as a .pickle file
--load_dataset_from_path	str	Load a pre-merged dataset instead of merging from sources. Need to pass path.
--path_to_model	str	Use a previously fine-tuned model for inference.
--sample_size	int	Number of samples to include (default: 10000).

---

---

🛠 Requirements

• Python 3.8+
• pandas
• transformers
• scikit-learn
• argparse

Note: Your script must implement or import the following functions:

• merge_data_for_finetuning(...)
• write_classifier_format(...)
• finetune(...)
• pull_kaggle_test_set(...)
• inference(...)

---

📁 Output Structure

• Training dataset saved to:
  ./data/training_data_version_<VERSION>_size_<SIZE>_sources_<SOURCE-NAMES>.pickle

• Fine-tuned model saved to:
  ./models/modelname_<BASE_MODEL>_version_<VERSION>_size_<SIZE>_sources_<SOURCE-NAMES>

• Inference test set saved to:
  ./data/test_data_version_<VERSION>_size_<SIZE>_sources_<SOURCE-NAMES>

• Test results (average loss) printed to logs.

Note that:

• Right now the parameter TRAIN_GENERATED_PERCENTAGE is set to 0.5 as per check we did with hacking.
• The script formats all the data sources you asked for into one dataset, and then samples randomly essays out of it such that 50-50 are generated-human.
• To use the finetuned model on kaggle, you need to upload to kaggle as a model the entire folder that has been saved under 'models', e.g. 'modelname_distilbert-base-uncased_version_v17-01-2025_size_1000_sources_fpe-persuade'
• If you want to do finetuning ON kaggle you would need to upload all the dependency .py that the code uses:
  • 'format.py'
  • all the folder 'external_sources' (which is all the data that format.py formats)
• so might be better to just do the data generation part locally, and then copy what was generated in your 'data' folder to kaggle.
• Then you could set args.load_dataset_from_path to the path where you put the dataset, and then all kaggle will do will be just finetuning!