Folder Structure

grader will need to store assignments and configuration in a sane way. This will allow grader and humans to skim through contents of submissions and results.

Each course (or section of a course) will have its own complete grader repository. That's right. For each course, you will need to clone down grader. This way, one can modify grader's source code (if it isn't general enough) without impacting any other sections (in this imaginary setup where everyone's using the same box for grading).

An Example Setup

Let's consider what the folder hierarchy might look like for CS 1570. Let's assume we cloned down grader some time ago, and named our root folder cs-1570-a for section a.

cs-1570-a/
    assignments/
        assignment1/
            config/
                assignment.yml
                Dockerfile
                grade-script/
                    grade.py
            results/
                student1/
                    student1.json
                    student1.md
                    student1.pdf
                student2/
                    student1.json
                    student1.md
                    student1.pdf
                ...                    
            submissions/
                student1/
                    main.cpp
                student2/
                    main.cpp
                ...
    bin/
        grader.py
    buildout.cfg
    grader.yml
    src/
        here-be-snakes.txt
    var/
        here-be-dragons.txt

I know what you're thinking. "That is intuitive and requires no explanation." Well, I took the time to type it, so YOU'RE GONNA GET AN EXPLANATION OK OK.

Buildout Folders

When buildout sets things up, it will drop executable binaries in the bin/ folder, and it will store all of its required libraries (eggs) in var/. We can basically disregard var/ completely. It is too scary in there. src/ will contain all the grader source modules, with bin/grader.py being the main interface and star of the show.

buildout.cfg will only be modified when we need to change package versions and the like.

grader.yml

grader will have some default config file that specifies defaults for the course. Refer to Grading Config Files for more details.

Assignments Folder

This folder houses everything relevant to an assignment. It contains one subfolder for each assignment.

Each assignment subfolder contains three folders: config, results, submissions.

Config

The config folder is, itself, a git repository. How bout that? It contains assignment-specific configuration (assignment.yml) to help grader properly run docker containers for the given assignment. It also contains a Dockerfile which is used by grader to construct a docker image for the assignment. Scripts/programs that actually grade the assignment are included in config/ as well. In the example above, grade-script/grade.py is the script to grade assignment1, but assignment.yml can specify what actually needs to be run.

It would make sense for Dockerfile to ADD grade-script/ to the image for assignment1, but that's up to the human grader to decide how they would like to handle things.

Details about configuration can be found on the Grading Config Files page.

By keeping config/ under version control, graders and instructors can work together to develop the meat of grader: its automated grading ability. It's also a nice sanity check if one is developing unit tests for grading and would like to hunt down bugs.

Submissions

Each student who submits an assignment will have a folder that contains their packed/unpacked submission. By leaving them unpacked (assuming assignments are all git repos), grader can automatically check out commits in submissions by hash or time stamp (with some effort).

Results

This is the directory for storing grading results. Raw data from the grading script, as well as rendered results can be found here. More on rendering results can be found on the Rendering Grade Results page.