Set up a workspace and inspect a repository

The environment is part of the experiment. A wrong working directory, an old dependency, or a hidden credential can change a run as much as a prompt. Keep the tested repository separate from the grader and from private evidence.

A concrete first run

Download the starter kit from the site. Extract it. Open that directory in Codex, Claude Code, or Cursor, then open a terminal there. Confirm Python 3.12 or later is available. These commands use no model API:

python3 --version
python3 -m unittest discover -s tests -v
python3 runner.py --suite suites.json --responses demo-responses.json --output report.json

The demo responses are deliberately imperfect fixtures. Their scores test the apparatus, not a model. Read the generated per-case reasons. Do not replace the fixtures with the answer key and call that a benchmark result.

A repository reconnaissance sequence

pwd
git status --short
git log -5 --oneline
rg --files
rg 'def |class ' --glob '*.py'

If rg is unavailable, use your editor's file search. In a new folder without Git history, initialize a repository only after confirming that it is the intended folder. Read any AGENTS.md or equivalent instructions. Read the README, dependency declaration, test configuration, and relevant source files before editing.

Ask the agent to map entry points, data flow, persistent state, and tests. Treat that map as a hypothesis. Open the named files and verify at least one path from user input to output. Otherwise, you may be evaluating an explanation of the code rather than the code.

Version control without memorizing everything

A commit records a coherent revision. A branch gives a sequence of commits a name. A diff shows changes relative to a chosen revision. A worktree gives a branch its own directory. Know what these mean before allowing an agent to switch branches or discard files.

git diff --stat
git diff -- path/to/file.py
git status --short

Replace the example path with a file you have actually located. Review both the diff and new untracked files. A diff of tracked files does not reveal every new artifact. Never use destructive reset or cleanup commands merely because the agent says the workspace is messy.

Environment manifest

Record repository revision, operating system, Python version, dependency lockfile, model identifier, prompt version, tool configuration, and network policy. Record unavailable information as unknown. A random seed is not a complete reproducibility guarantee for remote inference.

Exercise: ask the agent to produce a setup report containing the commands it actually ran, their exit statuses, and unresolved dependencies. Reject a report that says “tests pass” without naming the test command and the collected tests.