Paper-Reproducibility Agent

What it does

From PDF to verdict, autonomously

The agent reads the paper, locates and clones the artifact, follows the repo's own instructions to build and run it, and produces a structured report.

📄

Just a PDF

A local path or a URL — it's downloaded and read. Title, abstract, and repo/archive links are pre-scanned to seed the run.

🔎

Finds the repo

Reads the URL printed in the paper first, then falls back to web & GitHub search when the paper doesn't name one.

⚙️

Builds & runs it

Follows the README / Dockerfile / scripts on the local machine or a remote SSH/GPU host, guarded by a dangerous-command safety filter.

✅

Verifies claims

In deep mode, derives an evaluation plan from the paper's claims and adjudicates each one against the execution evidence.

Two modes

Both find & clone the repo — they differ in how far they go

Every run ends with the same structured report: an overall verdict, the repository, a paper summary, the reproduction steps followed, blockers, and a full command log.

--mode reproduce

Reproduce

Finds & clones the paper's repository
Follows the repo's own instructions to install and run it
Reports whether the artifact builds, runs, and matches the paper

--mode reproduce-deep

Reproduce · deep

Everything reproduce does, plus…
Derives an evaluation plan from the paper's specific claims
Executes every step and gives a per-claim verdict

Possible verdicts: reproducible partially reproducible not reproducible inconclusive

How it works

Focused sub-agents around a reproduction loop

Three prep sub-agents run concurrently to seed the loop; the main loop executes the artifact; two review sub-agents refine the report and conservatively downgrade overstated verdicts.

Summarize

Read the paper; extract approach & key results.

Find repo

Locate the artifact from the paper, then search.

Inspect env

Read the repo; propose an install/run plan.

Reproduce loop

Clone, install, and run the artifact.

Report

Adjudicate claims; critique & finalize.

summarize_paper find_repository extract_claims inspect_env verify_claim ×N critique_report

Web dashboard

A control panel that launches & monitors runs

dashboard.py is a dependency-free (Python stdlib) local web app — pick a PDF, mode, model, and execution target, click Run, and watch the pipeline live.

equinox — decentralized scheduling.pdf

reproduce · deepseek-v4-flash · local machine

running

Summarize

Find repo

Inspect env

Reproduce loop

Report

26iterations

141tool calls

19shell cmds

7m 12selapsed

vm_run_command · source equinox_env/bin/activate && python scripts/04_run_simulation.py …

Live per-run status, a 5-stage tracker, metrics, a log tail, a Stop button (kills the whole process group), and md/json/pdf report links on completion. It reads the same config as the CLI and isolates outputs under repro_workspace/.

Quick start

Install, configure, run

Install & set a backend key

# install
pip install -r requirements.txt

# pick a backend (auto-detected)
export ANTHROPIC_API_KEY=sk-ant-...
# …or GEMINI_API_KEY / OPENAI_API_KEY

Reproduce a paper

# follow the repo's own instructions
python repro_cli.py run --pdf paper.pdf --mode reproduce

# also verify each claim
python repro_cli.py run --pdf paper.pdf --mode reproduce-deep

# or drive it from the browser
python dashboard.py   # http://127.0.0.1:8765

A URL works too: python repro_cli.py run --pdf https://arxiv.org/pdf/2401.01234.pdf --mode reproduce-deep

Configuration

Any LLM backend · local or remote execution

Config comes from environment variables (a local .env is auto-loaded) and, optionally, an ae_config.toml; environment variables always win.

Backend	Trigger	Default model
Gemini	`GEMINI_API_KEY`	`gemini-2.5-flash`
OpenAI-compatible	`OPENAI_API_KEY`	`gpt-4o`
Anthropic	otherwise	`claude-sonnet-4-6`

OpenAI-compatible endpoints — OpenAI, Azure, vLLM, Ollama, LM Studio, or any Chat-Completions API. Set OPENAI_BASE_URL and OPENAI_MODEL for a non-OpenAI endpoint (otherwise it defaults to api.openai.com + gpt-4o).

Execution runs on the local machine by default, guarded by a dangerous-command safety filter and a secret-stripping environment. Point --vm-host at an SSH host — or set VM_* in .env — to isolate execution on a dedicated (e.g. GPU) machine.