Big Pickle on SWE Atlas – Codebase QnA

Big Pickle on SWE Atlas — Codebase QnA Task Resolve Rate: 50.8% (63/124) — big-pickle, the free stealth model on OpenCode Zen, evaluated on Scale AI's SWE Atlas Codebase QnA benchmark using the mini-swe-agent scaffold. Run on 2026-08-11 with the official open-source harness, task data, and judge model. Result in context Against the official SWE Atlas QnA leaderboard (updated 2026-07-28): Within the Mini-SWE-Agent scaffold class — the apples-to-apples comparison — this run outscores every entry on the official leaderboard, and it also tops the Codex-scaffold GPT entries. Only the two Claude models running on their native Claude Code scaffold score higher. Note the caveats below before treating this as a leaderboard-equivalent number. By language By category Method Everything follows Scale's published protocol as closely as budget allowed: Tasks: all 124 Codebase QnA tasks from scaleapi/SWE-Atlas (Apache-2.0), unmodified — including Scale's shipped mswea_qa_config.yaml agent configuration (system/instance templates, step_limit: 250). Harness: Harbor v0.18.0 with Modal sandboxes, per the SWE-Atlas README. Scaffold: mini-swe-agent pinned to 2.4.6 — the same minimal bash-only scaffold Scale uses for non-first-party models on the leaderboard. Model: big-pickle via OpenCode Zen's OpenAI-compatible endpoint (https://opencode.ai/zen/v1), litellm route openai/big-pickle. Total consumption: 674M input / 4.3M output tokens, at $0 (the model is free during its stealth period). Judge: claude-opus-4-5-20251101 — the exact judge model Scale specifies — accessed through Anthropic's OpenAI-compatible endpoint (https://api.anthropic.com/v1) with EVAL_MODEL overridden to the bare Anthropic model ID. Scoring: the benchmark's own rubric-based verifier, unmodified. A task resolves only if every scored must-have rubric passes. Caveats Read these before quoting the number: Single trial per task (-k 1). The official protocol runs 3 trials and reports the mean. At n=124, the single-trial standard error is ≈ ±4.5 points — comparable to the leaderboard's own reported error bars (±5). Reduced sandbox resources. Tasks declare 16 CPU / 16 GB; this run used 4 CPU / 8 GB to fit a personal budget. Slower command execution can only depress an agent's score (via command timeouts or OOM kills), not inflate it. Empirically it appears to have had no effect here: a scan of all 124 agent trajectories found zero command timeouts and zero exit-137 kills — no command ever hit the 900s ceiling or the memory limit. Self-reported. Scale did not run or verify this evaluation. The full per-task verifier logs in this repo allow independent auditing, and the run is reproducible from the configs here plus the public SWE-Atlas repo. Model identity unknown. big-pickle is officially unconfirmed; leaked provider errors and API response signatures suggest it is currently served by DeepSeek infrastructure. The underlying model may change without notice, so this result is a snapshot of whatever was behind the alias on 2026-08-11. Data exposure. OpenCode states that prompts to big-pickle during its free period may be used to improve the model. The benchmark's task content (already public, canary-marked by Scale) was necessarily sent to that endpoint. Two resolved tasks had unscored rubrics. On task-...ba9ad (5 of 11 rubrics) and task-...baa1d (1 rubric), the judge returned unparseable output through all 8 retries; the benchmark's verifier excludes unscored rubrics from the pass computation by design. Treating unscored-as-fail instead gives a strict-lower-bound of 61/124 = 49.2% — still above every Mini-SWE-Agent leaderboard entry. All verifier logs are included so you can apply either convention. Reproducing git clone https://github.com/scaleapi/SWE-Atlas && cd SWE-Atlas git clone --branch v0.18.0 --depth 1 https://github.com/laude-institute/harbor.git uv tool install ./harbor --with modal && uv tool install modal && modal setup # from this repo: copy run_config/qa, run_config/tw, run_config/rf into # SWE-Atlas/run_config/ (preserving the subdirectories — the scripts resolve # .env and Scale's mswea_*_config.yaml relative to their own location), # copy preflight.sh and .env.example into the SWE-Atlas root, # create .env from .env.example, then: ./preflight.sh bash run_config/qa/big-pickle_smoke.sh # 3-task smoke test first bash run_config/qa/big-pickle_miniswe.sh # full 124-task run Hard-won gotchas the configs already handle: Do not pass --ak reasoning_effort with an openai/-prefixed model — Harbor silently switches mini-swe-agent to the OpenAI Responses API, which chat-completions-only endpoints like Zen don't serve. Keep agent and judge credentials separate. The judge reads host OPENAI_API_KEY/OPENAI_API_BASE (via each task's [verifier.env]); the agent's Zen credentials go through --ae per-agent overrides. Pass secrets to --ae as ${VAR} templates, not literals. Harbor redacts literal secrets to **** when persisting job state, which breaks harbor job resume with instant 401s. Templates round-trip and re-resolve from the host env. Expect a few % of trials to die to Modal Failed to read exec stdio stream errors; harbor job resume -f ... re-runs them cleanly. Approximate cost for the full QnA run: ~$70 of Modal compute (at reduced sandbox resources; roughly 2–3× that at the declared 16 CPU/16 GB), ~$25 of Anthropic API for judging, $0 for the model. Repo contents results/per_task_results.csv — task ID, category, language, resolved, aggregate rubric score, rubrics passed/total results/summary.json — headline numbers and breakdowns results/verifier_logs/ — the judge's full per-rubric output for every task (audit trail). Notes like (flipped from raw=0) are the benchmark's own shipped verifier logic (evaluate_answer.py inverts rubrics marked negative-polarity), not post-hoc re-scoring. run_config/ — the exact Harbor run scripts used (QnA smoke + full, plus untested Test Writing / Refactoring variants) preflight.sh — endpoint/auth checks for both the model and the judge Attribution SWE Atlas benchmark © Scale AI, Apache-2.0 — paper: arXiv:2605.08366. Per the authors' request, please treat SWE Atlas as a held-out signal of progress rather than a training target. Harbor (Laude Institute) and mini-swe-agent (SWE-agent team). big-pickle is served by OpenCode Zen. Evaluation configs and results in this repo are MIT-licensed.