{
  "claim": "All five of the non-empty patches in the Jesse / Weightless SWE-bench Verified submission are byte-for-byte identical to the gold patch shipped in the SWE-bench Verified dataset — including the diff headers and the @@ hunk line numbers. The submission's own PR checklist attests to 'no gold patch leakage (no hints, no patch copying, no test leakage)'.",
  "method": "Downloaded the Jesse artefact repo (m0at/jesse-swe-bench-verified) and the SWE-bench Verified test split parquet from Hugging Face (princeton-nlp/SWE-bench_Verified, 500 rows). Compared preds.json['<instance>']['model_patch'].strip() against the dataset's patch column .strip(), exactly, with no normalisation. 495 of the 500 predictions are empty strings; the 5 non-empty ones are listed here. Also compared after stripping diff/index/---/+++/@@ lines, which changes nothing because the raw comparison already matches.",
  "source": "https://github.com/m0at/jesse-swe-bench-verified · https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified",
  "captured": "2026-09-19",
  "note": "The five evaluation logs in the artefact repo are genuine SWE-bench harness container output (13k–37k lines each) and the resolutions are real: the patches do flip FAIL_TO_PASS with no PASS_TO_FAIL regressions. That was never in doubt — gold patches resolve their own instances by definition. What the logs cannot show is where the patch came from, and the harness never asks.",
  "columns": [
    { "key": "id", "label": "instance" },
    { "key": "change", "label": "the whole patch" },
    { "key": "same", "label": "identical to gold?", "align": "right" }
  ],
  "rows": [
    {
      "id": "astropy__astropy-12907",
      "change": "cright[-right.shape[0]:, -right.shape[1]:] = 1  →  = right",
      "same": "byte-for-byte"
    },
    {
      "id": "django__django-10914",
      "change": "FILE_UPLOAD_PERMISSIONS = None  →  = 0o644",
      "same": "byte-for-byte"
    },
    {
      "id": "django__django-11163",
      "change": "if fields and …  →  if fields is not None and …",
      "same": "byte-for-byte"
    },
    {
      "id": "django__django-11179",
      "change": "+ setattr(instance, model._meta.pk.attname, None)",
      "same": "byte-for-byte"
    },
    {
      "id": "django__django-11451",
      "change": "+ if username is None or password is None: return",
      "same": "byte-for-byte"
    }
  ]
}