{
 "claim": "L1 of the pipeline is where \"near-duplicate deduplication\" happens, and L2 is selected from L1. 178 of 5,500 sampled L2 rows — 3.24% — are byte-for-byte identical to another row in the same 5,500. Every one of the 178 duplicate pairs crosses a repository boundary, so within-repo deduplication clearly worked and cross-repo copies of vendored and template files did not. PHP is 12.0% duplicate rows in-sample, Ruby 10.4%, Java and Rust 0.0%.",
 "method": "500 rows per split (5 windows of 100 at offsets 0%, 20%, 40%, 60% and 80%) pulled through the datasets-server /rows endpoint, 11,000 rows over 22 splits, of which these 5,500 are L2. SHA-256 over the UTF-8 bytes of the `content` column; a group is a set of rows sharing one digest. This is a floor, not the corpus rate: a 5,500-row sample of 266,878,376 rows will only catch duplicates common enough to hit twice.",
 "source": "https://datasets-server.huggingface.co/rows?dataset=openbmb%2FUltraData-Code&config=UltraData-Code-L2&split=php&offset=0&length=100",
 "captured": "2026-09-18",
 "note": "67 of the 106 duplicate groups carry byte-identical content with different scores attached, so the scoring pass is not deterministic. No group disagreed about the `category` label.",
 "columns": [
  {
   "key": "lang",
   "label": "language"
  },
  {
   "key": "sampled",
   "label": "rows sampled",
   "align": "right"
  },
  {
   "key": "dup",
   "label": "duplicate rows",
   "align": "right"
  },
  {
   "key": "rate",
   "label": "rate",
   "align": "right"
  },
  {
   "key": "groups",
   "label": "groups",
   "align": "right"
  },
  {
   "key": "biggest",
   "label": "largest group",
   "align": "right"
  }
 ],
 "rows": [
  {
   "lang": "C++",
   "sampled": 500,
   "dup": 5,
   "rate": "1.00%",
   "groups": 2,
   "biggest": 5
  },
  {
   "lang": "C#",
   "sampled": 500,
   "dup": 21,
   "rate": "4.20%",
   "groups": 19,
   "biggest": 3
  },
  {
   "lang": "Go",
   "sampled": 500,
   "dup": 1,
   "rate": "0.20%",
   "groups": 1,
   "biggest": 2
  },
  {
   "lang": "Java",
   "sampled": 500,
   "dup": 0,
   "rate": "0.00%",
   "groups": 0,
   "biggest": 1
  },
  {
   "lang": "JavaScript",
   "sampled": 500,
   "dup": 7,
   "rate": "1.40%",
   "groups": 6,
   "biggest": 3
  },
  {
   "lang": "PHP",
   "sampled": 500,
   "dup": 60,
   "rate": "12.00%",
   "groups": 29,
   "biggest": 7
  },
  {
   "lang": "Python",
   "sampled": 500,
   "dup": 7,
   "rate": "1.40%",
   "groups": 6,
   "biggest": 3
  },
  {
   "lang": "R",
   "sampled": 500,
   "dup": 2,
   "rate": "0.40%",
   "groups": 2,
   "biggest": 2
  },
  {
   "lang": "Ruby",
   "sampled": 500,
   "dup": 52,
   "rate": "10.40%",
   "groups": 24,
   "biggest": 6
  },
  {
   "lang": "Rust",
   "sampled": 500,
   "dup": 0,
   "rate": "0.00%",
   "groups": 0,
   "biggest": 1
  },
  {
   "lang": "Shell",
   "sampled": 500,
   "dup": 23,
   "rate": "4.60%",
   "groups": 17,
   "biggest": 4
  },
  {
   "lang": "all 11",
   "sampled": 5500,
   "dup": 178,
   "rate": "3.24%",
   "groups": 106,
   "biggest": 7
  }
 ]
}