{
  "claim": "Pareas's parallel lexer turns every input byte into an identifier for a DFA transition function, and composes them with a k×k merge table. k is the number of distinct transition functions reachable by composition, and it is decided entirely by the lexical grammar. I rebuilt pareas-lpg and measured k for the two shipped grammars and six variants.",
  "method": "g++ -std=c++20 -O2 on src/lpg (fmt 7.1.3 header-only, no Futhark needed), then `pareas-lpg --lexer <g.lex> --check --verbose-lexer`, which prints the merge table size. Bytes = k² × 2, because src/compiler/lexer/lexer.fut declares the table as `merge_table: [n][n]state` with `module state = u16`. Ubuntu 24.04, 4 cores, 15 GB RAM.",
  "source": "https://github.com/Snektron/pareas",
  "captured": "2026-09-18",
  "columns": [
    { "key": "grammar", "label": "lexical grammar" },
    { "key": "change", "label": "change from shipped" },
    { "key": "k", "label": "k (states)", "align": "right" },
    { "key": "bytes", "label": "u16 table", "align": "right" },
    { "key": "secs", "label": "generator", "align": "right" }
  ],
  "rows": [
    {
      "grammar": "src/json/json.lex",
      "change": "shipped, unmodified",
      "k": "4,176",
      "bytes": "34.9 MB",
      "secs": "15 s"
    },
    {
      "grammar": "json.lex − string",
      "change": "deleted the string rule",
      "k": "642",
      "bytes": "0.82 MB",
      "secs": "0.3 s"
    },
    {
      "grammar": "json.lex − number",
      "change": "deleted the number rule",
      "k": "1,482",
      "bytes": "4.4 MB",
      "secs": "1.5 s"
    },
    {
      "grammar": "json.lex, \\u00E9 fix",
      "change": "[0-9a-f] → [0-9a-fA-F] in \\uXXXX",
      "k": "4,249",
      "bytes": "36.1 MB",
      "secs": "17 s"
    },
    {
      "grammar": "src/compiler/lexer/pareas.lex",
      "change": "shipped, unmodified",
      "k": "1,927",
      "bytes": "7.4 MB",
      "secs": "4 s"
    },
    {
      "grammar": "pareas.lex − comment",
      "change": "deleted the // comment rule",
      "k": "798",
      "bytes": "1.3 MB",
      "secs": "0.9 s"
    },
    {
      "grammar": "pareas.lex + string literal",
      "change": "added one C-style \"...\" rule",
      "k": "7,952",
      "bytes": "126.5 MB",
      "secs": "139 s"
    },
    {
      "grammar": "pareas.lex + block comment",
      "change": "added one /* ... */ rule",
      "k": "> 38,000",
      "bytes": "> 2.8 GB",
      "secs": "did not finish"
    }
  ],
  "note": "The last row is the thesis's own predicted failure, measured. The stock generator dies on it with std::bad_alloc after about six seconds — MergeTable::resize allocates capacity² × sizeof(Transition) and the doubling step past 16,384 asks for 17 GB. I rebuilt it with resize() patched to count states instead of allocating; the count passed 38,000 and was still in the merge closure when I stopped it, so the k and byte figures on that row are lower bounds, not the answer."
}