FLUX 3 Image Klein: a 5.55B DiT that reads its layout as JSON

mdjsonmcpcopy↗ai

2026-10-15 · 19 min · image-generation · diffusion · flow-matching · black-forest-labs · quantization · distillation

Why read this

Notable· top 60%

Counts Klein's DiT to the parameter from the reference code, shows the boxes are caption text, and does the 8 GB arithmetic per component and stage.

• Original analysis
• Runs on a consumer GPU
• Widely used

·Image & video generation·Custom licence·Practitioner model

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 63 of 100, ranked 273 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

On 2026-10-15 Black Forest Labs released FLUX 3 Image Klein as open weights. Its launch thread says "5.6B parameters and runs on a single consumer GPU. Native 4K, bounding boxes, and pixel-perfect editing." It ships a base checkpoint, a distilled one, and FP8, NVFP4 and INT8 builds "small enough to run on GPUs and laptops with 8 GB of VRAM". In BFL's internal evals it "scores higher than Qwen Image 2.1 (the current open weights leader) in text-to-image and multi-reference generation". Hugging Face's own account called it "Klein 6B" with "JSON bounding-box layouts". vLLM-Omni had day-0 serving.

There is no paper. There is a blog post, a model page, three Hugging Face repositories and, most usefully, an Apache-2.0 reference implementation. I read all of it at commit 45e57c9. The weight files are gated behind a licence click, so I could not range-read their safetensors headers. I did not need to: src/flux3/dit.py defines every layer, I counted them by hand, and the Hub's safetensors metadata gives the total to the parameter. The two agree exactly.

This is the second open FLUX 3 release on this site. FLUX 3 covered the family announcement and FLUX 3 Action the robot policy. Klein turns out to be the same backbone with fewer streams. Every number below is labelled measured (from a file, the code or the Hub API), reported (BFL's or a partner's figure, not re-run) or reasoned (my arithmetic on the other two). I generated no images; this box has no GPU.

What ships

The two DiT totals come from the Hub's safetensors metadata for each repository's root model.safetensors. The BF16 file is 11,097,404,512 bytes, which is two bytes a parameter plus a small header, so the metadata counts the DiT alone (measured).

The other two components are not new, and the Hub can prove it. Git LFS stores a SHA-256 for every file:

• text_encoder/ has the same two shard hashes as Qwen/Qwen3-VL-4B-Instruct. It is the unmodified Qwen model, 4,437,815,808 parameters by its (ungated) headers, of which 415,347,712 are the vision tower (measured). The FLUX 3 Action article found the same encoder.
• ae.safetensors has the same hash as the one in FLUX.2-dev, and the ComfyUI repo names it flux2-vae.safetensors. It is the FLUX.2 autoencoder, stored in FP32. From the layer definitions in autoencoder.py it has 84,046,115 parameters (measured, by counting), and four bytes each lands within 27 KB of the file size, the rest being BatchNorm statistics and the header (reasoned).

So the distilled pipeline is 10.07B parameters (reasoned: 5,548,681,984 + 4,437,815,808 + 84,046,115). The reference CLI also runs the 9.4B upsampler in front of every generation unless you pass --no-pup, which makes the default path 19.48B (reasoned). The model card is honest about the label: "The 5.6B parameter label refers to the image transformer; the text encoder is separate."

The 5.6B, counted

dit.py is 223 lines. Hidden size 3,072, 24 heads of 128, an MLP of 9,216 with a gated SiLU, no biases anywhere, and RMS-normed queries and keys. Every transformer block is the same: separate Q, K and V projections, an output projection, and a fused gate-and-up MLP that runs in parallel with attention off one modulated input. That is 122,683,648 parameters a block (measured). Then:

Both sums match the Hub metadata to the parameter (measured). The difference between base and distilled is exactly one extra timestep embedder, which is the first hint of how the distillation works (below).

On the headline numbers: 5,548,681,984 is 5.55B. To one decimal that is 5.5, not 5.6; BFL's "5.6B" and the blog's "11.2 GB in BF16" round up once more than the arithmetic does, and Hugging Face's "6B" rounds to one significant figure. None of these is wrong in a way that matters. The measured file is 11.10 GB.

The shared trunk is the same object as FLUX 3 Action's. That article measured 28 shared blocks at 3,435,142,144 parameters with five mode blocks per stream; Klein has the identical 28 and five mode blocks for each of three streams instead of five. The 6.99B robot model and the 5.55B image model are one architecture with the video, state and action streams swapped for a reference-image stream (reasoned from the two counts).

Three streams, then one

A request becomes three token streams. The text is the caption, run through Qwen3-VL-4B; the DiT takes the hidden states after layers 4, 8, 12, 16, 20, 24, 28 and 32, stacked to 8 × 2,560 = 20,480 numbers a token (measured, text_encoder.py). The vision tower is loaded but never used: the caption is encoded as text only, and reference images reach the DiT through the autoencoder instead. The caption is padded to a multiple of 80 tokens and capped at 4,096. The image stream is the noisy target: the autoencoder downsamples 8×, then 2 × 2 patches of its 32 channels become one 128-channel token, so a 1024 × 1024 image is 64 × 64 = 4,096 tokens (measured). The image_cond stream holds the references, encoded the same way and kept clean.

For the first five blocks each stream attends only to itself, with its own weights. Text refines text, references refine references, and the noisy image refines itself. Then the three are concatenated, sample by sample, and 28 shared blocks run one attention over everything. That is a different split from the FLUX.1 MMDiT, where two streams keep separate weights but attend jointly from the first block; the diffusion transformer explainer has that design.

Position is a four-axis rotary embedding, 32 dimensions per axis: (t, row, column, l). Image tokens use row and column, scaled by 0.5. Text tokens use only the fourth axis, for token order. References are told apart by that same fourth axis: reference n sits at l = -(1 + n), so ten references with identical pixels still have ten different positions (measured, pipeline.py).

The token budget follows. A single reference is fitted under 4 MiB of pixels, up to 16,384 tokens. With two or more, each is fitted under 1 MP, 4,096 tokens apiece, so the 10-reference maximum is up to 40,960 reference tokens beside a 4,096-token target (measured caps; reasoned totals). Attention in the shared blocks is quadratic in all of it, which is why BFL's own editing timings run well above its text-to-image ones.

Every token also carries its own timestep. Text and references sit at zero; only the image stream gets the sampling time. In the base model that has a consequence the reference code does not use: the five mode blocks of the text and reference streams see nothing that changes between steps, so their outputs are the same at step 1 and step 50 and could be computed once (reasoned from denoise(), which only ever fills the image timestep). The distilled model breaks that, because its jump-target embedding is added to every stream. The KV caching for flow models piece is about when such reuse is exact.

Boxes are text

The feature BFL leads with, "bounding boxes that control where elements are placed or confine edits to a defined area", has no dedicated input in the DiT. There is no box encoder, no mask, no extra stream. A layout is a JSON list appended to the caption. The model card's own example:

Minimalist graphic illustration featuring a black silhouette of a person <silhouette_1> centered against a solid, vibrant chartreuse background <background_1>. ... [{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green ..."}, {"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A black silhouette of a person in motion ..."}]

Each box is [y_min, x_min, y_max, x_max] on a 0 to 1000 grid, whatever the aspect ratio, and each element is cited in the caption as <id>. Qwen3-VL tokenizes the whole thing, and the DiT learned during training what those numbers mean. Editing by box uses another row shape: from names a reference (ref_image_0, …) or target, with a src_bbox, a tgt_bbox, or both. A null target removes the element.

Two formats are in play, and the README and the card describe different ones. The CLI's --boxes file is in 0 to 1 ([0.08, 0.06, 0.48, 0.38]); bbox.py clamps each value, multiplies by 1,000 and rounds before anything reaches the model. The builder below does exactly that conversion. Drag the boxes, edit the caption, and read both:

On the default path the boxes go to the upsampler, not straight to the DiT. bbox.py hands it the boxes as "LOCKED BOUNDING-BOX ANNOTATIONS", lets it write a long caption that cites each <id> and may add rows of its own (a sky, a sun, a valley), and then reconcile() restores your coordinates verbatim. In the repository's balloon example two input boxes come back as six rows (measured, examples/base/balloons.json). With --no-pup, the DiT gets your caption plus your rows, as on the right of the builder. If the caption already cites every row as <id>, is_finished() treats it as a finished upsampler answer and skips the 9B model.

Consequences of "the layout is text", all stated by the card: boxes "guide placement and scale; they do not constrain pixels to a hard mask", "an element can extend past its box, and small requested elements may not appear", and "edits can change content outside the requested region". The "pixel-perfect editing" on the API is a separate step: the blog says the API adds "pixel restoration, which preserves unedited regions of an image exactly". The open weights do not include it.

expand

Sixteen steps without guidance: a flow map

The base model samples like any flow-matching model. Fifty Euler steps along a predicted velocity, with classifier-free guidance at 3.5: each step runs the caption and an empty caption as two samples packed into one forward pass. The distilled model runs 16 steps and no guidance, and its update is different:

Here $x_t$ is the latent at noise level $t$ (1 is pure noise), $t'$ is the next point on the schedule, and $v_\theta$ is the network. The base model's network sees only $t$ and predicts the instantaneous velocity, which is only accurate for small steps. The distilled network also sees $t'$, through the extra ref_time_in embedder, and predicts the average velocity over the whole jump. That is a flow map: it is trained to land where many small steps would have landed. The card says so in passing, asking for "an inference implementation that supports flow-map sampling".

The schedule is shifted toward high noise, $t \mapsto 6.93\,t / (1 + 5.93\,t)$. For 16 steps the times run 1.0, 0.99, 0.98, … 0.615, 0.497, 0.316, 0 (measured, by evaluating timesteps(16, 6.93)). Fourteen of the 16 steps are spent above t = 0.5, and the last step jumps from 0.316 to zero in one go. Only a flow map can take that jump.

The speed claim is "about 6x faster". The arithmetic is 50 steps × 2 guidance samples = 100 sample-passes against 16, a ratio of 6.25 (reasoned). The measured claim is softer: "On one H200 at 1 MP: about 3 s against 12 s per image" (reported), a 4× wall-clock gain. Text encoding (twice for the base model), the autoencoder and fixed overhead do not shrink with the step count, and packing two guidance samples into one forward costs less than two forwards (reasoned). Compare FLUX.2 [klein], from January: its distilled models run 4 steps. FLUX 3 Image Klein spends four times as many steps on a smaller network.

8 GB, by arithmetic

BFL publishes peak memory for its own reference code: on a B200 at 1024 × 1024, distilled, no upsampler, 31 GB in BF16, 26 GB in FP8 and 25 GB in NVFP4 (reported). The upsampler "adds about 28 GB". None of that is 8 GB. The 8 GB claim belongs to the INT8 files in the ComfyUI repo, and the reference code has no INT8 path. The ledger below adds up the real file sizes, component by component:

With INT8 everywhere and everything resident, the weights alone are 11.02 GB (reasoned: 5.83 GB DiT, 4.86 GB text encoder, 0.34 GB autoencoder). That does not fit an 8 GB card. One stage at a time it does: encode the caption with the 4.86 GB encoder, drop it, then sample with the 6.16 GB DiT and autoencoder, leaving about 1.84 GB for activations at 1 MP (reasoned). That is what ComfyUI's model management does, and it is why "8 GB" is true and also why it costs a model swap per prompt. The upsampler does not fit that budget at all: its INT8 file is 9.75 GB (measured).

The reported peaks say something too. Weights for the BF16 configuration are 20.31 GB, against a 31 GB peak, a gap of 10.7 GB; for FP8 and NVFP4 the gap is 10.5 GB (reasoned). That is the reference code's activations, workspace and CUDA context at 1 MP, plus the always-loaded safety classifier. A memory-careful runtime spends less, but not zero.

The quantized files are larger than "5.6 GB in FP8" suggests. The FP8 DiT is 6,270,949,144 bytes and the NVFP4 one is 5,277,674,144 (measured). NVFP4 at half a byte a weight would be under 3 GB, so a large share of each file stays at higher precision. The quantization manifests that would say which tensors are behind the same licence gate, so I could not itemize them.

BFL's per-GPU timings, measured by NVIDIA and labelled as such:

expand

The thread's "~4s on NVIDIA GeForce RTX 5090 (INT8)" is the 4.1 s bar, and the blog's "under 9 seconds on an RTX 4070" is the 8.9 s one (reported). Editing at 2048 × 2048 on that 4070 is 150 s: a reference doubles the sequence, and 4× the pixels is 4× the tokens. The thread's 0.55 s on GB300 in NVFP4 matches the blog's "under 0.6 seconds" (reported). The vLLM-Omni recipe quotes "~2s on a single GB200", with no precision given; the vLLM-Omni article covers what that stack adds for a single-stage image model.

The evals: about 10 Elo, internally

The claim over Qwen-Image 2.1 rests on two charts from "internal evaluations". No prompt set, no rater count, no confidence intervals and no per-category breakdown are published (checked in the blog, the card and the README).

expand

Read off the chart, Klein leads Qwen-Image 2.1 by about 10 Elo in text-to-image and about 8 in editing (reported, read by eye). In the Elo model a 10-point gap means the higher-rated model wins 51.4% of head-to-head votes (reasoned: $1/(1 + 10^{-10/400})$). That is a lead, and it is also a near-tie that a different prompt set could reverse. The gaps over FLUX.2 [klein] 9B are real: the blog's +91 Elo in text-to-image and +70 in editing are 62.8% and 59.9% win rates (reasoned).

The thread says "higher than Qwen Image 2.1 ... in text-to-image and multi-reference generation"; the blog extends it to image editing as well. The size comparison is fairer than most because both sides are counted the same way, generator only: Klein's 5.55B is 78% of Qwen-Image 2.1's measured 7,115,124,736 (reasoned, using the Qwen-Image 2.1 article's count). Whole pipelines are closer to each other than the generators: Qwen-Image 2.1 measured 25.6B on its recommended path, and Klein's default path is 19.48B. Both rewriters are Qwen3.5-9B fine-tunes with exactly 9,409,813,744 parameters (measured).

expand

The latency chart carries its own caveat in BFL's footnote: Klein "was timed on our servers. Other models were timed from the client, which includes polling and network time." On the API the distilled model costs from $0.009 per 1K image (reported).

The licence

The weights are under the FLUX Kommunity License v1.0, the same text that covers FLUX 3 Action. It grants use "solely for: (i) your Non-Commercial Purposes ... and (ii) solely if you are a Qualifying User, use Outputs ... for commercial purposes", and a Qualifying User is one whose revenue "together with your affiliates" is "less than US$5,000,000". Commercial use of the outputs comes with conditions: content filters or review, and AI-generated disclosure "to the extent required under applicable law". The licence also forbids using the model or its outputs "to improve any other artificial intelligence model that performs any function similar to a FLUX Model", so you cannot distill a competitor from it. The text encoder and the upsampler are Apache-2.0, and so is the reference code.

Two things are worth knowing before you build on it. First, the model page's FAQ is stricter than the licence: "Running Klein in a product needs a commercial license". The licence grants a Qualifying User the right to use it "in a production environment solely for the purpose of generating such Outputs". Read the licence, not the FAQ, and then read it with a lawyer. Second, this is a step back from the previous small model. FLUX.2 [klein] 4B, released on 2026-01-14, is Apache-2.0 at 3,875,544,576 parameters (measured, Hub metadata); only its 9B sibling was non-commercial. FLUX 3 Image Klein has no Apache tier. The "enterprise licensing" post in the launch thread is the commercial path above the US$5M line.

Limits

• No images were generated for this article. Every quality statement is BFL's. I verified structure, sizes and code paths, not outputs.
• The quantization manifests are gated. Which tensors stay at BF16 in the FP8 and NVFP4 files is unknown to me, and so is the INT8 recipe's quality cost. BFL publishes no quality numbers for any quantized build.
• "Native 4K" is a cap, not a measurement. The code allows sides up to 4,096 pixels in multiples of 16. A 4096 × 4096 image is 65,536 image tokens, 16 times the 1 MP case (reasoned). BFL gives timings only up to 2048 × 2048.
• The evals cannot be reproduced. Internal Elo with no prompts or vote counts is a claim, and a 10-point lead is inside what a different prompt mix could move.
• The prompt matters more than usual. The card says "short prompts produce generic output", and the default path solves that with a 9.4B rewriter that does not fit an 8 GB card. Without it, you write the 200-word caption yourself.

Verdict

Klein is the FLUX 3 trunk with three streams: five private blocks per stream, 28 shared blocks, a four-axis rotary embedding, and a flow-map distillation that takes 16 steps where the base takes 50 with guidance. The parameter claim holds for the DiT, at 5,548,681,984, and the pipeline that loads is 10.07B, or 19.48B with the upsampler the CLI defaults to. The bounding boxes are caption text, which makes them easy to generate from code and impossible to enforce as masks. The 8 GB claim is true for INT8 with one stage resident at a time. The eval lead over Qwen-Image 2.1 is real on BFL's charts and small. The licence is free for research and for companies under US$5M, and it is the first small FLUX model without an Apache option.