# Coarse-to-Real C2R#: faking video pairs from image pairs
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/coarse-to-real-c2r
> date: 2026-10-27
> tags: video-generation, diffusion, synthetic-data, sim2real, wan, dino
[@Papagina_Yi](https://x.com/Papagina_Yi/status/2116710992155717699) posted a paper that turns rough 3D and anime videos into photorealistic ones "without paired video training data, by learning from cheaper image pairs". The paper is [arXiv 2610.28954](https://arxiv.org/abs/2610.28954), *Pair-Free Coarse-to-Real Video Generation via Real-to-Coarse Image Supervision*, by Gonzalo Gómez-Nogales, Daohan Lu, Peiye Zhuang, Tinghui Zhou and Yi Zhou, with the work done at Roblox. The model is called C2R#. It follows the group's earlier [Coarse-to-real: generative rendering for populated dynamic scenes](https://arxiv.org/abs/2601.22301), which needed some real paired videos. This one needs none.
A video generator is hard to judge from prose, so I did the parts that can be checked without one. I read the paper's HTML and PDF, pulled its numbers into one place, checked what is public (not much), and rebuilt the central trick as a toy you can run on a CPU in minutes. Every claim carries a label: **measured** (I computed it from a file or a run), **reported** (the paper's own number, not re-run by me) or **reasoned** (my arithmetic on the other two). I did not see any generated video beyond the paper's still figures, and I make no claim about how good the output looks beyond what the paper's figures show.
## The problem: coarse in, real out
A coarse-to-real model takes a rough video, such as a game render, a grey geometry blockout or a stylized animation, plus a text prompt, and generates a photorealistic video that keeps the layout, camera motion and dynamics of the input (reported). The input is the structural control. A big video model supplies the appearance.
Training one the obvious way needs aligned pairs: the same scene as a coarse video and as a real one. Those are expensive to make, and the cost repeats for every new kind of coarse input. The earlier C2R paper's answer was to train mostly on unpaired real videos reconstructed from their own DINO features, with a small set of paired synthetic videos to anchor controllability (reported). The authors' complaint about that recipe is that the adapter sees features of real videos in training but features of coarse videos at test time.
C2R# removes the paired videos altogether. The one paired resource left is still images.
## The asymmetry the paper leans on
The observation is one sentence: paired real-to-coarse images are cheap, paired videos are not (reported). An image editing model can turn a photo into a flat-shaded render or an anime frame. A 3D reconstruction can be simplified into a blockout. Each such edit preserves subjects, layout, pose and camera.
The paper builds four coarse looks: coarser geometry, coarse geometry, coarser anime and coarse anime. Each has **14,000** aligned pairs at 1280 by 720, made with GPT-Image 2 or with reconstructed and simplified 3D scenes (reported). That is 56,000 pairs in all (reasoned). The paper says larger sets gave only marginal improvements.
Running an image editor on every frame of a video corpus would cost too much. So the editor is distilled into a small network that can be run on every frame.
## The two models
### DINO-R2C, the bridge
DINO-R2C is a DINOv3-L/16 backbone shared across the four looks, with one small residual MLP head per look, 319.92M trainable parameters in total (reported). It takes a realistic image and predicts the DINO feature grid of its coarse twin: the grid is 45 by 80 by 1024 at 720p. A frozen decoder from an RAE (a representation autoencoder) turns the predicted features back into coarse pixels, so the loss has three parts, all in the paper's Equation 2:
- an L2 loss on DINO features (weight 1.0),
- an L1 loss on RGB pixels (weight 1.1),
- an LPIPS perceptual loss (weight 0.82).
Why DINO space and not pixels? The paper's reasoning is that DINO features carry layout and semantics and are relatively insensitive to the large appearance gap between real and coarse images. It adds that training "converges more easily" there (reported; no ablation I could find that isolates this).
Training took 20,000 iterations at global batch 192 on 3 by 8 B200 GPUs, 4 h 02 min 58 s, which the paper converts to 97.19 GPU-hours (reported). At 720p on one B200 a single image takes 34.779 ms, and an 81-frame video takes a median 2.029 s, or 39.92 frames per second (reported).
### Camera augmentation, because the bridge sees video
DINO-R2C is trained on single images but applied to every frame of a video. Each frame is processed independently, so the pseudo-coarse video jitters. The fix in training is to apply the same random homography (crop, translation, rotation, shear, perspective) to the realistic image and its coarse twin, so the mapping stays stable under small viewpoint changes. The 2x setting doubles the 1x bounds: retained crop area in [0.50, 1.00], rotation up to ±3.55 degrees at 1x (reported). I come back to this in the toy.
### C2R#, the video model
C2R# freezes a **Wan 2.2 T2V-A14B** backbone (VAE, text encoder and both mixture-of-experts denoisers) and trains only adapters: one control adapter per expert, which turns the per-frame DINO control features into the denoiser's width through a zero-initialized output head, plus a zero-initialized fusion adapter that adds them after the patch embedding. At initialization the model equals Wan 2.2 (reported). Trainable parameters: **180,540,416**, 0.63% of the counted total (28.758B), one epoch over 389,854 real clips, 12,183 optimization steps at global batch 32. The loss is the standard rectified-flow velocity loss against the real video, with no control dropout, since a frozen backbone gets no gradient from a step without control (reported).
The 389,854 clips come from DL3DV and two SEKAI sets, at 1280 by 720, 16 fps, 81 frames, and their pseudo-coarse controls are made on the fly by DINO-R2C (reported). That is about 28 real clips per image pair (reasoned: 389,854 over 14,000).
At inference DINO-R2C and the decoder are gone. The frozen DINO encoder reads the real coarse video and drives the same adapters. Generation is 81 frames at 832 by 480 with 50 denoising steps, two forward passes per step for guidance, and a fixed seed of 4321 (reported).
## The decode step, and why it exists
The detail that makes this more than a pipeline diagram is in Section 3.3. DINO-R2C's predicted features could drive C2R# directly. The paper instead decodes them to coarse pixels with the frozen RAE decoder, then re-encodes those pixels with the same DINO encoder used at inference. The reason given: the predicted features "retain hidden cues of the realistic input that are absent when an actual coarse video is encoded at inference", which creates a train and test mismatch (reported). Decoding is an information bottleneck: whatever cannot be drawn as a coarse image is dropped.
The paper shows one example of the failure and excludes this variant from its tables.
I cannot judge more than the picture shows: one example, picked by the authors. It is the paper's only evidence for this design choice, and it is qualitative (reported). That makes it the part I most wanted to test.
## What I built: a numpy toy of the whole recipe
The toy keeps the structure and drops everything else. All of it is in the script below: numpy only, no GPU, a few minutes per setting on a shared 4-core box with two threads.
- **Real image**: 12 by 12 pixels, a sum of three to five Gaussian blobs plus fine Gaussian noise (std 0.15) standing in for texture.
- **Coarse image**: a 3 by 3 box blur of the real image, clipped and rounded to four levels. This is a deterministic function of the real image, so true pairs exist for evaluation, but the *trainer never gets them* except for the few image pairs.
- **DINO-R2C**: a one-hidden-layer MLP, real to coarse, trained on `n_pairs` still images with a random-shift augmentation applied to both halves of each pair.
- **Real videos**: 500 clips of 8 frames, blobs drifting at constant velocity. No coarse version is ever shown to the generator's trainer.
- **C2R#**: another MLP, coarse to real, trained on (control, real frame) with the control from R2C.
- **The decode step**: `snap()` rounds R2C's output back onto the four coarse levels, a crude stand-in for "decode to pixels and re-encode".
- **Test**: 600 held-out real images, with the true coarse image as the actual input. Metric: mean squared error to the real image, which includes the texture C2R cannot know about. The texture floor is 0.15 squared, 0.0225.
Four variants are compared: C2R trained directly on the few real pairs ("image pairs only"), C2R trained on R2C's raw output, C2R trained on the snapped output, and an oracle trained on the true coarse version of the same videos, which is the paired data a real project does not have.
Numbers (measured, mean of 3 seeds, error to the real image, ± one standard deviation):
| Image pairs | Pairs only | Pseudo, raw | Pseudo, decoded | Oracle |
|---:|---:|---:|---:|---:|
| 20 | 0.0664 ± 0.0051 | 0.1270 ± 0.0067 | **0.0349 ± 0.0005** | 0.0278 ± 0.0004 |
| 100 | 0.0696 ± 0.0010 | 0.1645 ± 0.0063 | **0.0309 ± 0.0002** | 0.0280 ± 0.0000 |
| 1000 | 0.0335 ± 0.0006 | 0.1614 ± 0.0104 | **0.0313 ± 0.0009** | 0.0282 ± 0.0003 |
Three things stand out, and one of them does not flatter the paper's setup.
**The fake pairs work.** With decoding, the video model trained on pseudo-controls from 20 image pairs lands within 0.0071 of the oracle (0.0349 against 0.0278), and at 100 pairs within 0.0029. Training straight on the 20 or 100 image pairs is about twice as bad (0.0664, 0.0696). The generator gets 4,000 unlabeled video frames to learn from, and the few image pairs only have to teach the bridge.
**Skipping the decode step is a failure, not a small loss.** The raw output is three to five times worse than the decoded one, and worse than not using the video corpus at all, at every pair count I tried. More image pairs do not rescue it (0.1270, 0.1645, 0.1614). That matches the direction of the paper's Figure 4, which I would not have bet on before running it. It does not prove the mechanism the paper names. I measured the correlation between R2C's residual (its output minus the true coarse) and the real image's texture. With the raw output it is 0.05 to 0.07, with the snapped output 0.02 to 0.03. That is small. So in my toy the raw output is wrong mostly because it is *off the set of things a coarse image can be*, a smooth soft version of one, not because it smuggles the real texture through. Decoding fixes both, and I did not separate them. In the paper's setting the decoder is a trained RAE rather than a rounding step, and whether its failure is leakage in the sense they describe is something only their ablation could show.
**With enough pairs, plain supervision catches up.** At 1000 image pairs, training directly on the pairs gives 0.0335, within 0.0022 of the decoded pseudo-pair result (0.0313). My toy images are tiny and my coarse function is easy, so this says nothing about 720p video. It does say something about the claim's shape: the trick is worth the most when pairs are scarce, and the paper's argument is that for video they are, by orders of magnitude (reasoned, from the paper's costs, not from my toy).
### Jitter: why augmentation matters
The same toy reproduces the shape of the paper's Table 1. I trained R2C with shift augmentation of 0, 1, 2 and 4 pixels, ran it frame by frame on the video clips, and measured flicker as the mean frame-to-frame change of the snapped pseudo-coarse video divided by the same quantity for the true coarse video. A value of 1 means no added jitter (measured, 200 pairs, 3 seeds, 4,000 training steps per network where the table above used 6,000):
| Shift augmentation | Relative flicker | R2C error on held-out images | Final C2R error |
|---:|---:|---:|---:|
| 0 | 2.098 ± 0.026 | 0.0335 | 0.0355 |
| 1 px | 1.074 ± 0.021 | 0.0139 | 0.0315 |
| 2 px | 1.042 ± 0.013 | 0.0136 | 0.0319 |
| 4 px | 1.020 ± 0.005 | 0.0160 | 0.0331 |
Without augmentation the bridge doubles the frame-to-frame change. One or two pixels of random shift removes nearly all of it, and also improves the image-level error, since the augmentation is effectively more data. Four pixels flickers least but fits the images worse, and the final model is slightly worse. The paper's own sweep (reported, 1,000 real videos, coarse geometry, 256 by 256) shows the same curve at far larger scale:
| DINO-R2C recipe | Perceptual flicker | Smoothed-frame flicker | Motion error | Boundary Dice |
|---|---:|---:|---:|---:|
| none, no LPIPS | 2.2028 | 4.1565 | 2.2685 | 0.7195 |
| none + LPIPS | 2.1793 | 3.9131 | 1.8678 | 0.7271 |
| 0.5x + LPIPS | 0.9314 | 1.7901 | 0.8819 | 0.8345 |
| 1x, no LPIPS | 0.7965 | 1.6825 | 0.9066 | 0.8327 |
| 1x + LPIPS | 0.9023 | 1.7108 | 0.8695 | 0.8354 |
| 2x + LPIPS (chosen) | 0.8649 | 1.6294 | 0.8429 | 0.8376 |
The paper picks 2x with LPIPS and says openly that it is a balance, since LPIPS raises both relative flicker numbers at 1x. My toy has no LPIPS and my metric is simpler, so the agreement is in the shape only: no augmentation roughly doubles the flicker.
The full toy script (numpy only, about 110 lines)
```python
"""Pair-free coarse-to-real, numpy only. python3 toy.py [n_pairs ...]"""
import sys, json
import numpy as np
S = 12; D = S * S; T = 8; TEX = 0.15; LV = 3
def blobs(rng, n, vel=False):
ys, xs = np.mgrid[0:S, 0:S].astype(float)
out = []
for _ in range(n):
k = rng.integers(3, 6)
p = rng.uniform(2, S - 2, (k, 2)); a = rng.uniform(.4, 1, k); s = rng.uniform(1.3, 2.6, k)
v = rng.uniform(-.5, .5, (k, 2)) if vel else np.zeros((k, 2))
fr = []
for t in range(T if vel else 1):
q = p + v * t
f = sum(a[i] * np.exp(-((ys - q[i, 0]) ** 2 + (xs - q[i, 1]) ** 2) / (2 * s[i] ** 2)) for i in range(k))
fr.append(f)
out.append(np.stack(fr))
return np.concatenate(out) if not vel else np.stack(out) # (n,S,S) or (n,T,S,S)
def blur(x): # 3x3 box blur, edge padded
p = np.pad(x, ((0, 0), (1, 1), (1, 1)), mode="edge")
return sum(p[:, i:i + S, j:j + S] for i in range(3) for j in range(3)) / 9
def add_tex(rng, f): return f + TEX * rng.standard_normal(f.shape) # real = field + fine texture
def coarse(real): return np.round(np.clip(blur(real), 0, 1) * LV) / LV # blur, posterize to 4 levels
def snap(x): return np.round(np.clip(x, 0, 1) * LV) / LV # "RAE decode": back onto the coarse set
class MLP:
def __init__(s, i, o, h=160, rng=None):
r = rng; s.W1 = r.standard_normal((i, h)) / np.sqrt(i); s.b1 = np.zeros(h)
s.W2 = r.standard_normal((h, o)) / np.sqrt(h) * .5; s.b2 = np.zeros(o)
s.m = [np.zeros_like(p) for p in s.P()]; s.v = [np.zeros_like(p) for p in s.P()]; s.t = 0
def P(s): return [s.W1, s.b1, s.W2, s.b2]
def f(s, x): return np.tanh(x @ s.W1 + s.b1) @ s.W2 + s.b2
def fit(s, X, Y, steps, rng, lr=2e-3, bs=128, aug=None):
for _ in range(steps):
idx = rng.integers(0, len(X), bs); x, y = X[idx], Y[idx]
if aug: x, y = aug(x, y, rng)
h = np.tanh(x @ s.W1 + s.b1); e = (h @ s.W2 + s.b2 - y) / bs
g2 = h.T @ e; gb2 = e.sum(0); dh = (e @ s.W2.T) * (1 - h * h)
G = [x.T @ dh, dh.sum(0), g2, gb2]; s.t += 1
for k, (p, g) in enumerate(zip(s.P(), G)):
s.m[k] = .9 * s.m[k] + .1 * g; s.v[k] = .999 * s.v[k] + .001 * g * g
p -= lr * (s.m[k] / (1 - .9 ** s.t)) / (np.sqrt(s.v[k] / (1 - .999 ** s.t)) + 1e-8)
def shift_aug(strength):
def a(x, y, rng):
if strength == 0: return x, y
dy, dx = rng.integers(-strength, strength + 1, 2)
sh = lambda z: np.roll(z.reshape(-1, S, S), (dy, dx), (1, 2)).reshape(-1, D)
return sh(x), sh(y) # same shift on image and its coarse partner
return a
def run(n_pairs, seed, aug=2, steps=6000):
rng = np.random.default_rng(seed)
# (1) the only paired data: n_pairs still images
fi = blobs(rng, n_pairs); Ri = add_tex(rng, fi); Ci = coarse(Ri)
# (2) unpaired real "videos": 500 clips x 8 frames
fv = blobs(rng, 500, vel=True); Rv = add_tex(rng, fv).reshape(-1, S, S)
# (3) held-out actual coarse inputs with real ground truth
ft = blobs(rng, 600); Rt = add_tex(rng, ft); Ct = coarse(Rt)
flat = lambda z: z.reshape(len(z), -1)
r2c = MLP(D, D, rng=rng); r2c.fit(flat(Ri), flat(Ci), steps, rng, aug=shift_aug(aug))
soft = r2c.f(flat(Rv)) # pseudo-coarse, features only (leaky)
hard = snap(soft) # decode back to the coarse set
out = {}
def c2r(ctrl, tgt, name):
m = MLP(D, D, rng=rng); m.fit(ctrl, tgt, steps, rng)
p = m.f(flat(Ct)); out[name] = dict(mse_real=float(np.mean((p - flat(Rt)) ** 2)),
mse_field=float(np.mean((p - flat(ft)) ** 2)))
c2r(flat(Ci), flat(Ri), "paired_only") # D: just the n_pairs images, supervised directly
c2r(soft, flat(Rv), "pseudo_no_bottleneck") # B
c2r(hard, flat(Rv), "pseudo_bottleneck") # C
# oracle: true coarse for the same video frames
c2r(flat(coarse(Rv)), flat(Rv), "oracle_true_coarse")
# diagnostics
Ctrue_v = coarse(Rv)
out["r2c_image_mse_test"] = float(np.mean((snap(r2c.f(flat(Rt))) - flat(Ct)) ** 2))
tex = flat(Rv) - flat(fv.reshape(-1, S, S))
res_soft = soft - flat(Ctrue_v); res_hard = hard - flat(Ctrue_v)
cor = lambda a, b: float(np.corrcoef(a.ravel(), b.ravel())[0, 1])
out["leak_corr_soft"] = cor(res_soft, tex); out["leak_corr_hard"] = cor(res_hard, tex)
# flicker: frame-to-frame change of pseudo-coarse relative to that of true coarse (video-level)
ps = hard.reshape(500, T, D); tc = Ctrue_v.reshape(500, T, D)
out["flicker_rel"] = float(np.abs(np.diff(ps, axis=1)).mean() / np.abs(np.diff(tc, axis=1)).mean())
return out
if __name__ == "__main__":
ns = [int(a) for a in sys.argv[1:]] or [20, 100, 1000]
res = {}
for n in ns:
rs = [run(n, s) for s in range(3)]
keys = [k for k in rs[0]]
res[n] = {}
for k in keys:
if isinstance(rs[0][k], dict):
res[n][k] = {m: [float(np.mean([r[k][m] for r in rs])), float(np.std([r[k][m] for r in rs]))] for m in rs[0][k]}
else:
res[n][k] = [float(np.mean([r[k] for r in rs])), float(np.std([r[k] for r in rs]))]
print(n, json.dumps(res[n]), flush=True)
json.dump(res, open("results.json", "w"), indent=1)
```
Run it with `python3 toy.py 20 100 1000`. The flicker table is `toy.run(200, seed, aug=a, steps=4000)` for `a` in 0, 1, 2, 4 and seeds 0 to 2, reading `flicker_rel` and `r2c_image_mse_test` from the result. Each run rebuilds its own data from the seed, so the numbers above are reproducible to the printed digits on the same numpy.
## The paper's results, as reported
The main comparison uses 300 actual coarse videos (100 game renders, 100 geometry blockouts, 100 stylized animations), with no real ground truth, because a coarse video has no photoreal twin. So it uses reference-free quality scores (DOVER), a learned editing judge (VEFX), VBench consistency, three control-alignment measures and a realism score from a Qwen vision-language model on 0 to 100 (reported). Pick a metric:
The paper's own reading (reported): C2R# leads DOVER technical and aesthetic quality and all three VEFX scores, is second on background consistency and realism, and improves nine of ten metrics over the original C2R, but the original keeps the higher realism mean, 75.50 against 74.17. The adapted Editto model has the lowest DINO displacement, and default VACE 2.2 leads flow and camera-pose agreement, which the paper frames as a quality and control trade-off. My reading of the same table (reasoned): C2R# is fourth on DINO displacement, fifth of seven on flow agreement and fourth on camera-pose error, so the headline is not "most faithful to the input". It is "best-looking among methods that stay reasonably close", judged by learned scorers and one VLM. Since the realism score is a single model's opinion on 300 videos, a 1.33-point gap to the predecessor (75.50 against 74.17) is not something I would read anything into.
The cleaner evidence for the *pair-free* part is in the transfer test. Adapting the Editto Sim2Real baseline with the generated pairs improves all ten reported metrics, with realism going from 52.62 to 58.52. Doing the same to VACE 2.2 is mixed, with some metrics better and the rest worse (reported). So the pseudo-pairs carry real signal, and how much a given model profits depends on the model.
The figure is what the paper chose to show, and the paper provides no confidence intervals or per-category breakdowns of Table 2 that I could find. I can say what the picture shows: the baselines are visibly different from one another, and in the stylized rows some keep a drawn look. I cannot say more about quality than that.
## What it costs (reasoned)
Two costs hide behind "pair-free".
The bridge is cheap: 97.19 GPU-hours on B200s, once per set of coarse looks (reported). The generator is cheap in trainable parameters: 180.54M, or 0.63%. But the pseudo-coarse controls are made *during* C2R# training, one bridge forward per frame. One epoch is 12,183 steps at batch 32 of 81-frame clips, about 31.6 million frames. At the paper's 39.92 frames per second for the bridge, that is about 791,000 seconds, or about 220 B200 GPU-hours spent just on the controls (reasoned; the real number depends on how the authors batch it, and they do not report the C2R# training time that I could find). That is more than double the bridge's own training cost. The paper's summary that it "moves the expensive part of supervision from videos to images" is right about the data; compute just moves into the loop.
The predecessor had a different cost shape. Its Stage I fine-tunes the entire 14.288B backbone and Stage II trains 44.33M of control and fusion parameters, while C2R# trains 180.54M on a frozen 28.758B total (reported, Table 4). That is, C2R# trains less of the model than Stage I of the predecessor did, but more adapter.
## What is released
- **C2R#, the paper**: "We will release the model and project webpage". The abstract says will, not has. When I read the project page through a summarizer, it listed no code, weights or paper links, and the arXiv page listed no repository (reported by a tool, not seen in a browser by me). I would treat the weights as unreleased until a repository exists.
- **The predecessor**: a Hugging Face model, [`gonsaBRK/coarse2real`](https://huggingface.co/gonsaBRK/coarse2real), tagged `arxiv:2601.22301` with base model Wan2.1-T2V-14B, licence **CC BY-NC-ND 4.0**, access set to *manual* gating. I could not download it, so everything I know comes from its public metadata. It lists two files: `c2r-dino-adapter.safetensors` at 88,670,672 bytes and `c2r-dit-backbone-14B.safetensors` at 28,577,096,712 bytes (measured, from the file listing). At two bytes per parameter in bf16, those are about 44.3M and 14.29B parameters, which agree with the paper's Table 4 for the predecessor (44.33M adapter trained in Stage II, 14.288B backbone) (reasoned: the dtype is my assumption). That is a useful sanity check on the paper's table. It is not a check on any quality claim.
- **Licence of the work itself**: the arXiv listing is CC BY-NC-ND 4.0. The authors call the current implementation "constrained by the quality of the non-commercial backbone model and training data available", so a commercial use is not simple (reported).
I could not run the predecessor's model or any part of C2R#. The toy is not C2R#: it shares the structure, not the scale, the modalities, the DINO features or the video model.
## Where it can break
The authors name the main weakness themselves (reported): the method rests on pseudo-coarse videos approximating real coarse ones, and quality may fall when the gap between them grows. My toy makes that concrete. The decode step pulls R2C's output back toward the coarse set, but the set is whatever the four trained looks cover. A new coarse style needs new image pairs and a new bridge, and until then it is out of distribution (reasoned).
Two other limits the paper states: evaluation on real coarse inputs has no ground truth, so every score is a proxy or a judge; and the frozen video prior trades control for realism. When the input has an abrupt pose or motion change, C2R# may "adjust the motion pace or synthesize intermediate states" and drift from the control (paper, Figure 7). The training data is real, smooth video, so the model learns controls that look like smooth real motion, and a game that snaps between poses is unlike anything it saw (reasoned).
## What to take from it
- The idea is a data trick, not an architecture. Learn the *inverse* of the thing you want on cheap paired data, use it to manufacture training inputs for the forward direction from plentiful unpaired outputs, then swap the real input in at test time. The target is always a real video, so the generator never learns to imitate the bridge's flaws.
- The swap only works if the manufactured input looks like the real one. In the paper that is the decode and re-encode step. In my toy, skipping the equivalent made the model three to five times worse than not using the video at all (measured), and the correlation with texture was too small to call it leakage (measured). Whichever it is, check the distribution of your fake inputs against your real ones before you train.
- Image-trained, frame-wise bridges flicker without augmentation. Shift or homography augmentation on both halves of each pair fixed most of it in my toy and in the paper's sweep.
- Nothing runnable yet from the authors for C2R#. If you want to try the idea, the toy and a few hundred paired images of your own coarse domain are more than enough to see whether the fake-pair route is stable before touching a 14B video model.