StarCraft FP #9: A Clean-up Model Trained on Leave-One-Out Renders

The eye-level frames from posts #7 and #8 are the original pixels on rebuilt geometry. Where no frame ever showed a surface they are filled from around it, and where neighbouring directions disagree they turn blotchy. Steep units suffer most: the Firebat’s chest and the SCV’s back. This post covers a clean-up model meant to repaint those areas in the unit’s own art.

No answers at eye level

Training an image model needs input/answer pairs, and there are no eye-level frames to serve as answers. The pairs come from the original camera instead (refine_data.py):

• input: the unit’s voxel model drawn through the original camera in direction d, without using frame d (and, harder, without d’s neighbours too);
• answer: frame d itself.

Those renders have the eye-level renders’ faults (filled surfaces, pixels borrowed from a nearby surface, a neighbour’s lighting and registration, stretching), but here the right answer is known. Input and answer share the viewpoint, so the model learns to mend art, not to change the view.

The model

refine.py is a 3.1M-parameter U-Net (4 levels, GroupNorm, SiLU). Its input has 6 channels: the render’s RGB, coverage, and two masks from the renderer, for filled pixels and borrowed pixels. Its output is a correction added to the RGB; the last layer starts at zero, so an untrained model returns the render unchanged, and the outline always stays the render’s.

def loss_fn(out, x, y):
    m = inside both outlines
    w = m * (1 + filled + borrowed)           # the faulty pixels count double
    return L1(out, y, w) + 0.5 * L1(gradients(out), gradients(y), w)

It trains on random 128 px patches, so it mends locally instead of recalling whole poses. Directions 3 and 11 of every unit are held out for validation.

Version 1: better numbers, same pictures

v1 trained on 2618 pairs from 77 poses for 20k steps (17 minutes on a laptop RTX 3060). On the held-out directions it improved the renders from 18.7 to 21.2 dB against the original frames. At eye level the Firebat stayed blotchy and the SCV got muddier.

Two reasons:

• The training renders had almost nothing to fill. With one neighbour 11 degrees away, a leave-one-out render through the original camera has 0% filled pixels. At eye level 20-30% of a steep unit is filled. The model learned to fix seams and registration, not to paint large unseen areas.
• L1 loss averages. Where the answer is uncertain, the L1-optimal output is the mean of the plausible answers, which is a blur.

Version 2

Both problems have direct fixes:

• Harder pairs: besides leaving out d and its neighbours, a “blob” variant replaces random smooth blobs covering 20-35% of the unit with the renderer’s fill and marks them as filled (28% on average, against 0% before).
• An adversarial critic: a conditional PatchGAN with spectral normalisation scores 34 px patches as real frames or cleaned renders, with a hinge loss weighted 0.03 after 2000 steps. It pushes the model to paint texture in the art’s style instead of the average.

Running it on Kaggle

To keep the local GPU free, the GPU work moved to a free Kaggle T4. A private dataset holds the tools, the units’ anim files, their voxel models and the model files, plus a todo.json job list. A private kernel runs the jobs: build a unit, re-render one from given voxel models with a clean-up model, generate pairs, or train.

kaggle datasets version -p kaggle_eyelevel_in -m "pairs v2" --dir-mode zip
kaggle kernels push -p kaggle_eyelevel
kaggle kernels output kayo01/sc-eyelevel -p results/

The first run built the Protoss units (Probe, Zealot, Dragoon) and re-rendered ten units with v1 in 40 minutes. The second trained v2 (6188 pairs from 91 poses, 30k steps) and re-rendered the same ten units in 67 minutes.

One trap on the way: pushing the kernel right after kaggle datasets version mounted the previous version of the dataset, because kaggle datasets status reports “ready” for the old version while the new one is still processing. The kernel now stops at once if a file the job list needs is missing.

Result: neither version goes in

v2 raised its held-out score from 17.9 to 19.7 dB on the harder pairs. At eye level it traded v1’s blur for grit: more texture, but noisy, with stray colour speckles. On a unit that already looked right, both versions made it worse.

The game keeps the plain renders. The leave-one-out pairs teach the model the faults of renders through the original camera, and an eye-level render’s faults differ (stretching, surfaces seen edge-on), so both versions improve the score on what they were trained for without fixing the frames that matter. With about 90 poses to learn from, the critic invents texture instead of recovering it. Units whose art is steep (the SCV and Firebat, drawn from 44-49 degrees) need a better shape more than a better painter.