PixelGPT 24×24 — Small-scale generative model research

01

Problem

How much can a small model learn when the image space has a fixed resolution and palette? PixelGPT generates directly in palette-index space. Each of the 576 predictions maps to one pixel on the final canvas.

02

Dataset

The original research collection pairs native 24×24 images with captions and five-color palettes. The training script loads indexed pixels and palettes, aligns MiniLM caption embeddings with the images, and creates a seeded validation split.

For distribution, the release pipeline checks dimensions and palette size, excludes flagged samples, assigns public categories, and exports deterministic splits. The public subset is rebalanced across categories, with near-duplicates grouped within splits to reduce leakage. The approximately 218K examples refer to the training run. The two published dataset counts are below.

03

Model architecture

A 14-layer decoder-only transformer (width 512, 8 attention heads) predicts one of five palette indices per position. A 384-dimensional MiniLM embedding and the chosen RGB palette condition each block through adaptive normalization.

The sequence follows a serpentine scan over 4×4 blocks. Inference uses classifier-free guidance, temperature, and top-p sampling, then maps the sampled indices back to the 2D grid.

04

Training

The project README documents an approximately 30-minute run on one NVIDIA RTX 5090 with about 218K examples. This is a record of that run.

The PyTorch trainer uses teacher forcing and next-token cross-entropy on palette indices. Its validation loop evaluates held-out token loss and saves sample grids for fixed caption and palette conditions.

05

Evaluation

The training code evaluates token loss on a held-out split. I also inspected generated grids across prompts and palettes. The examples below illustrate that qualitative check. A formal image-quality or prompt-adherence benchmark is unavailable.

The local app accepts a seed, prompt, palette, and sampling settings so readers can reproduce and compare generations on the same runtime.

Saved model outputs

Selected grids from local inference runs. The labels show the original English prompts.

06

Inference & deployment

The GitHub release runs the FP16 generator and MiniLM locally in a Python/FastAPI process. The browser canvas calls the local API. /api/generate returns the complete image, and /api/generate/stream sends partial grids as NDJSON every 24 steps.

The hosted canvas is a separate client-side port. A Web Worker runs ONNX Runtime Web with WASM, an INT8 decoder, and a KV cache. It returns progressive pixels to the canvas without calling a generation server. The initial model download and CPU sampling can take time.

07

Results & distribution

The source, local inference app, and public dataset are available to inspect. The complete dataset is sold separately and has paying customers. The public subset recorded 500+ downloads in the preceding month as of November 7, 2026.

08

Limitations & learnings

The resolution and palette limit fine detail, and the text encoder works best with short, concrete object prompts. The 576 sequential predictions still impose a latency cost, even with caching.

Using palette indices makes output constraints explicit and lets the editor recolor sprites without rerunning inference. For another narrow task, I would start by testing which output structure can be built into the representation and which quality checks require held-out examples.

09

Resources

Open the browser demo, inspect the Python implementation, or explore either dataset release.