What actually happens when you press the button

Architecture

A standard modern decoder-only transformer — LlamaForCausalLM: SiLU MLPs, RMSNorm (ε=1e-5), rotary position embeddings (θ=10000), grouped-query attention, tied input/output embeddings, no biases. The four tiers are that same recipe, narrowed and shortened until it fits the parameter budget.

Tokenizer

A byte-level BPE with 2,048 tokens, trained on fineweb-edu itself (~2.7 characters per token). This matters more than it sounds: a typical small-LM vocabulary of ~49k tokens would cost 28M embedding parameters on its own — more than three times the largest model here. A lean vocabulary is what makes "a 500k-parameter language model" a sentence that means something. The parameter counts above are real totals, embeddings included.

The vocabulary is deliberately the same for every tier, and stays at 2,048 even for the 8M model. Two reasons: at a 500k total budget anything larger would be almost entirely embeddings, and a shared vocabulary is the only thing that makes the losses on the leaderboard comparable at all — change the tokenizer and cross-entropy stops meaning the same thing between two models.

Data

HuggingFaceFW/fineweb-edu, config sample-10BT — filtered educational web text. Instead of every run re-streaming and re-tokenizing the corpus, this site tokenizes a 100.00B-token slice once into a flat uint16 file (~186.3 GB) that all runs read from, so the workers are never waiting on data. Building it is a one-time cost per container; a run that asks for more tokens than are cached yet waits for the writer to catch up, and says so in its log. Every run starts at its own random offset, so two identical configurations don't see identical token order.

Training

Queue

One worker thread per available device claims the oldest queued run atomically, so two workers can never grab the same job. This deployment currently has 4 GPU workers, which means 4 runs can train at once. Estimated times shown in the wizard start from a prior and then recalibrate from the median throughput of runs that actually finished here.

Publishing

When a run finishes you can push it to your own Hugging Face namespace with the OAuth token from your session: config.json, model.safetensors, tokenizer files, generation_config.json, a training_run.json recording the complete recipe, and a generated model card. It loads with transformers like any other model on the Hub.

Durable storage

A Space's filesystem is wiped on every rebuild and restart, so finished runs, their loss curves and the token cache are mirrored to a Hugging Face bucket. The archive is append-only and keyed by run id — a finished run is written once and never mutated, which is why a rolling deploy running two containers at once cannot corrupt it. History comes back on boot; weights are pulled on demand the first time somebody opens the playground.

Runs that are mid-training when a container dies are not recoverable — the optimizer state only ever exists in that process. They come back marked interrupted.

Deliberately over-trained

1.5 billion tokens into an 8M-parameter model is roughly 9× past the Chinchilla compute-optimal ratio of 20 tokens per parameter — and into a 500k-parameter model, over 150× past it. That's intentional. Tiny models keep improving long after the compute-optimal point, and the goal here was never FLOP efficiency — it's watching cross-entropy fall from 7.6 (uniform noise over 2,048 tokens) toward 4, and seeing a network that started as pure randomness begin to emit English words.

Nothing trained here is a useful assistant, and nothing it says is factual. That was never the point.