hyperdex-most-undertrained-slm

NanoDex-50M · 49,995,456 parameters · by @GGUFGuy

Progress —

Tokens seen

—

Step

—

Loss

—

Throughput

—

Loss curve

cross-entropy vs tokens seen

ClassLlamaForCausalLM

Layers12

Hidden size448

Attention heads8 (2 KV, head dim 56)

FFN size2669

Context512 tokens

Vocabulary2,048

Parameters49,995,456

Datasetfineweb-edu

Token budget—

OptimizerAdamW β=0.9/0.95 wd=0.1

Schedule2% warmup → cosine

Learning rate—

Best loss—

Worker—

Queued—

waiting…