hyperdex-test

NanoDex-1M · 1,000,224 parameters · by @GGUFGuy

Progress —

Tokens seen

—

Step

—

Loss

—

Throughput

—

Loss curve

cross-entropy vs tokens seen

Architecture

ClassLlamaForCausalLM

Layers5

Hidden size96

Attention heads6 (2 KV, head dim 16)

FFN size472

Context512 tokens

Vocabulary2,048

Parameters1,000,224

Run

Datasetfineweb-edu

Token budget—

OptimizerAdamW β=0.9/0.95 wd=0.1

Schedule2% warmup → cosine

Learning rate—

Best loss—

Worker—

Queued—

Logs

waiting…