useless-parameters

GPT-2 Small · 114,838,272 parameters · by @GGUFGuy

Progress —

Tokens seen

—

Step

—

Loss

—

Throughput

—

Loss curve

cross-entropy vs tokens seen

ClassLlamaForCausalLM

Layers12

Hidden size768

Attention heads12 (12 KV, head dim 64)

FFN size3072

Context512 tokens

Vocabulary2,048

Parameters114,838,272

Datasetfineweb-edu

Token budget—

OptimizerAdamW β=0.9/0.95 wd=0.1

Schedule2% warmup → cosine

Learning rate—

Best loss—

Worker—

Queued—

waiting…