nanodex-100k-1m

NanoDex-100K · 100,064 parameters · by @GGUFGuy

Progress —

Tokens seen

—

Step

—

Loss

—

Throughput

—

Loss curve

cross-entropy vs tokens seen

Architecture

ClassLlamaForCausalLM

Layers2

Hidden size32

Attention heads2 (1 KV, head dim 16)

FFN size147

Context512 tokens

Vocabulary2,048

Parameters100,064

Run

Datasetfineweb-edu

Token budget—

OptimizerAdamW β=0.9/0.95 wd=0.1

Schedule2% warmup → cosine

Learning rate—

Best loss—

Worker—

Queued—

Logs

waiting…