FFN width: 1,344 · automatic

Trainable parameters

1 transformer block
13,354,496
Input token IDs
Token embeddings
parameters
Transformer block× 1
Normalization
Multi-head attention
parameters
+
Normalization
Feed-forward network
parameters
+
One block, total
Attention FFN Norms
Final normalization
Language-model head
parameters
10,000 next-token logits per position

Counts follow the model above. Select a layer to see its calculation.