tinyPIP-0

A ~111k parameter character-level transformer. Zero build step. Open this file in any browser.

Parameter calculator

Embedding dim (n_embd): Layers (n_layer): Heads (n_head): Context (block_size): Vocabulary size:

total parameters

MLP width is 4 x n_embd. Biases are off in linear layers. Output head is tied to the token embedding.

Parameter breakdown

token embedding (vocab x dim)
position embedding (block x dim)
attention qkv + proj (per layer)
MLP fc + proj (per layer)
layer norms (per layer)
final layer norm
total

Architecture

tokens  -> token embedding + position embedding
        -> [ LayerNorm -> causal self-attention -> residual
             LayerNorm -> MLP (GELU, 4x width) -> residual ] x n_layer
        -> final LayerNorm
        -> linear head (tied to token embedding)
        -> next-character logits

Run it

pip install torch
python train.py       # writes out/tinypip0.pt
python generate.py --prompt "ROMEO:" --tokens 300