Original Tweet: https://x.com/karpathy/status/2030705271627284816

I stated my prod run about 40 min ago. its working

I just did a test run with local qwen coder and getting some ok results. going to try gemma 4. I provide more instructions in the markdown than to claude.

Are you training LLM the example from the repo or something else? So far I'm just testing the example

Here is the complete list of changes from the base (Kaparthy's original) Hardware compatibility (T4 vs H100) • Replaced Flash Attention 3 kernel with

F.scaled_dot_product_attention

— T4 doesn't support the custom FA3 kernel •

WINDOW_PATTERN

changed from

"SSSL"

to

"LLLL"

— sliding window attention requires Flash Attn 3 •

DEVICE_BATCH_SIZE

reduced from 128 → 16 to fit T4's 16GB VRAM Training loop fixes Fixed

train_loss

to accumulate correctly across all

GRAD_ACCUM_STEPS

micro-batches — original only captured the last micro-batch loss • Added a 3-step warmup probe at startup to dynamically estimate

total_steps

for the LR schedule — original assumed a fixed step count Dataset • Pointed data loading at TinyStories

data/

directory instead of the original dataset Agent-discovered at runtime (not our code changes) •

TOTAL_BATCH_SIZE

reduced from

2^17 → 2^14

across three consecutive experiments — smaller batches = more optimizer steps in the 5-minute window = better convergence on a time-budgeted T4 run