AFTER HOURS / THE LAB
Ideas in
the making.
You found the side room. This is where I explore, test, and learn. Everything here is still in progress.
WIKILM / UNDER CONSTRUCTION
thenextideaā
A model.
From first principles.
PERSONAL RESEARCHIn progress
WikiLM
A decoder-only language model being developed from scratch, exploring the path from raw Wikipedia text to a custom tokenizer and Transformer.
PythonTransformersByte-level BPE
In development. Training results and benchmarks are not yet available.
WHAT IāM EXPLORING
From text to predictions.
- A reproducible pipeline for preparing the training corpus.
- A custom byte-level byte-pair encoding tokenizer.
- A hand-written decoder-only Transformer architecture.
- Single-GPU training with mixed precision and gradient accumulation, targeting a 6GB VRAM budget.
NEXT ON THE BENCH
Train. Evaluate. Iterate.
Evaluate held-out perplexity and generated text samples, then document the results. The memory budget is a target, not a measured result.
There is no public repository or demo to share yet.
Return to the portfolio ā