What has my attention.
Did anyone see this coming? I sure didn’t. I thought “AI” was a sci-fi technology, possibly, not even possible.
I have been using it since ChatGPT was released, and the progression has been incredible. This amazing emerging field has captured my attention like nothing ever has. I see a future where AI is a positive force, but only if we make it so. I am dedicated to making it so. I am dedicated to answering the question “How?”
AI memory systems
How AI systems remember: recall systems and knowledge bases today, and a future where memory may be stored in calculation and compute, not in memory space.
Classifier and linear probing techniques
Reading a model from the inside with simple classifiers: what they can do, what it takes to trust one, and what a safety classifier taught me.
The shape of data itself
How the shape of data decides what can be learned from it: rows that are not examples, labels that leak, splits that cut both ways, and the one fine-tune I ran.
Other questions I’m asking
- Once we solve the bandwidth bottleneck, where will the next one be?
- How long will it take before we design new hardware that allows new math?
- How would we approach a real signal that factually argues against ‘just a tool’?
- How small can a model really be?
- What is the least amount of data that can represent a concept?
- Is there a ‘natural way’ that embeddings organise themselves during training?
What I’m writing about
- Personal experiences and findings over billions of tokens used.
- AI benchmarks and architectural ideas
- What does a post AGI world look like?
- How can we make the answer “a utopia”?
- Tooling and methods for AI assisted coding