Archives

A personal reflection on how AI has fundamentally changed the way I approach work, ideas, and ambition.

In this post, I swirl my own thoughts on the very topic of swirling thoughts in the context of AI, LLMs and their potential for novel hypothesis generation. I ground this discussion in some of the other ongoing parallel lines of thoughts/investigation including GRPO, reinforcement learning, the future of science and scientific peer review and the notion of being able to "purchase" scientific innovation.

How OpenAI's DeepResearch reveals what industry labs aren't sharing about next-gen AI assistants

Converting chat LLMs into reasoning agents through GRPO and reinforcement learning from execution feedback — achieving performance improvements on Project Euler's algorithmic challenges with multi-step reasoning, tool use, and code execution.

A brief review of core concepts in linear algebra, with a focus on applications in machine learning.

A rapid refresher on the basics of statistics and how to apply them in Python.

A rapid review of the Pandas library for the impatient data scientist.

Data loading is a critical part of training deep learning models. In this post, we'll explore the best practices for loading data into neural networks, with a focus on PyTorch.

In this post, we'll explore how to estimate the size of a Transformer model, including the number of parameters, FLOPs, peak memory footprint, and checkpoint size.

Large-scale neural network training is a critical component of modern machine learning workflows. This post provides an overview of the challenges and solutions for training deep learning models at scale.

Scaling laws have been a cornerstone of deep learning research for decades. In this post, we'll explore the history of scaling laws in deep learning, and how they've evolved over time.

A deep dive into the softmax, logsoftmax, and logsumexp functions, their gradients, and how they relate to each other.

A step-by-step illustration of how you can visualize and derive the Jacobian of a matrix product, using a concrete example.

Flash Attention is a new attention mechanism that can be used to speed up training and inference in large-scale AI models. This article provides an overview of Flash Attention and its applications in AI research and development.

My solutions to srush's AutoDiff Puzzles. This is useful as a quick refresher for computing gradients.

A review of backpropagation, the workhorse of deep learning.

A refresher on the fundamentals of computer science.

My solutions to srush's GPU Puzzles

o3 and Human-AI scientific collaboration

Should AI have "Personality" or will "Bland and Mid" suffice?

A highly compressed guide to quickly refresh (mostly) all you need to know about attention mechanisms in deep learning.

Organizing research presumably related to OpenAI's o1 reasoning model.

My solutions to srush’s tensor puzzles. This is useful as a quick refresher for things you can do with vanilla tensor operations.

An in-depth exploration of the Byte Pair Encoding (BPE) tokenizer, its mechanics, and its applications in natural language processing.

Learn how to automate your VPS application deployment process using GitHub Actions for a seamless CI/CD experience.