Archives

A rapid refresher on the basics of statistics and how to apply them in Python.

A brief review of core concepts in linear algebra, with a focus on applications in machine learning.

A rapid review of the Pandas library for the impatient data scientist.

Data loading is a critical part of training deep learning models. In this post, we'll explore the best practices for loading data into neural networks, with a focus on PyTorch.

Flash Attention is a new attention mechanism that can be used to speed up training and inference in large-scale AI models. This article provides an overview of Flash Attention and its applications in AI research and development.

In this post, we'll explore how to estimate the size of a Transformer model, including the number of parameters, FLOPs, peak memory footprint, and checkpoint size.

My solutions to srush's AutoDiff Puzzles. This is useful as a quick refresher for computing gradients.

A review of backpropagation, the workhorse of deep learning.

A refresher on the fundamentals of computer science.

My solutions to srush's GPU Puzzles

Large-scale neural network training is a critical component of modern machine learning workflows. This post provides an overview of the challenges and solutions for training deep learning models at scale.

A step-by-step illustration of how you can visualize and derive the Jacobian of a matrix product, using a concrete example.

Scaling laws have been a cornerstone of deep learning research for decades. In this post, we'll explore the history of scaling laws in deep learning, and how they've evolved over time.

A deep dive into the softmax, logsoftmax, and logsumexp functions, their gradients, and how they relate to each other.

Should AI have "Personality" or will "Bland and Mid" suffice?

A highly compressed quide to quickly refresh (mostly) all you need to know about attention mechanisms in deep learning.

Organizing research presumably related to OpenAI's o1 reasoning model.

An in-depth exploration of the Byte Pair Encoding (BPE) tokenizer, its mechanics, and its applications in natural language processing.

Learn how to automate your VPS application deployment process using GitHub Actions for a seamless CI/CD experience.

My solutions to srush’s tensor puzzles. This is useful as a quick refresher for things you can do with vanilla tensor operations.