Posts

All the articles I've posted.

• GPU Puzzles

  Published:Dec 23, 2024 |  at 05:32 AM |  in 10 min read

  My solutions to srush's GPU Puzzles

• Computing the Jacobian of a Matrix Product

  Published:Dec 23, 2024 |  at 03:31 AM |  in 6 min read

  A step-by-step illustration of how you can visualize and derive the Jacobian of a matrix product, using a concrete example.

• AutoDiff Puzzles

  Published:Dec 20, 2024 |  at 05:32 AM |  in 19 min read

  My solutions to srush's AutoDiff Puzzles. This is useful as a quick refresher for computing gradients.

• Back to Backprop

  Published:Dec 12, 2024 |  at 06:42 AM |  in 29 min read

  A review of backpropagation, the workhorse of deep learning.

• Estimating Transformer Model Properties: A Deep Dive

  Published:Dec 7, 2024 |  at 05:44 AM |  in 8 min read

  In this post, we'll explore how to estimate the size of a Transformer model, including the number of parameters, FLOPs, peak memory footprint, and checkpoint size.

• Neural Scaling Laws (Then and Now)

  Published:Dec 3, 2024 |  at 04:07 AM |  in 12 min read

  Scaling laws have been a cornerstone of deep learning research for decades. In this post, we'll explore the history of scaling laws in deep learning, and how they've evolved over time.