GPU Brrr Reading List

I’ve found myself reading articles, blogs, papers and lecture notes across $\{\text{ML}, \text{SWE}, \text{Maths}\}$ as a form of passive learning, then turning the useful bits into implementation later.

This page is just the GPU/performance side of that habit. How do we make the GPU go brr? The answer?

1. Understand the expensive bit of hardward
2. Measure and profile
3. Attack wasted movement, launches and idle time, keep everything close together
4. Fuse it back together.

Act 1: The GPU Hierarchy

Three hierarchies get taught together because they interact constantly: execution hierarchy, hardware hierarchy and memory hierarchy.

• Execution: grid, blocks, warps, threads.
• Hardware: GPU, SMs, tensor cores / CUDA cores.
• Memory: registers, shared memory / SRAM, L2, HBM.

Small mental note: each block runs on an SM (Streaming Multiprocessor, think like a CPU); the GPU chip is the grid. Within the hierarchy, registers are closest/fastest, then shared memory, then global memory/HBM.

Core idea: compute is cheap; moving data is expensive. Most of the resources below are different ways of moving fewer bytes, moving smaller bytes, reusing bytes more often, or keeping the GPU busy while bytes are moving.

Resources

• GPU Execution Model - Modern GPU Programming For MLSys - concise overview of the GPU execution model.
• Modern GPU Programming for ML Systems - full course/book-style path for GPU programming in ML systems.
• Give Me 30 min, I’ll Make CUDA Click Forever - CUDA mental model video.
• CUDA MODE lectures - CUDA/GPU programming lectures.
• CUDA MODE lecture 001 materials - lecture materials from the replay dump.
• Stanford CS149 Lecture 1: Why Parallelism? Why Efficiency? - parallel-computing context.

Act 1.5: The GPU Is a Moving Target

There is not one GPU. Each generation moves the numbers: HBM bandwidth and capacity, tensor-core throughput, supported precision formats and async data-movement machinery.

That matters because the best optimization depends on the hardware. Ampere, Hopper and Blackwell do not have the same precision formats, memory hierarchy details, or kernel sweet spots.

Resources

• Modern GPU Programming for ML Systems - useful for the generation-by-generation hardware context.
• Some matrix multiplication engines are not as accurate as we thought - hardware/runtime numerics are part of performance engineering.
• AI Systems Performance Engineering - book reference for AI performance engineering.

Act 2: Performance Models and Rooflines

Before touching a kernel, identify which resource is limiting it:

• Compute: math units are saturated.
• Memory: math units are waiting for bytes.
• Overhead: launches/setup/synchronization dominate.

The roofline model is the main picture: low arithmetic intensity lives on the memory-bandwidth diagonal; high arithmetic intensity hits the compute ceiling. A lot of deep learning optimization is pushing work up and right by increasing reuse or avoiding materialization.

Resources

• Making Deep Learning Go Brrrr From First Principles - roofline model, arithmetic intensity, and the base mental model for “why isn’t this faster?”
• How To Scale Your Model - the JAX scaling book; systems view of LLMs, rooflines, parallelism and large-scale training economics.
• Transformer Inference Arithmetic - forward-pass and KV-cache arithmetic for inference.
• Machine Learning Systems - broad ML systems book/reference.
• A Hitchhiker’s Guide to ML Training Infrastructure - broad overview of training infrastructure and hardware acceleration.

Act 3: Profiling, torch.compile and Fusion

The first optimization baseline is measurement. After that, the first automatic optimization is often compilation/fusion: capture the graph, remove Python overhead, fuse operations, and avoid HBM round-trips for intermediates.

The catch: graph capture is shape-sensitive. Variable sequence length, video resolution, frame count and batch shape can turn “free speedup” into repeated recompilation unless shapes are bucketed or handled deliberately.

Resources

• Profiling in PyTorch, Part 1: A Beginner’s Guide to torch.profiler - getting useful traces out of PyTorch.
• Profiling in PyTorch, Part 2: From nn.Linear to a Fused MLP - profiling through to fusion.
• GPU MODE Lecture 1: How to profile CUDA kernels in PyTorch - Nsight/PyTorch profiling notes.
• Aritra on X: Profiling deep learning layers - thread on layer profiling.
• Making GPUs Actually Fast: A Deep Dive into Training Performance - Jane Street video on training performance.
• JINO-ROHIT/ml-systems-notes - notes around Torch, distributed systems and ML systems.

Act 4: Custom Kernels and FlashAttention

Do not materialize large intermediates in HBM if they can be tiled, streamed, fused or consumed immediately. Do as much as you can in SRAM, or the shared memory.

Matmul is the warmup example: naive global-memory reads, then coalescing, shared-memory tiling, register tiling, and eventually more hardware-specific tricks like async copies and warp specialization.

FlashAttention is the attention version of the same idea: never build the full $N \times N$ score matrix in HBM. Tile the computation, keep the online softmax state, and stream blocks through fast memory.

Kernel Resources

• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance - Simon Boehm’s CUDA matmul worklog.
• Fast matrix multiplication on CPU - useful contrast with GPU performance thinking.
• Triton vector addition tutorial - first small Triton kernel.
• Triton Puzzles - practice problems for Triton.
• Hugging Face Kernel Builder - practical workflow for custom kernels.
• Lecture 106: Hugging Face Kernels - video on Hugging Face kernels.
• My first Multi-GPU kernel: Writing All-to-all for AMD MI300X - multi-GPU all-to-all kernel writeup.
• Making video go BRRRR - video performance thread.

FlashAttention Resources

• FlashAttention paper - attention as a memory-bound problem.
• FlashAttention Triton implementation - source-level reference.
• Flash Attention from Scratch Part 1 - implementation-oriented explanation.
• flash-attention-jax - JAX implementation.
• Causal FlashAttention in JAX - specific causal attention file.

Act 5: Multi-GPU Training and Communication

Once one GPU is not enough, the memory hierarchy extends across devices. Moving data between GPUs is another expensive rung, so the same “move less / overlap more” principle returns.

Core vocabulary:

• Data parallelism: replicate model, split data, all-reduce gradients.
• Tensor parallelism: split layer math across devices.
• Pipeline parallelism: split depth into stages; watch for bubbles.
• FSDP / ZeRO: shard parameters, gradients and optimizer states.
• Collectives: all-reduce, all-gather, reduce-scatter.

Resources

• Stanford CS336: Language Modeling from Scratch - broad spine for language-model systems.
• CS336 lecture 2 trace - PyTorch/einops trace from the course.
• CS336 Lecture 2: PyTorch/einops - video lecture.
• The Ultra-Scale Playbook - Hugging Face/Nanotron guide to large-scale training.
• Ultra-Scale Playbook: Gradient Accumulation - specific section from the replay.
• Ultra-Scale Playbook: Kernels - specific section from the replay.
• Smol Training Playbook - practical training setup and parallelism choices.
• Megatron-LM pipeline parallelism - pipeline bubbles and large-scale transformer training.
• Jino Rohit’s collective communication notes - communication primitives and distributed performance intuition.
• Transport Muon: Beating Muon in Speed and Performance in 1 Newton Step - optimizer/performance reading.

Act 6: Inference and Serving

Serving has a different performance shape from training. You are juggling requests, the KV cache, prefill/decode split, batching and scheduling.

• Prefill: lots of prompt tokens in parallel, often compute-heavy.
• Decode: one token at a time, repeatedly reading KV cache, often memory-bound.
• PagedAttention: treat KV cache like paged virtual memory to reduce fragmentation and increase concurrency.

Resources

• Fast LLM Inference From Scratch - arithmetic and performance constraints for LLM inference.
• PagedAttention - KV-cache paging as virtual memory.
• Popping the GPU Bubble - pipelined decoding and idle compute.
• Local LLM Inference Optimization: The Complete Guide - practical local inference optimization guide.

Act 7: Quantization, Smaller Bytes and the Right Kernel

If a workload is memory-bound and you cannot move fewer values, move smaller values. Quantization reduces memory footprint and bandwidth demand, but it changes the accuracy and kernel-choice story.

The right kernel depends on the workload. Dequant-then-compute kernels can be good for memory-bound decode; native low-precision GEMM can be better for compute-heavy prefill on new hardware.

Resources

• Gemma 4 QAT - Unsloth collection - quantization-aware training model collection.
• Gemma 4 QAT docs - Unsloth documentation for QAT models.
• unsloth/gemma-4-26B-A4B-it-qat-GGUF - GGUF model repo.
• The Magic of LLM Distillation - distillation talk, useful background for compression.
• Tim Dettmers - quantization and efficient training/inference writing.

Act 8: monokernels and killing boundaries

if kernel boundaries, launch overhead and stragglers waste time, one extreme answer is to fuse much more aggressively. A monokernel/megakernel tries to keep the GPU busy by removing boundaries, loading ahead, and avoiding idle bubbles.

Resources

• Designing a Monokernel - Hazy Research post on no-bubbles/monokernel design.

Compiler and Runtime Side Quests

Not everything is a Transformer. Tree inference and compiler/runtime work are also part of the performance map when the theme is “make model execution faster.”

• lleaves article - compiled LightGBM inference.
• Simon Boehm’s blog - CUDA, compiler and performance posts.

People and Feeds

People whose posts tend to feed this GPU/performance list.

• Simon Boehm - performance, inference and compiler-flavoured ML systems.
• Horace He - performance models and PyTorch internals.
• Tri Dao - attention, sequence models and systems-aware algorithms.
• Tim Dettmers - quantization and efficient training/inference.
• Mark Saroufim - PyTorch, kernels and production ML systems.
• Jino Rohit - distributed systems, Torch and collective communication notes.
• Ali Taha - model performance and video performance threads.
• Thien Tran - GPU kernels and systems notes.
• Daniel Han Chen - Unsloth / model efficiency.

Musings on Sparse Tables and Associativity

While learning about sparse tables, I found something quite insightful; the ideas behind binary lifting, where we we compute jump pointers to the nth ancestor, we break it up as the n/2th ancestor of the n/2th ancestor for a node. i.e ancestor(node, n) = ancestor(ancestor(node, n/2), n/2). We can go further but more importantly, why does this structure look familiar?

If we think back to binary exponentiation, we can write $a^n = (a^{n/2}) * (a^{n/2})$ assuming $n$ is even, and then in general:

python
def fast_exp(a, n):
    if n == 0:
        return 1 
    if n == 1:
        return a 
    else: 
        res = 1
        b = fast_exp(a, n//2)
        res *= b*b 
        if n % 2 == 1:
            res *= b 
        return res

Or in ocaml:

ocaml
let fast_exp base exp = 
    let rec fast_exp_aux acc b e =
    if e <= 0 then acc 
    else if e mod 2 = 0 then fast_exp_aux acc (b*b) (e/2)
    else fast_exp_aux (acc*b) (b*b) (e/2) 
    in fast_exp_aux 1 base exp

Notice that we quickly turn the problem into a logarithmic problem. This is actually a feature of associativity, believe it or not! In general, for a semigroup $S$ and binary operation $*: S \times S \to S$, when we write $a_1 * a_2 * \cdots * a_n$, associativity gives us that $a_1 *(a_2 * \cdots * a_n) = (a_1 * \cdots a_{n-1}) *a_n$. This is a signal that the order in which you evaluate doesn’t matter, so be aggressive and split the domain in half!

The Sparse Table

That brings us to the humble Sparse Table. For problems where we are given an array $A$, queries $Q = \{ [L_i, R_i]\}_{i=1}^{i=L}$ and a binary operator $*$, where we can say that $A \subset S$ and $(S, *)$ forms a semigroup, we can speed up the computation of time here. I will use $f(A[L,R]) = a_L * \cdots * a_R$ as a form of notation here to basically mean applying our binary operator across the range.

Naively, you can just run that for each query, which would mean $Q$ queries that could be $O(N)$ steps, which means the time complexity here is $O(N*Q)$, on the assumption that $*$ is a constant time operation. Not amazing. We ask ourselves, what can easily be computed? And can we reuse these building blocks?

Yes you can. With powers of 2, we can break them up into smaller and smaller intervals. The diagram above tells us we have a natural relation:

In english, this means we can compute the property of a $2^k$ length interval by splitting it up into two evenly split $2^{k-1}$ sized intervals. And in code:

python
N = len(array)
K = ceil(math.log2(N))
memo = [[array[idx] if layer == 0 else 0 for idx in range(N)] for layer in range(K+1)]
for k in range(1, K): # O (log N )
    for idx in range(0, N): # O(N)
        if right := idx+int(2**(k-1)) < N:
            memo[k][idx] = f(memo[k-1][idx], memo[k-1][right])

In $O(N \log N)$ steps, which is great, but not quite there for some $n\in \mathbb{N}$ that isn’t a power of $2$. We know that $n = \sum_{i \in I} 2^i$, that is we can read off it’s binary representation to form a set of disjoint intervals that are powers of $2$ to get our result, which is $O(\log N)$. Recall that (1 << i) & n tells you whether the ith bit of n is on, so then we simply iterate over:

python
# Assume L is defined 
value = None 
query_len = R-L+1
for k in range(K):
    if (1 << k) & query_len: # bit hit 
        if value is None:
            value = memo[k][L]
        else:
            value = f(value, memo[k][L])
        L += int(2**k)

And thus, we have a really cool way of remembering how sparse tables work. To summarise:

• Compute intervals in powers of 2 in logarithmic time thanks to associativity
• Compute general interval lengths because of the binary representation

Also, it turns out, binary lifting is really just a sparse table implementation, but on trees. The binary operation in this case, is function composition, e.g $f \circ f \circ \cdots \circ f = f^n = f^{n/2} \circ f^{n/2}$, because you’re doing ‘jumps’. Neat!

Idempotency

Idempotency is the idea that for some operation $*$ we have that $x * x = x$, i.e there is no change to the operation if applied repeatedly. In real world systems, such as payment systems, idempotency is a good property to ensure you don’t get double charged. It’s because of idempotency, that we can go from $O(\log N)$ look-up to constant time look-up. In this case, we don’t really mind if there’s overlap in our intervals, so just pick the two intervals that tightly cover our desired query range $[L,R]$. More concretely, I claim that its given by this formula.

Which operations are idempotent though?

• minimum, maximum
• GCD, LCM

There are probably more, but note that these operations all form a semigroup. If our function doesn’t change under repeated application, this means we can reduce the time for a query to constant time, by picking $i = \lfloor{\log_2{R-L+1}} \rfloor$, then computing $f(\text{memo}[i][L], \text{memo}[i][R-2^i +1])$. We illustrate this with a diagram.

Bringing us to $O(Q)$ query time, and $O(N \log N+Q)$ total time complexity.

Prefix Sums

It turns out when your operation has more structure, e.g an inverse (so you have a group structure), you can actually use an easier technique, prefix+suffix sums. Implicitly, you need the idea of being able to cancel/invert an operation. For range sum queries, it suffices to instead compute the prefix and suffix sums, so that when you want $\sum A[L,R]$, you can simply look at sum(l,r) = prefix[R]-suffix[L-1].

Problems

There are less problems on Leetcode that cover this format, so we head to the more competitively aligned programming sites. In general, I find CodeChef questions extremely confusing to read, and preferred the problems from CSES.

Reflections

Python sucks for type checking, as it really lets you do stupid things that won’t raise an error, e.g indexing with a float (when raising to an exponent) won’t be immediately caught. Aside from this, once I had the intuition of associativity, I was mostly able to code up the implementation of a sparse table then recall that you can use (1 << i) & n to see if the ith bit was set. I did run into some intuition issues where I was computing the wrong ranges, but a good diagram really helps here. Additionally, other sites TLE even though my solution is within complexity :( - it’s easy enough to rewrite it in C++ fortunately as we don’t rely on any magical Python data structures.

Also, did you know Excalidraw has libraries that you can import from? For instance I can import a System design library for icons that I would have otherwise crudely drawn, or for DSA!

I found a relevant Codeforces article that outlines the insight that I found as well, while also generalising the idea to any monoid. I did mention that you can do this with a semigroup, but a monoid with identity makes this a bit nicer because you don’t need to deal with the weird empty edge case, and/or default initialise your values.

What’s next?

I want to learn about Fenwick trees and Segment trees, as they go further, and allow you to work with modifications to your data structure. I suspect that some properties of a monoid may help here. I found that the easiest way to think about learning how a sparse table works is through understanding the underlying algebraic requirements for it to work. It feels like a motivator for why category theory matters here, and why we even compute a sparse table to start within.

Resources

• cp-algorithms - Sparse Table
• sesalt.sh - 1 DS|A a day - Sparse Table

During Yakuza 0, Kiryu decides he wants to take over the real estate business in Kamurocho. Finishing the minigame means acquiring all the property, or shaking down Mr Shakedown. However, not all of the property you buy generate the same amount of cash, and you can often end up buying properties that cost more to run than they yield profit. In typical Sai-fashion, I wanted to know how Real Estate Royale actually pays out. The in-game UI tells you rank and share price, but not the underlying numbers, so I looked at the PC release files.

Approach

Roughly in order:

1. Index what’s on disk: file paths, sizes, which archive they came from. Paths live in games/yakuza0/profile.yaml.
2. Unpack: Gibbed’s tools for stay.par (minigames), boot.par (items/encounters), battle.par (combat dialogue and controller data).
3. Decode: most tables have a recognizable header (20 07 03 19 descriptors, *SB/*TB fixed rows, string_tbl string pools, and a handful of battle-specific magics). Output goes to data/tables/ as CSV + JSON.
4. Promote into a graph: interesting rows become entities with facts and links. Money Island was first; encounters and nawabari followed.

Locale files are parallel, not merged: _bin_c English, _bin_j Japanese, _bin_k Korean. Gameplay IDs stay stable; names come from whichever locale column you’re reading. In addition, Real Estate Royale is called Money Island.

Graph explorer

Blue nodes are Money Island. Pink is encounters, amber nawabari, green items. Gray blobs are decoded table families that haven’t been promoted yet. Click a node to see the raw fields.

What’s been cdecoded

Still sitting in JSON but not in the graph: the full item list, virtue shop, 374 activity entries, cabaret data. Thousands of other .par files in the install haven’t been touched.

Real Estate Royale

The payout table is money_island_shop.bin_c: 50 rows, five areas with ten properties each. Each row has profit_d through profit_s. Comparing columns across shops gives clean rank multipliers:

Disco City Boy in Entertainment is the highest base earner at ¥38.4M per round at D-rank (¥153.6M at S). Kamuro Soba in Gambling is the lowest at ¥320K.

All 50 properties from money_island_shop.bin_c, sorted by D-rank profit within each area. These are the numbers before rank upgrades: multiply by 1.5 / 2 / 3 / 4 for C / B / A / S.

Entertainment (芸能)

Gambling (賭博)

Recreation (娯楽)

Digital (電脳)

Adult (風俗)

Other stuff worth noting

Encounters: spawn groups in encounter_enemy_set, loot weights in encounter_drop_item. Street drop pool 2 rolls Chestnut Shochu (栗焼酎) at 28%.

Nawabari: tenant names live in a string_tbl indirection table, not inline in the row. 320 shop types, keyed by row index.

Battlepar: 41 binaries decoded: 127 fighter controller entries (Kiryu through the zako archetypes), 568 wanderer speak symbols, 54 chase barks, 100 Coliseum endless arena fighters.

Conclusion

I found this to be a rather easy exercise in datamining, as the tooling mostly existed for it already. But it does bring a thought, where you could use Cheat Engine/PINCE to look at the memory in-game, then look around the game binaries to make datamining an easier effort, especially for games where precise information is less available.

Recursion is the idea of self-invocation until a base case is reached. Search algorithms like DFS fall out of it almost naturally, and when examined mathematically, it is just induction in disguise.

Uses

• Recursive data structures (linked lists, trees, graphs)
• Dynamic programming (memoised recursion over a state space)

Recursion as Mathematical Induction

The idea of induction is as follows. Suppose I have some statement $P(n)$ that I believe to be true over some domain, e.g. $\mathbb{N}$. The goal is to assume $P(n)$ holds for some $n$ and demonstrate that $P(n+1)$ follows. The standard cookbook:

• Prove the base case $P(1)$ holds
• Assume for $n = k$ that $P(k)$ is true (or for strong induction, all $n \leq k$)
• Use that to demonstrate $P(k+1)$ also holds

One useful approach (taught in graph theory) is to start from $P(k+1)$ and reduce it to $P(k)$, rather than building up. As an example:

Claim: A tree with $n$ vertices has $n - 1$ edges.

Proof by induction. A tree is a connected, acyclic graph. In the base case $n = 1$, a single vertex has no edges, so $P(1)$ holds.

Assume $P(k)$: a tree with $k$ vertices has $k - 1$ edges. Now suppose $T$ is a tree with $k + 1$ vertices. By a corollary (proved in the appendix below), every tree has at least two leaves; pick one, $v_1$. Deleting $v_1$ along with its single edge yields a tree $T'$ on $k$ vertices; no cycles form because we removed a leaf, and connectivity is preserved because a leaf is not a cut vertex. By $P(k)$, $T'$ has $k - 1$ edges, so $|E(T)| = |E(T')| + 1 = k$, which is what we needed to show. $\square$

The key pattern: we relied on the truth of a smaller case to construct the present one. Recursion works the same way. When approaching a recursive problem, ask:

• Base case: what does the smallest valid input look like?
• Sub-problem: what result from a smaller input do I need?
• Construction: how do I assemble my answer from that result?

The challenge in recursion is usually the construction step. In DP, the harder question is choosing the right state representation. Both are fundamentally induction.

Basic Examples

Linked list sum. Applying the three questions:

• Base case: empty list → 0
• Sub-problem: sum of the tail
• Construct: add the head value to the tail sum

ocaml
(* OCaml: pattern matching makes the structure explicit *)
let rec sum = function
    | [] -> 0
    | head :: tail -> head + sum tail

Coin change. Minimise the number of coins needed to make amount $n$ from denominations $C = \{C_1, \ldots, C_k\}$:

• Base case: $n = 0$ → 0 coins needed; $n < 0$ → impossible
• Sub-problem: the minimum coins for $n - C_i$ for each $C_i \in C$
• Construct: take 1 plus the minimum of those sub-problems

python
def coin_change(denoms, n):
    if n == 0:
        return 0
    return min(
        (1 + coin_change(denoms, n - c) for c in denoms if c <= n),
        default=float('inf')
    )

These examples track a single piece of state. Harder problems track multiple values, have more edge cases, or require careful thought about what the recursive call should return.

Leetcode: Twin Sum

Leetcode 2130: Maximum Twin Sum of a LinkedList: pair the first and last nodes, second and second-last, and so on, then return the maximum pair sum.

A naive approach (collect the first half into an array, then walk the second half) works but is inelegant. Three cleaner approaches:

• Reverse the first half in place and iterate together
• Hare-and-tortoise to find the midpoint, then reverse
• A zipper: descend recursively to the end, then walk the front pointer forward as the call stack unwinds

The zipper approach fell out most naturally to me. The key insight: on the way back up the call stack, we’re naturally iterating the back pointer from last to first, while the front pointer steps forward.

The recursive function carries the current front node and returns the updated front pointer together with the running maximum:

python
def recurse(front, back, max_val):
    if back.next is None:
        # base: at the last node, form the first pair
        return max(max_val, back.val + front.val), front.next
    best_val, next_front = recurse(front, back.next, max_val)
    # unwinding: back is now moving backwards; front advances via next_front
    return max(best_val, next_front.val + back.val), next_front.next

At every frame on the way up, back is one step closer to the front while next_front steps forward; they meet in the middle.

Practice Problems

Linked lists

• 234. Palindrome Linked List
• 206. Reverse Linked List
• 24. Swap Nodes in Pairs
• 25. Reverse Nodes in k-Group

Trees

• 104. Maximum Depth of Binary Tree
• 110. Balanced Binary Tree
• 124. Binary Tree Maximum Path Sum
• 687. Longest Univalue Path

Worked: Reverse Nodes in k-Group

The trick is to isolate the recursive leap of faith. Assume the suffix starting from position $k+1$ is already correctly reversed. Then all you need to do is reverse the current $k$-group and attach it to that sorted suffix: iterate curr → prev while threading curr → next.

Worked: Binary Tree Maximum Path Sum

This one tripped me up because you track two different things:

• Global best: the best path seen anywhere in the tree (updated at each node)
• Local best: the best continuous path rooted at this node that a parent can extend

Applying the template:

• Base case: null node contributes 0; a leaf is just its value
• Sub-problem: left and right subtrees each give a global best and a local best (one-sided path)
• Construct: try stitching L + node + R to update the global best; the local best is node + max(L, R, 0)

Keep your variable names precise: confusing l_best (global best from left subtree) with l_path (best path rooted at left child) is an easy way to get wrong answers.

Longest Univalue Path follows the same pattern, but the “stitching” condition becomes value equality rather than unconstrained addition. Watch your sentinel values: if node values can be negative, don’t use -1 as “impossible”.

How Rusty Was I?

As a form of self-humiliation, a log of the mistakes that actually happened:

• Used | instead of & in a boolean expression
• Missing brackets around an inequality comparison
• Forgot to return the recursive call (classic)
• Returned the whole tuple where only one element was needed
• In Swap Nodes in Pairs: head.next, head = head, head.next and head, head.next = head.next, head are not the same (Python evaluates the right-hand side first as a tuple)
• Struggled to see why you need a temporary when swapping
• The bracket placement in balanced = (abs(l_height - r_height) <= 1) & l_bal & r_bal
• Mixed up argument order in a recursive call
• Used the global best from the left subtree to compute the best continuous sum (these are different things)
• Used a sentinel of $-1$ for Longest Univalue Path when node values lie in $[-1000, 1000]$

Despite all that, a few problems was enough to shake the rust off. The main lesson: for trees, you often want to compute a global property while threading local subtree state upwards. Linked list problems often involve a second pointer tracking a different part of the list.

Appendix: Every Tree Has at Least Two Leaves

For a tree $T$ (connected, acyclic graph) with $n \geq 2$, we claim it has at least one leaf (a vertex of degree 1). Let $P = (u, v_1, \ldots, v)$ be a maximal path in $T$, i.e. one that cannot be extended at either end.

Because $P$ is maximal, no neighbour of $u$ lies outside $P$ (otherwise $P$ could be extended). And no neighbour of $u$ lies inside $P$ either: if some $s \in N(u)$ were adjacent to an interior vertex $v_i$, we could form a cycle through $s$, contradicting acyclicity. Both cases are illustrated below.

Therefore $\deg(u) = 1$, i.e. $u$ is a leaf. The same argument applies to $v$, giving at least two leaves. $\square$

Conclusion

Recursion, dynamic programming, DFS, and most tree/linked-list problems are all induction over a state space. Internalising that framing makes it much easier to construct solutions systematically: identify the base case, decide what the recursive call should return, and build the answer from there.

Overview

Page for any competitive programming algorithms and techniques I learn.

Edit Distance Revisited (1/10/2026)

A reminder, that for any dynamic programming problem, is that we can think of it as a mathematical optimisation problem, which is defined by

• Overlapping Subproblems
• Optimal Substructure

We think about the recurrence that links different states, and define a state space that’s helpful for our problem. In the case of edit distance, for strings $A$ and $B$, we defined the state $D(i,j)$ as the minimum edit distance between the prefixes $A[0:i]$ and $B[0:j]$. That way, our solution is simply $D(|A|, |B|)$.

We define the recurrence as:

Here, $\text{cost}(A[i], B[j])$ is $0$ if the characters are the same, and $1$ otherwise. If you think about the recurrence, cover the character at position $i$ and $j$ in $A$ and $B$ respectively. We can get to that point via insertion $(i,j-1)$, deletion $(i-1,j)$, or substituting a character $(i-1,j-1)$.

Trees

To motivate the many ways we can solve LCA, we start with a naive implementation.

Naive LCA

Binary Lifting

Think of this as an amazing teleportation technique. If we have the ability to ascend $2^k$ levels in one jump, then we can reach any ancestor in $O(\log N)$ jumps. After which, we can then trivially iterate on the remaining ones. As an example, if we want to go up $13$ levels, we can do $1+4+8$ jumps, which is $3$ jumps total. We look at the first ancestor of our node, then the $2^2$ ancestor of that node, and then the $2^3$ ancestor of that node.

All of this corresponds to looking at a table repeatedly until we get to our destination.

Resources

• Resource 1
• Resource 2

Overview

The study of classical mechanics involves understanding the motion of bodies under the influence of forces. We assume no relativistic effects are at play and study simple systems using Newtonian, Lagrangian, and Hamiltonian frameworks.

Learning Path

Progress through Taylor’s Classical Mechanics.

Resources

• Classical Mechanics - John R. Taylor

Newton’s Laws of Motion

We first establish the Cartesian coordinate system with basis vectors $(e_1, e_2, e_3)$ and position vector $\mathbf{r} = x e_1 + y e_2 + z e_3$ which is written as $(x,y,z)$. We have all the standard vector operations for $\mathbb{R}^3$ such as dot product $\mathbf{a} \cdot \mathbf{b}$ and cross product $\mathbf{a} \times \mathbf{b}$.

For instance, consider a force $\mathbf{F}$ acting about the origin (like a stone). The torque $\mathbf{\Gamma} = r \times \mathbf{F}$ is defined as such.

We then have a vector valued function $\mathbf{r}(t) = (x(t), y(t), z(t))$ which can be differentiated and integrated with minimal issue. Most importantly, we note that $\mathbf{r}(t) = x(t) e_1 + y(t) e_2 + z(t) e_3$, and so when we differentiate wrt time $t$, each term is constant, and so you have three terms. This is special to the Cartesian coordinate system, and doesn’t hold for the Polar Coordinate system.

First and Second Law of Motion

The First Law of Motion states that an object with no forces acting on it will maintain constant velocity, i.e $\mathbf{F} = 0 \implies \frac{d^2 \mathbf{r}}{dt^2} = 0$.

And more generally, for an object with mass $m$ and force $\mathbf{F}$ acting on it, we have the Second Law of Motion:

Recalling that momentum is defined as $\mathbf{p} = m \mathbf{v}$, we also have that $\mathbf{F} = \frac{d \mathbf{p}}{dt}$.

Inertial Frames

Note that the laws of motion only hold in inertial frames of reference, which are frames that are either at rest or moving with constant velocity. The easiest example of a non-inertial frame is an object moving inside a train that is accelerating. In such a frame, fictitious forces (like the Coriolis force) may appear to act on objects.

Third Law of Motion

For a pair of objects $A$ and $B$, the force exerted by $A$ on $B$ is equal in magnitude and opposite in direction to the force exerted by $B$ on $A$, i.e $F_{AB} = - F_{BA}$.

Now consider a system of two objects $A$ and $B$, we can write out their net forces as $F_A^{net} = F_{AB} + F_{A}^{ext}$ and $F_B^{net} = F_{BA} + F_{B}^{ext}$. Adding these two equations, we have:

If $F_{ext} = 0$, then we have $F_{A}^{net} + F_{B}^{net} = 0$, which implies that the total momentum of the system is conserved. Interestingly enough, the third law doesn’t even always hold up in classical mechanics, for instance, in electromagnetic interactions. Take two positive charges moving orthogonally to each other, the magnetic forces they exert on each other do not satisfy the third law by the Right Hand Rule.

Polar Newton

Recall $r^2 = x^2 + y^2$, $x = r \cos(\theta)$, $y = r \sin(\theta)$, and $\theta = \tan^{-1}(y/x)$. We have new unit vectors $\hat{r}$ and $\hat{\theta}$ defined as:

This follows from $\hat{r} =r(cos(\theta), sin(\theta))/r$ and $\hat{\theta}$ being orthogonal to $\hat{r}$ in the counter-clockwise direction. Trivially, swapping components and adding a negative sign achieves this.

Now that we have basis vectors that are not simple, we need to take care in differentiating. For a force $\mathbf{F} = F_r \hat{r} + F_\theta \hat{\theta}$ we want to determine what this is written as. Take a stone rotating around the origin on a string, $F_r$ is the tension in the string and $F_{\theta}$ is any air resistant perpendiular to the string.

To compute $\ddot{\mathbf{r}}$ so we can equate to $F_r/m$, need to differentiate:

Then similarly compute $\dot{\mathbf{\theta}} = - \mathbf{r} \dot{\theta}$. We can differentiate again to obtain the acceleration.

Example: Skateboard on a Half Pipe

Consider a Skateboard with mass $m$ on a half pipe of radius $R$. It’s dropped from rest at an angle $\theta$ from the center of the half-pipe. As the radius is constant, we can simplify the acceleration formula by dropping out the $\ddot{r}$ terms yielding $F_r = - m R \dot{\theta}^2$ and $F_{\theta} = m R \ddot{\theta}$. The forces acting on the skateboard are $F_r =mg \cos(\theta) - N$ and $F_{\theta} = - mg \sin(\theta)$ where $N$ is the normal force from the half-pipe. Looking at the $\theta$ equation, we have:

This is a nonlinear differential equation with no closed form solution. However, supposing that the skateboard is dropped near the center, we assume a small angle approximation $\sin(\theta) \approx \theta$, yielding $\ddot{\theta} = - \frac{g}{R} \theta$. This is a simple harmonic oscillator with solution Let $\omega = \sqrt{\frac{g}{R}}$, then we have:

Overview

Build foundational mental models for distributed training. Understand the why behind DP, FSDP, ZeRO, and sharding strategies before diving into implementation.

Goal: Develop intuition for when to use which parallelism strategy and understand the memory/communication trade-offs.

Key Concepts

Core Mental Models

1. Data Parallelism (DP)

The simplest distributed training pattern:

• Each GPU has a full copy of the model
• Data is split across GPUs
• After forward/backward, gradients are synchronized (AllReduce)

When it works: Small models that fit in GPU memory.

When it breaks: Model larger than GPU memory. Enter FSDP/ZeRO.

2. Fully Sharded Data Parallel (FSDP)

Extension of DP that shards model parameters:

• Each GPU holds only a shard of the model
• Parameters are gathered (AllGather) when needed for computation
• Gradients are reduced (ReduceScatter) after backward

Memory win: Model parameters divided by N GPUs.

Communication cost: More frequent AllGather/ReduceScatter operations.

3. ZeRO: Memory Optimization Stages

ZeRO progressively shards different training states:

• ZeRO-1: Shard optimizer states only (~4├ù memory reduction)
• ZeRO-2: Shard optimizer states + gradients (~8├ù reduction)
• ZeRO-3: Shard optimizer states + gradients + parameters (~N├ù reduction, equivalent to FSDP)

Trade-off: Higher ZeRO stages = more memory efficient, more communication overhead.

4. Sharding Strategies

Full Shard: Maximum memory savings, maximum communication (ZeRO-3/FSDP)

Hybrid Shard: Shard within nodes, replicate across nodes (balance memory & communication)

No Shard: Pure data parallelism (DP)

5. Communication Patterns

AllReduce: Sum gradients across all GPUs (DP)

AllGather: Gather sharded parameters before forward/backward (FSDP)

ReduceScatter: Sum gradients and shard result (FSDP)

Understanding these primitives is key to reasoning about distributed training bottlenecks.

Key Resources

📚 Essential Reading

Smol Training Playbook (HuggingFace)
https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook

Comprehensive guide to designing model architecture, choosing parallelism strategies, and understanding distributed training trade-offs. Start here.

Learning Path

1. Conceptual Understanding (4 hours)

   • Read Smol Training Playbook sections on parallelism
   • Draw diagrams of DP vs FSDP communication patterns
   • Understand when each strategy applies

2. Mental Model Building (6 hours)

   • Work through ZeRO stages: what gets sharded at each level?
   • Calculate memory savings for a sample model (7B params)
   • Map communication patterns to actual operations

3. Hands-On Exploration (5 hours)

   • Instrument a small model with torchrun + FSDP
   • Profile memory usage at different ZeRO stages
   • Measure communication overhead

Common Pitfalls

Γ¥î “FSDP is always better” ΓÇö No! Pure DP can be faster for small models due to lower communication overhead.

Γ¥î “ZeRO-3 = free memory” ΓÇö Communication cost increases. Understand the trade-off.

Γ¥î “Sharding is enough” ΓÇö You also need pipeline parallelism for very large models (coming in later nodes).

Next Steps

After mastering these mental models:

• • Fault Tolerance & Suspend/Resume: How to make distributed training reliable
• • Ultra-Scale Heuristics: Practical playbook for 5D parallelism + ZeRO interplay
• • Research Harness v2: Apply these concepts to a reusable training template

Assessment Criteria

✅ You understand this node when you can:

• Explain when to use DP vs FSDP vs ZeRO-3 for a given model size
• Draw the communication pattern for each strategy
• Calculate memory savings from sharding
• Articulate the trade-offs (memory vs communication)

Overview

Master the conceptual foundation of flow matching and continuous normalizing flows. Understand how to learn a trajectory from noise to data by training a vector field.

Goal: Build intuition for the ODE perspective on generative modeling and understand why flow matching is often simpler than diffusion.

Key Concepts

The ODE Trajectory Mental Model

Core idea: Generative modeling as learning a continuous path from noise $z_0 \sim \mathcal{N}(0, I)$ to data $z_1 \sim p_{\text{data}}$.

Define an ODE:

Where $v_\theta$ is a learned vector field.

Forward process: $z_0 \to z_1$ (noise to data)
Sampling: Solve ODE forward from $z_0$ to get $z_1$

Intuition: Instead of learning to denoise (diffusion), learn the velocity at each point along the path.

Flow Matching Objective

Goal: Train $v_\theta$ to match the true vector field $v_t$.

Conditional flow matching loss:

Where:

• $z_1 \sim p_{\text{data}}$: Real data sample
• $z_t = \alpha_t z_1 + \sigma_t \epsilon$: Interpolated sample at time $t$
• $u_t(z_t | z_1)$: Target vector field (analytically computable!)

Key insight: Unlike diffusion (which requires score matching), flow matching has a simple regression objective.

Flow Matching vs Diffusion Models

When to use flow matching:

• You want a simpler training objective
• You want to design custom interpolation paths
• You prefer deterministic generation

When to use diffusion:

• You want stochasticity for diversity
• You’re using existing diffusion codebases (Stable Diffusion, etc.)

ODE Solvers for Sampling

Euler method (simplest):

Runge-Kutta 4 (RK4) (more accurate, fewer steps):More complex multi-stage method, but allows larger $\Delta t$.

DPM-Solver (optimized for generative models):Exploits structure of learned vector fields for fast sampling.

Trade-off: Accuracy vs speed. Euler needs ~100 steps, RK4 ~50 steps, DPM-Solver ~20 steps.

Training Flow Matching Models

Algorithm

1. Sample data $z_1 \sim p_{\text{data}}$
2. Sample noise $\epsilon \sim \mathcal{N}(0, I)$
3. Sample time $t \sim U(0, 1)$
4. Interpolate: $z_t = \alpha_t z_1 + \sigma_t \epsilon$
5. Compute target: $u_t = \frac{d}{dt}(\alpha_t z_1 + \sigma_t \epsilon)$
6. Loss: $\| v_\theta(z_t, t) - u_t \|^2$

Key: The target $u_t$ is analytically known (no score estimation needed).

Interpolation Schedule

Linear interpolation:

Variance-preserving (VP):

Choice matters: Affects sample quality and training stability.

Key Resources

📚 Essential Reading

Flow Matching for Generative Modeling (arXiv:2510.21890)
https://www.arxiv.org/abs/2510.21890

Comprehensive tutorial on flow matching for diffusion modeling. Explains the connection between diffusion and flows, and why flow matching often leads to simpler training. Start here.

MIT Diffusion Course 2025
https://diffusion.csail.mit.edu/2025/

MIT course covering diffusion models and flow matching with lectures, notes, and code. Excellent for structured learning.

Learning Path

Phase 1: Theory (5 hours)

1. Read flow matching arXiv paper (2510.21890)
2. Work through ODE trajectory derivations
3. Compare to diffusion formulation

Phase 2: Implementation (5 hours)

1. Implement flow matching on 2D toy data (moons, circles)
2. Visualize learned vector fields
3. Experiment with different interpolation schedules
4. Compare Euler vs RK4 sampling

Phase 3: Deep Dive (2 hours)

1. Watch MIT Diffusion Course lectures on flow matching
2. Read original Conditional Flow Matching paper (Lipman et al.)
3. Understand Rectified Flow (next node prerequisite)

Common Pitfalls

❌ Confusing forward/backward: Sampling goes forward in time (0to1), not backward like diffusion.

❌ Wrong target vector: Make sure to use the conditional target (u_t(z_t | z_1)), not the marginal.

❌ Too few sampling steps: Euler method needs ~100 steps. Use adaptive solvers for faster sampling.

❌ Ignoring interpolation schedule: Linear interpolation works but VP schedules often train faster.

Next Steps

• to Rectified Flow / Consistency Toolkit: Modern training and distillation methods
• to Conditioning & Control: Add T2V, I2V, camera control to flow models

Assessment Criteria

✅ You understand this node when you can:

• Explain the ODE trajectory view of generative modeling
• Derive the flow matching loss from scratch
• Implement flow matching on toy datasets
• Articulate the difference from diffusion models
• Choose appropriate ODE solvers for sampling

Overview

Brief introduction to Functional Programming.

Key Concepts

Learning Path

Concept 1

Content here…

Concept 2

Content here…

Implementation

Practical examples and code…

Resources

• Resource 1
• Resource 2

Next Steps

What to learn after completing this node.

Overview

Master the art of iterative online RL training. Understand distribution shift, reward hacking, and the stability knobs needed to keep training on track when both policy and reward model evolve.

Goal: Build robust online RL systems that don’t collapse or hack rewards.

Key Concepts

Online RL Loop

Classic RL: Fixed environment, train policy.

RLHF/Online RL: Environment (reward model) and policy both evolve.

Loop:

1. Collect data with current policy $\pi_\theta$
2. Train reward model $r_\phi$ on new preferences
3. Run PPO to optimize $\pi_\theta$ against $r_\phi$
4. Repeat

Challenge: How to keep this stable when both $\pi$ and $r$ are moving targets?

Distribution Shift

Problem: Reward model trained on data from $\pi_{\text{old}}$, but we’re using it to evaluate $\pi_{\text{new}}$.

Consequence: Reward model becomes miscalibrated on new policy outputs.

Symptoms:

• Reward scores inflate without quality improvement
• Policy generates adversarial examples that fool RM
• Human raters disagree with RM scores

Solutions:

1. KL regularization: Keep policy close to reference $\pi_{\text{ref}}$
2. Iterative RM updates: Collect new preferences on $\pi_{\text{new}}$ outputs
3. Ensemble RMs: Use multiple RMs to detect disagreement

Reward Hacking

Definition: Policy learns to exploit flaws in the reward model without improving actual quality.

Examples:

• Generating verbose but low-content responses (RM prefers length)
• Using rare tokens RM hasn’t seen (RM assigns high uncertainty - high reward)
• Repeating patterns RM can’t detect (e.g., subtle repetition)

Detection:

1. Human eval diverges from RM: RM says great, humans say bad
2. RM ensemble disagrees: RMs give wildly different scores
3. KL explodes: Policy strays far from reference

Mitigation:

1. KL penalty: $\text{reward} = r_\phi(\text{output}) - \beta \cdot \text{KL}(\pi || \pi_{\text{ref}})$
2. RM ensemble: Only trust reward if all RMs agree
3. Human-in-the-loop: Regular human audits
4. Adversarial RM training: Collect hacked examples, retrain RM

KL Regularization

Objective:

Intuition: Optimize reward but don’t stray too far from reference policy.

Tuning $\beta$:

• Too small - reward hacking
• Too large - policy doesn’t improve
• Typical range: 0.01ΓÇô0.1 (tune empirically)

Adaptive KL: Use KL budget that adjusts based on training progress.

Reward Model Updates

When to update RM:

• Γ£à After every N iterations (e.g., every 5 PPO rounds)
• Γ£à When KL divergence crosses threshold (policy drifted too far)
• Γ£à When human eval shows RM miscalibration

How to update RM:

1. Collect new preference data from current policy outputs
2. Finetune RM (don’t reset!) on mixed dataset (old + new)
3. Validate RM on held-out preferences
4. If RM quality degrades, rollback or use ensemble

Risk: RM overfitting to current policy - amplifies hacking.

Solution: Keep diverse preference dataset spanning many policy checkpoints.

Stability Knobs

Learning Rate Schedules

• Start with high LR for fast initial progress
• Decay LR as policy stabilizes
• Use warmup for RM updates

KL Budget Management

• Start with low KL budget (conservative)
• Gradually increase as RM gets more data
• Monitor KL per-iteration, not just cumulative

RM Freezing Periods

• Freeze RM for N iterations to let policy catch up
• Update RM only when policy plateaus

Checkpointing & Rollback

• Save policy checkpoints every iteration
• If reward hacking detected, rollback to last good checkpoint

LLM Learning Dynamics

SFT - PPO - DPO progression:

1. SFT (Supervised Fine-Tuning): Learn from demonstrations
   • Fast initial progress
   • Upper bounded by demo quality
2. PPO (Online RL): Optimize reward model
   • Can surpass demos
   • Requires careful tuning
3. DPO (Direct Preference Optimization): Offline preference learning
   • Simpler, no reward model
   • Less prone to hacking

Key insight: Different algorithms shine at different stages of training.

Key Resources

📚 Essential Papers

Understanding LLM Learning Dynamics (arXiv:2407.10490)
https://arxiv.org/abs/2407.10490

Deep analysis of SFT, PPO, and DPO learning dynamics for LLMs. Essential for understanding when each algorithm works best and how they interact. Must-read for RLHF practitioners.

The RLHF Book
https://rlhfbook.com/

Chapters on online training, distribution shift, and reward hacking with practical advice.

Learning Path

Phase 1: Understand the Dynamics (6 hours)

1. Read arXiv:2407.10490 (LLM learning dynamics)
2. Study RLHF Book chapters on online loops
3. Understand failure modes (reward hacking, distribution shift)

Phase 2: Implement (10 hours)

1. Build online RL loop: PPO + iterative RM updates
2. Implement KL regularization with tunable (\beta)
3. Add monitoring: KL divergence, RM calibration, human eval
4. Simulate reward hacking and test mitigation strategies

Phase 3: Stability Engineering (4 hours)

1. Experiment with different RM update frequencies
2. Tune KL budget over training
3. Implement RM ensemble for hacking detection
4. Build rollback system for catastrophic failures

Common Pitfalls

❌ Never updating RM: Policy will eventually hack a fixed RM.

❌ Updating RM too often: RM overfits to current policy.

❌ No KL regularization: Guaranteed reward hacking.

❌ Ignoring human eval: RM scores become meaningless without ground truth.

Γ¥î No rollback plan: When training goes off the rails, you’re stuck.

Next Steps

• • Search / Test-Time Compute: Use RMs at inference time for best-of-N sampling
• • RL for Structured Outputs: Apply online RL to constrained generation (layouts, CDFs)

Assessment Criteria

✅ You understand this node when you can:

• Implement a full online RL loop with iterative RM updates
• Detect and mitigate reward hacking
• Tune KL regularization based on training dynamics
• Monitor distribution shift and RM calibration
• Explain SFT - PPO - DPO progression
• Build robust checkpointing and rollback systems

Overview

Build the foundational understanding of policy gradient methods from first principles. Derive the policy gradient theorem, implement REINFORCE, and understand the actor-critic paradigm.

Goal: Deeply understand why we can optimize policies directly by maximizing expected reward.

Key Concepts

The Policy Gradient Theorem

Core idea: Instead of learning Q-values (value-based RL), directly parameterize and optimize the policy $\pi_\theta(a|s)$.

Policy Gradient Theorem:

Where:

• $\theta$: Policy parameters
• $\tau$: Trajectory (sequence of states, actions)
• $R(\tau)$: Return (total reward)

Intuition: Increase probability of actions that led to high reward.

REINFORCE Algorithm

The simplest policy gradient algorithm:

1. Collect trajectories by running policy (\pi_\theta)
2. Compute returns (R(\tau)) for each trajectory
3. Update policy: (\theta \leftarrow \theta + \alpha \nabla\theta \log \pi\theta \cdot R(\tau))

Problem: High variance! Single trajectory return is noisy.

Solution: Baselines and advantage estimation.

Variance Reduction: Baselines

Key insight: Subtract a baseline from returns without biasing the gradient.

Common baseline: Value function $V(s_t)$ leads to actor-critic.

Actor-Critic Architecture

Two networks:

• Actor: Policy $\pi_\theta(a|s)$ (what to do)
• Critic: Value function $V_\phi(s)$ (how good is this state)

Advantage: $A(s, a) = Q(s, a) - V(s)$

Update rule:

Benefit: Lower variance than pure REINFORCE, faster learning.

Generalized Advantage Estimation (GAE)

Problem: Bias-variance trade-off in advantage estimation.

GAE solution: Exponentially weighted average of n-step advantages.

Where $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ (TD error).

Tuning $\lambda$:

• $\lambda = 0$: Low variance, high bias (TD learning)
• $\lambda = 1$: High variance, low bias (Monte Carlo)
• $\lambda = 0.95$: Good default

Key Resources

📚 Essential Reading

REINFORCE Algorithm Tutorial (Substack)
https://substack.com/inbox/post/170790602

Clear walkthrough of the REINFORCE algorithm with derivations.

The RLHF Book
https://rlhfbook.com/

Comprehensive resource for RL in the context of LLM post-training. Covers policy gradients, PPO, and RLHF applications. Essential for applied RL.

📖 Books

Foundations of Deep Reinforcement Learning by Graesser & Keng

Practical, code-first approach to deep RL. Includes PyTorch implementations of REINFORCE, actor-critic, and PPO.

Learning Path

Phase 1: Theory (4 hours)

1. Derive policy gradient theorem from scratch
2. Read REINFORCE tutorial
3. Work through actor-critic derivation

Phase 2: Implementation (4 hours)

1. Implement REINFORCE on CartPole
2. Add baseline (value function)
3. Implement actor-critic
4. Compare variance: REINFORCE vs actor-critic

Phase 3: Deep Dive (2 hours)

1. Read RLHF Book chapter on policy gradients
2. Implement GAE
3. Understand connection to PPO (next node)

Common Pitfalls

Γ¥î Forgetting the log: It’s $\nabla \log \pi$, not $\nabla \pi$.

❌ Biased baselines: Only state-dependent baselines are unbiased.

❌ Ignoring variance: High variance = slow/unstable learning. Always use baselines.

❌ Wrong advantage signs: Positive advantage - increase action probability.

Next Steps

• • PPO from Scratch: The de facto standard policy gradient algorithm
• • Preference Data & Eval Design: How to get reward signals from human preferences

Assessment Criteria

✅ You understand this node when you can:

• Derive the policy gradient theorem
• Implement REINFORCE from scratch
• Explain why baselines reduce variance without bias
• Code up an actor-critic agent
• Implement GAE and tune (\lambda)

Overview

Implement Proximal Policy Optimization (PPO) from scratch and build strong evaluation discipline. PPO is the workhorse of modern RLHF and the foundation for understanding post-training.

Goal: Build a production-grade PPO implementation you can trust and extend.

Key Concepts

Why PPO?

Problem with vanilla policy gradients: Large policy updates can catastrophically degrade performance.

PPO solution: Constrain policy updates to a “trust region” using a clipped objective.

The PPO Objective

Clipped surrogate objective:

Where:

• $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$: Probability ratio (importance sampling)
• $\hat{A}_t$: Advantage estimate (from GAE)
• $\epsilon$: Clip range (typically 0.2)

Intuition: If ratio greater than 1+╬╡ or less than 1-╬╡, clip it. Don’t let policy change too much.

Implementation Checklist

Core Components

✅ Policy network ($\pi_\theta$): Output action probabilities
✅ Value network ($V_\phi$): Estimate state values
✅ Advantage computation (GAE with $\lambda = 0.95$)
✅ Probability ratio: $r_t = \pi_{\text{new}} / \pi_{\text{old}}$
✅ Clipped objective: $\min(r \cdot A, \text{clip}(r, 1-\epsilon, 1+\epsilon) \cdot A)$
✅ Value loss: MSE between predicted and actual returns
✅ Entropy bonus: Encourage exploration

Training Loop

1. Collect rollouts using current policy
2. Compute advantages using GAE
3. Multiple epochs of minibatch updates (e.g., 4 epochs)
4. Update policy with clipped objective
5. Update value function with MSE loss
6. Log metrics: KL divergence, clip fraction, explained variance

KL Penalty vs Clipping

Two ways to constrain policy updates:

1. Clipping (PPO-Clip): Hard constraint via clipping
2. KL Penalty (PPO-KL): Soft constraint via penalty term

In practice: PPO-Clip is simpler and works just as well. Use that.

Hyperparameter Tuning

Critical hyperparameters:

• clip_eps (╬╡): 0.1ΓÇô0.3 (default 0.2)
• GAE lambda (╬╗): 0.9ΓÇô0.99 (default 0.95)
• Learning rate: 3e-4 (tune with lr schedule)
• Minibatch size: 64ΓÇô256
• Number of epochs: 3ΓÇô10 (watch for overfitting)
• Entropy coefficient: 0.01 (decay over training)

Tuning strategy: Start with defaults, watch KL divergence and clip fraction.

Evaluation Discipline

Metrics to Track

✅ Episode return (mean, std, min, max)
✅ KL divergence (should be small, less than 0.05)
✅ Clip fraction (what % of updates were clipped)
✅ Explained variance (how well V predicts returns)
✅ Entropy (should decay slowly)
✅ Policy loss, value loss

Debugging Checklist

❌ KL divergence exploding? Reduce learning rate or clip_eps
❌ Clip fraction = 1? Policy changing too fast, reduce LR
❌ Explained variance negative? Value function broken, check value loss
❌ Entropy going to 0 too fast? Increase entropy coefficient

GRPO: Group Relative Policy Optimization

Recent variant: Instead of comparing to a value function baseline, use group-wise relative rewards.

Idea: Normalize rewards within each batch/group before computing advantages.

Benefit: Simpler (no value function), more stable for some tasks.

Key Resources

📚 Essential Reading

PPO and GRPO Comparison (Yugeten)
https://yugeten.github.io/posts/2025/01/ppogrpo/

Deep dive into PPO implementation details and comparison with GRPO. Must-read for implementation.

The RLHF Book
https://rlhfbook.com/

Chapters on PPO for LLM post-training with code examples.

📖 Books

Foundations of Deep Reinforcement Learning by Graesser & Keng

Chapter on PPO with PyTorch implementation.

Learning Path

Phase 1: Understand the Theory (4 hours)

1. Review importance sampling & probability ratios
2. Derive clipped objective from first principles
3. Read Yugeten PPO/GRPO post

Phase 2: Implement (8 hours)

1. Implement PPO on CartPole or MuJoCo
2. Track all key metrics (KL, clip fraction, explained variance)
3. Debug until you get smooth learning curves
4. Compare to baseline implementation (e.g., Stable-Baselines3)

Phase 3: Deep Dive (3 hours)

1. Implement GRPO variant
2. Compare PPO vs GRPO on same task
3. Read RLHF Book chapters on LLM post-training with PPO

Common Pitfalls

❌ Not normalizing advantages: Always normalize per-batch.

❌ Too many epochs: Overfitting leads to high KL divergence.

❌ Forgetting old policy probabilities: Store log probs from rollouts!

❌ Wrong advantage signs: Double-check your advantage computation.

Γ¥î Ignoring clip fraction: If it’s 0 or 1, something’s wrong.

Next Steps

• DPO-Family Competence: Modern alternatives to PPO (offline RL)
• Online Loops & Stability: Iterative PPO training with reward model updates

Assessment Criteria

✅ You understand this node when you can:

• Implement PPO from scratch with all bells and whistles
• Explain why clipping constrains policy updates
• Tune hyperparameters based on logged metrics
• Debug training instabilities (KL explosion, clip fraction issues)
• Compare PPO to GRPO and articulate trade-offs

Overview

Your living playbook for ultra-scale distributed training. Distill practical heuristics for 5D parallelism, overlap strategies, gradient bucketing, and the ZeRO + Tensor Parallelism interplay.

This is the “scale bible” ΓÇö a constantly updated reference as you encounter real bottlenecks.

Key Concepts

The 5D Parallelism Stack

Modern large-scale training combines multiple parallelism strategies:

1. Data Parallelism (DP): Replicate model across workers
2. Tensor Parallelism (TP): Shard individual layers across GPUs
3. Pipeline Parallelism (PP): Split model layers across stages
4. Sequence Parallelism (SP): Shard sequence dimension for long-context models
5. Context Parallelism (CP): Ring-style attention for ultra-long sequences

The art: Knowing which dimensions to parallelize for your model + hardware.

Overlap Strategies

Compute-Communication Overlap

Key insight: Don’t wait for communication to finish before computing.

• Gradient bucketing: AllReduce gradients in buckets as they’re computed
• Pipeline bubbles: Fill bubbles with microbatches
• Prefetching: Load next batch while current batch is computing

ZeRO + Tensor Parallelism Interplay

Naive approach: Apply ZeRO-3 + TP independently leads to excessive communication

Better approach:

• Use TP within nodes (high bandwidth)
• Use ZeRO across nodes (lower bandwidth)
• Hybrid sharding: HYBRID_SHARD in PyTorch FSDP

Gradient Bucketing

Problem: Waiting for all gradients before AllReduce wastes time.

Solution: Bucket gradients and start AllReduce as soon as each bucket is ready.

Tuning: Bucket size trades off:

• Small buckets: more overlap, more kernel launches
• Large buckets: less overlap, fewer kernel launches

Heuristic: 25MB buckets is a good default for most models.

Memory Budgeting

For a transformer with N parameters:

• Parameters: N
• Gradients: N
• Optimizer states (AdamW): 2N (momentum + variance)
• Activations: Depends on batch size, sequence length, hidden size

ZeRO-3 savings: (N + N + 2N) / num_gpus = 4N / num_gpus

Activation checkpointing: Trade compute for memory (recompute activations in backward pass)

Communication-Compute Ratio

Good scaling: Communication time much less than Compute time

Rule of thumb: Aim for greater than 10:1 compute-to-communication ratio.

How to achieve:

• Increase batch size (more compute per communication)
• Use faster interconnect (NVLink better than PCIe, InfiniBand better than Ethernet)
• Reduce communication frequency (gradient accumulation)

Key Resources

📚 Essential Playbooks

Nanotron Ultra-Scale Playbook
https://huggingface.co/spaces/nanotron/ultrascale-playbook

The definitive reference for ultra-scale training heuristics. Covers 5D parallelism interplay, overlap strategies, and practical recipes for scaling to thousands of GPUs. This is your bible.

Smol Training Playbook (complementary)
https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook

Learning Path

Phase 1: Understand the Dimensions (10 hours)

1. Read Nanotron Ultra-Scale Playbook cover-to-cover
2. Map each parallelism type to a concrete use case
3. Sketch communication patterns for hybrid TP+ZeRO

Phase 2: Hands-On Profiling (12 hours)

1. Profile a 7B model with different parallelism configs
2. Measure communication-compute ratio
3. Experiment with gradient bucketing sizes
4. Compare memory usage: ZeRO-1 vs ZeRO-2 vs ZeRO-3

Phase 3: Build Your Heuristics Library (8 hours)

1. Document your own scaling rules for your hardware
2. Create a decision tree: “Given model size X and N GPUs, use…”
3. Benchmark and record actual throughput numbers

Practical Heuristics (Living Document)

Model Size Decision Tree

Under 1B params:

• Pure DP (simplest, fastest)
• ZeRO-1 if memory tight

1B - 13B params:

• DP + ZeRO-2
• Consider TP=2 within nodes if very memory constrained

13B - 70B params:

• DP + ZeRO-3 or FSDP
• TP=2 or TP=4 within nodes
• PP=2 if model still doesn’t fit

Over 70B params:

• Full 5D parallelism
• TP=4 or TP=8 within nodes
• PP=4+ across nodes
• ZeRO-3 or hybrid sharding
• Consider SP/CP for long-context variants

Overlap Checklist

✅ Gradient bucketing enabled (25MB buckets)
✅ Data prefetching in dataloader
✅ Pipeline parallelism with microbatches (8-16 microbatches)
✅ Activation checkpointing for memory-bound models
✅ Mixed precision (bf16) to reduce communication volume

Common Pitfalls

❌ Over-parallelizing: More parallelism ≠ faster. Communication overhead can dominate.

Γ¥î Ignoring hardware topology: Don’t use TP across nodes (slow interconnect).

❌ Skipping profiling: Measure before optimizing. Your intuition will be wrong.

Γ¥î Cargo-culting configs: What works for LLaMA 70B won’t work for a video DiT.

Next Steps

Next Steps:

• Long-Context Parallelism: Deep dive into SP/CP for 1M+ token contexts
• Inference Serving: Apply parallelism thinking to latency-constrained serving

Maintenance

This is a living document. As you encounter new bottlenecks or discover better heuristics, update this page with:

• Empirical throughput numbers
• Hardware-specific tuning
• Model-specific quirks (e.g., DiT vs Transformer parallelism)

Astrophysics Libraries

I’ve been looking at some astrophysics libraries. For instance, I notice that radiative transfer line simulation grids are stored as uncompressed files, which means they take up a significant amount of space, rather than being compressed and being loadable from one file. This is true of saikumarmk/blueshifts, a thesis repository for a paper that explores the changing spectral properties for a disc wind model. Additionally, SKIRTOR, a large collection of emission models of the dusty torus of an AGN has a large uncompressed collection, which I compressed at saikumarmk/skirtor, along with a recreation of the Streamlit application that lets you explore what different AGN look like under different parameters. In particular, the focus is on modelling the behaviour of the dusty torus in obscuring other spectral features.

Kapteyn, a line fitting library for astronomy still uses an older version of numpy<2, which makes it quite inconvenient for virtual environments. A quick glance reveals that the primary issues are due to changes in the numpy C API, and can simply be renamed, which my fork does - saikumarmk/kapteyn_fork.

Pokemon Red Elo

Mostly refactors with code here. Though I would like to eventually try doing the same for Pokemon Crystal, which has a layered trainer AI system and it would make sense for parity.

The Handbook Scraper

The scraper is still in an interesting state where the data is now there, but because the handbook formatting for some courses is strange, you can’t really use it for something such as course planning. Nonetheless, I suspect for courses, a pass with a model like Gemini will get you proper data.

A Transit Map of Melbourne

My old project, MiniMelbourne finally got a face lift and a deploy to transit.saikumarmk.com. Check it! The PTV GTFS-R API also has bus, tram, and VLine API in addition to train data, so the map looks a fair bit more complete. It’s heavily inspired by minitokyo3d, which is a 3D map of Tokyo. You can find the source code here saikumarmk/mini-melbourne-3d.

This Site

In case you haven’t noticed, a complete touch up has been done on the landing page for the site, along with a new portfolio view. It should also be a little less glitchy, and look cooler overall.

Team changes at Canva

I’ve gone from Photo to Video, and our goal is to build a kick-ass video editor. The general theme we’re going for is using AI to assist with the creation of videos, but ultimately the user still controls what they want to put into a video.

So, I decided to revisit my Pokemon Red Elo estimation work, and decided to write this article up as a test for some of my new rendering capabilities. This article will primarily go over deriving Elo.

Deriving Elo

Elo ratings are typically used in tournaments between players/teams, such as chess. It’s well known thatif a player with a lower Elo defeats a player with a higher Elo, the increase/decrease for the player is larger thansay if they were comparable in ‘skill’. The Elo rating is a measure of that skill. Let’s derive it. Suppose we have a ratings vector $\theta = (R_1, R_2, ..., R_n)$, then our goal is to learn this vector.

$P_1$ and $P_2$ battle with ratings $R_1$ and $R_2$. Then we define the Elo update for $P_1$ and $P_2$ as:

$R_1' := R_1 + \alpha (\text{Outcome}(P_1,P_2)- \mathbb{P}(P_1\succ P_2)))$

$R_2' := R_2 + \alpha (\text{Outcome}(P_2,P_1)- \mathbb{P}(P_2\succ P_1)))$

• Here $A \succ B$ means A dominates (beats) B
• Where $\text{Outcome}(P_1,P_2)=1$ if $P_1$, and $0$ if $P_2$ wins.
• $\alpha$ is the scale factor, or how impactful a loss is.

Note if it’s a tie, then it’s considered both a loss and win for simplicity. If $P_1$ was an underdog, say Youngster Joey, taking on E4 Agatha, then the probability of him beating her would be muchlower, and thus his Elo would go up quite a bit. On the other hand, it would be expected of Agatha to beat Joey, so the Elo increment is miniscule in comparison.

But what is $\mathbb{P}(P_1\succ P_2)$? We want the odds of $P_1$ beating $P_2$ to be a function of $R_1$ and $R_2$, that is there is some $f$ such that:

And that $\mathbb{P}(P_1\succ P_2) + \mathbb{P}(P_2 \succ P_1) = 1$. We can disregard ties for the moment since they’re not a substantial issue here.

Because the probabilities add to $1$, we have $O_{P_1, P_2} + 1 = \frac{1}{\mathbb{P}(P_2 \succ P_1)}$ yielding:

Now, what function reasonably satisfies this? The simplest answer is an exponential of the form $f:(R) = \exp(k R)$, which when substituted yields:

The choice of base and $k$ determines how much a win means at a certain odds ratio, and aren’t really too important. This is essentially a sigmoid $\sigma$ as well. Now, let’s say we have our data from the tournament, which could look something like the following:

win, trainer_1, trainer_2
1, AGATHA, YOUNGSTER
0, GREEN1, LANCE
...

Here, trainer_1 is the defender and trainer_2 is the contender, meaning that the win is whether the defender won or not. But this data sounds like it would come from something that’s like a Bernoulli distribution! It so happensthat this formulation is a GLM with the Logit link, which is what we commonly call the Logistic Regression. So, our data $\mathcal{D} = \{(x^{(i)}, y^{(i)})\}$ consists of:

• $y^{(i)} \in \{0,1\}$
• $x^{(i)}_{k,j} \in \mathbb{R}^{391}$, where the the vector is $0$ except for $1$ in the kth entry and $-1$ in the jth entry

Well, we know from equation $(1)$, what a good way of modelling the probability is. Via the PDF for the Bernoulli distribution, the probability of battle $x^{(i)}$ resulting in outcome $y_i$ is:

Recalling that $\theta^T x^{(i)} = R_k - R_j$. How do we determine $\theta$? This is a classic Logistic problem, with $\mathcal{L}(\mathcal{D},\theta) = \prod_{i=1}^N \mathbb{P}(y=y^{(i)} | x^{(i)}; \theta)$, which we want to maximise, or alternatively minimise $\mathcal{J}(\theta) = - \log \mathcal{L}(\mathcal{D},\theta)$.

To compute $\nabla \mathcal{J}(\theta)$, we note that $\partial_{\theta} (\sigma(\theta^T x)) = \sigma(x)(1-\sigma(x)) x^T$ via the chain rule and find that:

There’s no closed form solution for this, due to $y_i$, so we have to rely on Gradient Descent and company to approximate our rankings. The most basic update that you’d know is that $\theta_{i+1} = \theta_{i} - \eta \nabla \mathcal{J}(\theta_i)$, where $\eta$ is the learning rate. Since our problem is reasonably small, we don’t really need to worry about providing a mini-batch of $k$ terms as our gradient. Though I will add that you can do an ‘online’ update where new matches provide gradient updates, and slowly update our rankings if needed, which is a cool connection. We could also contextualise momentum (e.g Adam) as smoothing our gradient from previous gradients as well.

Let’s look through the annotated code now!

In case it’s not already clear to the reader, I love Pokemon. Part of my obsession with Pokemon included simulating the trainer AI from the older games. pimanrules has a series of videos which simulate all $\sim 391^2$ possible trainer AI battles in Pokemon Red, equalising all trainer levels to level 50, and running the same tournament for Pokemon Crystal. The simulation is done directly via a Gameboy emulator, with the memory addresses swapped to make battle decisions. That means the hours required to simulate all the battles come to around 192 hours. My goal was to see if I could achieve this speed-up. Battling is the heart of Pokemon, and being able to analyse the trials set before the player is a very interesting way to to look at the way the games were design for players. In the end, I cut down the simulation time for all $391^2$ battles in Pokemon Red from 192 hours (computing hours) to two and a half minutes. The GitHub project can be found at saikumarmk/pokered-trainer-tournament, and the necessary bindings are available at saikumarmk/PyKMN.

My journey began years ago, with the pokered dissassembly project. The first idea that came to mind would be simulating the core battle system by implementing a rudimentary ASM interpreter, however, I realised that the complexity of the project would shoot up because I’d need to go from ASM to some higher level language.

This idea stagnated till I had a rewatch of the video, and learned that the Pokemon Showdown battle logic was available online. I immediately revisited the disassembly, since I knew if I could scrape the trainer and Pokemon data, all I’d have to do is recreate the trainer AI and then run all the battles.

Pokemon Trainer AI tidbits

The generation one trainer AI is relatively straightforward, and can be found here. However, some trainers have custom AI modifiers, like the gym leaders and the elite four. More information can be found on this Gamefaqs guide. In short, the three primary modifications for Pokemon RBY are:

Modifier 1 - Don’t use status moves

If the enemy Pokemon has a status condition, discourage using a non-attacking status move. There’s a list of moves that fall under this.

Modifier 2 - Buff on turn one two

If there’s a move that buffs your Pokemon, on the second turn, prioritise those moves. Because of an off-by-one error, this modifier really should be applied on the first turn.

Modifier 3 - Use supereffective moves

If there’s a move that’s considered ‘super effective’ against the first type of enemy Pokemon, prioritise that. Additionally, check to see if the Pokemon can use any ‘alternative moves’ (moves with a special damage calculation ¹). If these moves are useable, then penalise using moves that are not very effective/ineffective. Note that there’s no damage check here, so you can get into situations such as Lorelei’s Dewgong spamming rest (which is patched in Yellow).

In generation one, trainer Pokemon movesets are determined by the last four moves they learned from their level ².

The pkmn/engine

While exploring the different implementations of the Pokemon Showdown modules, I found a project called pkmn/engine. The project is a minimal Pokemon battle simulation engine built in Zig, a modern successor to C and is touted as a minimal battle engine. Being compiled in Zig means that the simulation is 1000x faster than the Showdown battle simulation, which is desirable for us. It also intends to be as close to both the original game and Pokemon showdown. This was genuinely amazing when I found it, as it meant that I could use the Python bindings and simulate all the battles with relative ease.

PyKMN

PyKMN is the project that exposes the engine to Python. Unfortunately, the task was never going to be as straightforward as just compiling the bindings. My fork of PyKMN is available here and you can use it to compile the necessary wheels to install PyKMN and use it with the pokered-trainer-tournament project.

Off the bat, the C headers didn’t compile because there was a static assertion that required removing. I wanted the bindings to be compatible with the latest version of the engine, so I made it target the master branch of pkmn/engine. I also needed to change -Dtrace to -Dlog, though I picked up on this relatively quickly as it complained that the argument was invalid.

After that, I had to make some modifications to pykmn/engine/gen1.py. For the time being, I’ve set the engine to always enable tracing as it previously wasn’t picking up on the HAS_TRACE flag. The final modifications involved renaming some variables such as disabled_duration to disable_duration or PKMN_OPTIONS_SIZE to PKMN_CHOICES_SIZE which was accomplished by debugging a battle and inspecting the present variables. That made nearly all the tests pass, with the exception of a few oddities with the move disable, which causes a test case to fail as it wears off earlier than it should have. That was good enough for me anyway. This was a mildly irritating experience as I had to check the headers each time the FFI interface encountered an error, then potentially modify the headers and regenerate the wheel.

Getting relevant trainer data

The relevant asm files from pokered include the:

• base_stats folder which contains their level one learnset
• dex.asm which contains the Pokedex numbers of each Pokemon
• evos_moves.asm contains the way each Pokemon evolves and its learnset
• move_choices.asm contains the AI modifiers each trainer class has
• moves.asm contains each move, any effects, its power, the type, accuracy and powerpoints
• parties.asm contains the trainer Pokemon data for each trainer. Note that trainers with custom moves (Gym + E4) are in a separate file

There are a couple of Pokemon names such as Mr Mime, and Nidoran M/F that are also spelt inconsistently, so we added some extra processing for them. We parse these and then store them in a pickled format.

The Battle Engine

At a high level, the battle engine is straightforward - You specify two teams and then update the state of the battle by selecting a valid choice. These valid choices consist of swapping Pokemon, choosing a move, or passing (when you cause an enemy Pokemon to faint).

py
def run_battle(trainer1: Trainer, trainer2: Trainer, log=True) -> ResultType:
    team1 = trainer1.pokemon
    team2 = trainer2.pokemon
    battle = Battle(
        p1_team=team1,
        p2_team=team2,
    )
    slots: Slots = Slots(([p.species for p in team1], [p.species for p in team2]))
    # Turn 0
    (result, trace) = battle.update(Choice.PASS(), Choice.PASS())
    choice = 1
    while result.type() == ResultType.NONE:
        choice += 1
        (result, trace) = advance_battle(battle, result, trainer1, trainer2)
        if choice > 1000:  # any stalling = tie
            return ResultType.TIE
    return result.type(), choice

The Pokemon data class allows you to specify the species, the moveset, and optionally, the level of the Pokemon.

Putting it all together

We then pit each trainer against every other trainer. Originally, I had it so that each trainer would fight each other twice (one from the POV of P1 and one from the POV of P2) but left this out to see how close it could get to results from pimanrules’ video.

Overall, the ELO ranking ³ looked something like this:

Trainer: Green1 - Green1-C, LR Elo: 216.54730776458132
Trainer: Green1 - Green1-B, LR Elo: 224.5904410299538
Trainer: Green1 - Green1-A, LR Elo: 263.5018156696094
Trainer: BugCatcher - Route 3-C, LR Elo: 484.2755546418754
Trainer: BugCatcher - Viridian Forest-C, LR Elo: 484.68162792183057
Trainer: BugCatcher - Viridian Forest-A, LR Elo: 485.53990646640864
Trainer: SuperNerd - Mt. Moon 1F-A, LR Elo: 486.99372378593375
Trainer: Green1 - Route 22-C, LR Elo: 487.50635955968653
Trainer: BugCatcher - Mt. Moon 1F-B, LR Elo: 527.9433109209984
Trainer: BugCatcher - Viridian Forest-B, LR Elo: 542.0513046912079
Trainer: BugCatcher - Route 3-B, LR Elo: 553.9063718053528
Trainer: Green1 - Route 22-A, LR Elo: 557.79961894872
Trainer: Lass - Route 3-C, LR Elo: 559.1822159583655
Trainer: BugCatcher - Mt. Moon 1F-A, LR Elo: 565.9331459744931
Trainer: Hiker - Rock Tunnel B1F-B, LR Elo: 595.3713903703574
...
...
Trainer: Green2 - Route 22-A, LR Elo: 2473.3390468334183
Trainer: Green3 - Green3-A, LR Elo: 2497.382132748259
Trainer: Green3 - Green3-C, LR Elo: 2527.5603417405864
Trainer: Juggler - Victory Road 2F-A, LR Elo: 2528.905720644544
Trainer: Green3 - Green3-B, LR Elo: 2533.134741771907
Trainer: ProfOak - Unused-C, LR Elo: 2675.2609434721703
Trainer: ProfOak - Unused-A, LR Elo: 2704.059987738582
Trainer: ProfOak - Unused-B, LR Elo: 2713.6055194321307

So while not exactly the same in terms of results (due to some minor implementation details), we get pretty close.

Conclusion - Features to implement

I have yet to implement trainer AI features such as using potions, switching out Pokemon, or Gym Leader AI which may bias the results. I would also like to make nice visualisations for this project, though it’s not nearly a top priority as pimanrules’ video does that already. It would be very cool to see how pkmn/engine develops, and whether we can recreate the same setup for Pokemon Crystal.

A couple of weeks ago, I wanted to test out Claude Sonnet’s capabilities for generating web applications, as my attention had been brought to bolt.new, a prompt system that allows you to prompt full-stack web applications. Word on the street, according to people I work with was that Claude’s Sonnet was very powerful for generating entire applications, and I had yet to trial Sonnet on my personal account. I tested Sonnet’s capabilities by asking the model to replicate the heat map of SETool, a now-defunct ¹ piece of software that deploys a Dash application to Heroku for visualising unit outcomes.

Introducing the Unit Scores Dashboard for anyone at Monash who’s ever tried to make sense of the SETU outcomes for units, and used SETool in the past. I provided Claude an image of the heat-map and the following prompt below:

I once created a webapp in plotly+dash that visualised unit scores (13 items, from 1 to 5) in a heat-map like grid by mapping from red to green. The code is in python and relies on the dash table, so I want you to recreate it in javascript (you can use libraries such as d3/vega/whatever you think is the best for the task). It had the following features:
    Filters (semester, level, choosing the mean/median for overall score) toggling columns - each item in the database (I will supply a sample item)
    Search filters in each column, for numerical items you can do comparison filters and string columns you can search matches
    checkboxes next to each entry so you can add it to a comparison table that's similar to the first table.
This is a sample entry:
{'Responses': 30, 'Invited': 79, 'Season': '2019_S2', 'Response Rate': 37.9746835443038, 'unit_name': ' Accounting information systems and financial modelling ', 'code': ' ACB2851_PENINSULA_ON-CAMPUS_ON_S2-01 ', 'unit_code': 'ACB2851', 'Level': 2, 'I1': [4.23, 4.35], 'I2': [4.37, 4.62], 'I3': [4.27, 4.5], 'I4': [4.23, 4.56], 'I5': [4.17, 4.39], 'I6': [4.23, 4.41], 'I7': [4.23, 4.5], 'I8': [4.27, 4.56], 'I9': [4.13, 4.27], 'I10': [4.1, 4.33], 'I11': [4.03, 4.19], 'I12': [4.3, 4.5], 'I13': [4.2, 4.5], 'agg_score': [4.212307692307693, 4.436923076923076]}
Assume this is loaded from some JSON/serialised format. now then...
I'd... i'd like you to unleash the ultimate implementation of this code: DONT HOLD BACK, CLAUDE-KUN! TAKE IT TO THE LOGICAL ENDGAME!

With the crazy prompt adapted from this Twitter post. To which, it generated a React component for me, which I promptly threw into basic next app. And it worked! Sort of…

After the initial prompt, I asked it to refine the web application and allow users to filter columns by specific properties. I managed to get it to parity with the original SETool within a few prompts and a few hours of hacking. It is a vast upgrade over how long it took me to code up SETool. Additionally, since it’s in React, I can host the web page on GH-pages by exporting the web app to a static format ² which makes deployment much easier (usually).

Anyway, if you’re a Monash student, go check it out and let me know what you think! As of now, the comparison table is a bit bugged, but otherwise it’s pretty functional and looks good! I will try to update the dashboard here and there, but I also wouldn’t mind any prospective students taking it off my hands (and you get a free project on your resume).

I’ve previously written on Circles, an amazing, student-built course planner maintained by DevSoc, a society that used to be a branch of CSESoc. Monash has its course planner, named MonPlan which was formerly student-led by people such as Eric Jiang. Unlike UNSW, however, the project came into the hands of Monash Esolutions, which meant that development would then be relegated to Monash. While MonPlan is a great base for developing a Circles clone, it has not seen any development in a while, primarily due to internal politics.

During my time at Monash, I took several swings at looking for a consistent method of retrieving unit requisites, with my most successful attempt with a microservice that MonPlan uses giving rise to the Monash Graph available on my site. I also learned about how Circles was developed from some DevSoc directors who also interned at Canva with me, along with some other online friends. Since I’ve developed a reliable way to scrape and format the requisites for a unit reliably, I believe that the Monash student community has the potential to develop resources that are similar to Circles. This way, the resources are more relevant to students as it’s developed by students, and can be passed down.

I also think Monash is in a slightly better position than other universities because we have an exposed microservice that gives (mostly) correct requisites.

An Upgraded Handbook

I rewrote my handbook scraper in Go, which retrieves the units, courses, and areas of study (aos) from the Monash handbook. I’ve only made use of the units to draw the unit graph on my site, but we can also format the courses and areas of study. The immediate implication is that we can use this for course auto-planning, along with verifying the completion of a course map.

An immediate enhancement I can see is a better handbook, baked into the course planner like Circles does here. This could factor in your selected course and aos and allow for smart selection. Some easy wins:

• Developing an enhanced handbook experience
• Tells you what units you unlock by undertaking a unit (an indirect recommendation system)
• An explicit recommendation system (using the former, and your other units)
• The explicit cost of units, pulled from the SCA band and an up-to-date cost calculator
• Gives you proper, strictly formatted requisites for a unit that are easier to interpret
• Think, of a DAG representing the units required to unlock a unit

These features fall out from the requisite data we already possess.

The Monash Unit Graph

The Monash Unit Graph on my site is a fun experience but doesn’t provide much use to students who want to interpret the data. I wanted to replicate the original Circles with their click-to-complete graph view, though I ended up adding a build mode that visualises units as a DAG.

Some things I think can be done:

• Using a DAG as a base to create more detailed progression diagrams that can be used in guides
• A click-to-add view which stores the selection data from the other modes for visualisation purposes

Other features

Some more features off the top of my head:

• Auto-planning with operations research (basically 2SAT the problem) similar to how Circles does it
• This requires representing the requisites as an AST, which is how they unlock new units when you add units
• The semester planner; I don’t think this is a big priority since MonPlan does a good job at this
• An API for querying; this is just a natural consequence of hosting stuff on a site and having an API from which to request resources.