AfterSessionExercises.txt

(1) In Session 4 slides "Activation Sharding (Tensor or Model Parallelism)", we consider the case where we shard the activations over the columns and the weights over the rows. We find that there is an in-layer overlapped algorithm.

Consider instead the same column sharding for the activations but the weights also being sharded over columns. To compute the output, each chip needs to "all-gather" the activations. Can you find a pipelined order for all-gather the activations that allows us to overlap gathering the input activations with computating the output activations?

(2) Consider Session 4 slides "Batch Sharding with Weight Sharding (FSDP)".

Given B, E, number of chips N, calculate the number of FLOPs per chip per layer for the math and the number of bytes of bandwidth per layer. (Answer at the bottom of the file.)

Call this F (FLOPs) and I (ICI bytes read).

(3) Assuming hardware specific numbers of 275 * 10^12 FLOP/s and 100 GB/s of bandwidth under what conditions can the communications be hidden, in terms of I and F. (Answer at the bottom of the file.)

(4) How does the answer in (3) compare to the arithmetic intensity of the system? (Answer at the bottom.)


















































































































































































































































































































































































(2 answer) --  Each chip needs to do a matmul of (B/N,E) by (E,E) which is 2*B*E^2/N flops

The weights are E,E and, we typically assume 2 bytes / param so we need to gather them as 2*E^2 bytes. (Don't worry if you don't know why this is 2 bytes / param)

(3) We need F/(275 * 10^12) > I/(100 * 10^9). So F/I needs to be greather than (275 * 10^12)/(100 * 10^9) = 2750.

(4) The arithmetic intensity of the system is exactly (275 * 10^12)/(100 * 10^9) = 2750!