It could be that emerging Non-Volatile Memory (NVM) MRAM, like the VG-SOT-MRAM from European research center IMEC could help increase the cache density (replacing part or all the SRAM cache) AND also improve power efficiency as it is Non-Volatile (load the parameters in the cache once, and they stay there without needing energy to retain the information: it is a bit like bi-stable E-ink).

I think this goes back to the 85 TB/s figure in the “unfortunately using …” paragraph of this TNP article, which refers back to Figure 4 in the Chiplet paper. For HBM4 at 10 TB/s, the V100 would still need to perform 10-20 flops-of-processing/byte-of-accessed-mem for fullish utilization of the GPU’s oomph — if I correctly mis-exfoliate … (100 flop/byte at 0.9 TB/s on the fig.)

I fully expect that to happen at some point, but heaven only knows when. This Dot Cog Boom will probably not take as long as the Dot Com one did. We are a lot faster at the hype cycle these days, eh?

Well said. Thanks.

Sram scaling is dead after 3nm so not sure that the idea that the way forward is to expand SRAM to fit these huge models inside is the right one. This is where Graphcore started; either they were early, or wrong. A related point on sram scaling is that the compute becomes ever cheaper and thus ever more irrelevant in $ and mm2 versus the memory. So why prematurely optimise to an extremely fixed function compute solution, surely better to have something more general purpose (but not GPU sized)? The SRAM can always be stacked vertically to make room for that.

The parameter count is still vast versus the size of the training set. The algorithms for training and the model structure and assumptions themselves need to be fixed to deal with this fundamental flaw before genAI for all becomes economic; a very fixed function ASIC-write-large doesn’t alter the situation. I expect more innovations to follow before long that should improve efficiencies from the top down. Attention is seemingly, *not* all that is needed.

Dojo is using pretty advanced packaging. Similar in concept, for sure, with precise interconnects.

ASICs are way less costly, way faster and much cheaper in quantity. You pay in turn around time and debugging effort. I’ve worked at places where the original product went out in FPGA so they could do updates as needed. Once it was stable, they did a cost reduction ASIC spin to save money. Didn’t lower the price of the hardware… but kept more money.

I would assume in this case going straight to ASIC is the way to go, especially since they’re logically fairly simple designs, they just want close memory, connectivity, and a bunch of matrix engines. Once they debug it in FPGAs… they cost reduce like crazy since the payout is super quick. It’s just three to six months from sending the design files to TSMC before you get hardware back…. that’s the only reason they’d do this on FPGAs first.

If I am multiplying it right, 175B GPT-3 minimum latency calls for 414GB of SRAM. That doesn’t sound like ~4 bit quantization to me.