In 2026 running AI workloads at scale is extremely power inefficient.
Let’s see what could be done to improve this.
Flash instead of RAM
Today models are stored in dynamic RAM chips that require uninterrupted power to maintain charges on capacitors.
An interesting solution is to store model weights into stacked Flash:
https://spectrum.ieee.org/high-bandwidth-flash
Imagine a system that:
- Can store weights for very large models, let’s say hundreds of billion parameters
- Can operate in deep power saving modes when not in use
- Goes from standby state to operational very quickly and efficiently (no need to reload weights in RAM)
Memory compression techniques
Large matrices like in the case of attention blocks in Transformer-class models can already be represented as sparse matrices.
What about the precision of values?
Right now all values in a matrix are represented with the same type (FP16, FP8, NVFP4, …) and models are typically quantized to fit memory constraints
But it seems like some entries in a matrix would be affected more negatively by quantization than others, depending on the value.
What if a variable length representation was used at rest, in Flash?
Too inefficient to “decompress” at runtime?
The information on depth of each value could be used also in the multiplication block, to achieve better efficiency?