@jonny@neuromatch.social As long as inference is powered by GPU's then transformer models are going to be O(N²) for model size N. Even without a GPU, that exponent won't drop to less than ~1.3. We see this in the power forecasts from the 2024 DOE report (Figure 5.1, 5.2), sure looks like geometric growth to me. It's rare to see that noted explicitly. I don't trust post 2024 figures and other sources naively draw a straight line. It's the same pattern of trying to throw ∞ resources at a problem that we saw in the 80's with CRAY and simulations. (Then some smart people figured out a change of algorithm and solved the same simulations on a workstation. Bye bye, CRAY.) One big difference, I suppose, with LLM AI is that they have no idea of what problem they are trying to solve, no way of knowing if they get there, and no proof they ever will. At least with Perceptrons in the late 50's everyone knew why they gave up. This time some idiots have bet a significant chunk of global GDP on this useless 🐎 💩. https://escholarship.org/uc/item/32d6m0d1