(I am having a terrible time finding the fly file size; I can get connection + neuron counts, but not an estimate of how much data each neuron simulation takes. Things like hyper-parameters, different neurons acting differently. So I’ll punt on that question and hope someone more competent will educate me. The numerics look very consistent with ~6gb to me, but I could easily be wrong in either direction.) The phrase “LLM algorithm” is ambiguous, there are two that are relevant: 1) the algorithm that the LLM uses to take a context and produce the next word. 2) the algorithm that was used to create an LLM who is good at task (1). It is true that the ML algorithm in (2) was created by a human, including all its parameters and data. It is not true that any human intentionally picked the algorithm (1); that was discovered by a complicated stochastic process. Let me give a concrete example of what I mean by “nobody understands the algorithm in (1)”. Take adding two (let’s say 2 digit) numbers; this is something that claude has been able to do for awhile. There are many ways to add two numbers; ranging from: bitwise add+carry we use in our chips memorizing a 100*100 sized table of the answers encoding the numbers as vectors, adding those, and then decoding the vector Which does claude use? Nobody knew until quite awhile after the first ML algorithm had the capability. The answer is weird, the model computes this by memorizing the results for the ones digit, approximating the overall magnitude of the solution, and massaging these things together to get the answer. Nobody programmed this addition algorithm; it was generated by (2). This is our understanding of addition of 2 digit numbers; imagine our miserable understanding on how, say, claude decides what genre a role playing discussion is in. My point at the start of the thread is also very much in line with the sweat shop analogy. I do not believe anyone currently knows how to look inside the factory right now; no journalist, manager, or ceo has the keys. This goes for both humans and the LLMs, though I’m more confident about our ignorance in LLMs.