It looks like Microsoft agrees with Meta on this, as they’ve (84 co-authors; 7 per page) just developed their “Phi-3 Mini” model, with ½ as many parameters as Llama 3 8B, yet similar MMLU performance (perf varies on other tests), and aimed at phones ( https://www.theregister.com/2024/04/23/microsoft_phi_3_mini/ ). It’s great to see this competition towards smaller yet as “accurate” models that can help reduce inference infrastructure costs.

They are adding AI so they can continue to make money in ads and social networking and messaging and trying to figure out how to make money in an AI-infused metaverse that people will live, work, and play in. Not a world I want to participate in. I will take the real one, thank you very much. But I have no doubt that lots of people will spend a lot of time there, just as they do with Facebook and TikTok today.

Meta on the other hand seems to focus more on a LeCun perspective that smaller models may work better: “a trillion words, it would take […] 22,000 years for a human reading eight hours a day to go through […]”. This, though, may require further exploration of model architectures, and here, Llama 3 remains “a relatively standard decoder-only transformer” (“Meta Platforms rolled out Llama 3” link). Hopefully, some more architectural explorations, and/or a mixture of simpler models, or even a combination of logic-based and biomimetic AI, can help get a “GPA of 83.9” with less than 800B parameters (to promote inference efficiency).

Funnily enough, the “Unreasonable Effectiveness of Data” paper states: “we can expect Semantic Web technology to work best where an honest, self-correcting group of cooperative users exists and not as well where competition and deception exist”. ANN-based AI would then come to the rescue of “logic-based” AI in such “human” situations, as in Noam Brown’s (OpenAI) approach to Texas hold’em poker, and Diplomacy (maybe).