I'm not OP, but George Hotz said in his lex friedman podcast a while back that i... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

sputknick on April 18, 2024 | parent | context | favorite | on: Meta Llama 3

I'm not OP, but George Hotz said in his lex friedman podcast a while back that it was an MoE of 8 250B. subtract out duplication of attention nodes, and you get something right around 1.8T

qeternity on April 18, 2024 [–]

I'm pretty sure he suggested it was a 16 way 110 MoE

brandall10 on April 19, 2024 | [–]

The exact quote: "Sam Altman won’t tell you that GPT 4 has 220 billion parameters and is a 16 way mixture model with eight sets of weights."

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact