> Did they fix the underlying issues or does it just have more data?
IIRC they do have slightly more data, but that's not the primary cause of improvement, the key factor is simply more parameters and more training. No significant actions have been taken "fix the underlying issues" - you should assume that any major differences between GPT-2 (which is horrible in comparison to GPT-3) and GPT-4 are emergent behavior from the model having more horsepower.
IIRC they do have slightly more data, but that's not the primary cause of improvement, the key factor is simply more parameters and more training. No significant actions have been taken "fix the underlying issues" - you should assume that any major differences between GPT-2 (which is horrible in comparison to GPT-3) and GPT-4 are emergent behavior from the model having more horsepower.