LLMs performed best on questions related to legal systems and social complexity, but they struggled significantly with topics such as discrimination and social mobility.
“The main takeaway from this study is that LLMs, while impressive, still lack the depth of understanding required for advanced history,” said del Rio-Chanona. “They’re great for basic facts, but when it comes to more nuanced, PhD-level historical inquiry, they’re not yet up to the task.”
Among the tested models, GPT-4 Turbo ranked highest with 46% accuracy, while Llama-3.1-8B scored the lowest at 33.6%.
I was trying to solve a betweenle a couple weeks ago, had it down to the words immediately before and after the word-for-the-day, and couldn’t think of it. I went to three different AI engines and asked them what word was between those two, alphabetically. All three engines repeatedly gave me “answers” that did not occur between the two words. Like, I’d ask "what 5-letter English words are alphabetically between amber and amble, and they’d suggest aisle or armor. None of them understood ‘alphabetically’.
deleted by creator
I just tried chatgpt with all of these failure points–a 5-letter word that fits alphabetically between amber and amble (ambit), a sentence that ends in the letter R (it ended with “door”), and a poem that rhymes (aabbccbb). These things appear to be getting ironed out quite quickly. I don’t think it’s much longer before we’ll have to make some serious concessions.