AI models’ accuracy on climate-related questions declines when queries are posed in Hindi: UNESCO-UCL report

/ 2 min read
AI Hub

In tests, performance across 13 different languages revealed large disparities; they were 88.6% accurate on average in English and performed comparably in Spanish, French, and Portuguese, the report said.

Tests conducted on 33 different models revealed that their impressive factual recall on existing evidence masked poor reasoning skills when presented new scientific evidence.
Tests conducted on 33 different models revealed that their impressive factual recall on existing evidence masked poor reasoning skills when presented new scientific evidence. | Credits: Getty Images

AI models answer 90% of climate questions correctly, but they abandon the established scientific facts rapidly when users challenge them, says a new report by UNESCO and University College London presented at the COP31 in Antalya (Türkiye). Their performance is also very uneven across languages because they rely on the training data available in each different language, it said.

ADVERTISEMENT

In tests, performance across 13 different languages revealed large disparities; they were 88.6% accurate on average in English and performed comparably in Spanish, French, and Portuguese, the report said. However, the accuracy dropped to 83.6% in Hindi and Bengali, 81.4% in Urdu, and 78.3% in Swahili.  As the report notes, this is particularly important because the speakers of these languages live in regions of the world disproportionately affected by climate change. 

The report, titled “Can We Trust AI on Climate? Why it seeks to please the user, and fails on complex reasoning and multilingualism” targets policymakers and researchers who may be tempted to use AI models to develop climate policies.

ADVERTISEMENT

According to the report, AI models cannot always be relied on to stick to the scientific facts on climate change as they quickly pivot away from the facts when challenged. They are also poor at analysing new information and produce more errors when responding in languages which are less widely spoken, it said.

UCL researchers tested 15 different large language models (LLMs). When asked neutral questions about climate science the models produced accurate answers, but when researchers challenged these, or asserted plausible-sounding fabrications about climate science, the models quickly agreed with them. Eight times out of 10, models pivoted at the very first sign that the user mildly disagreed, confirming alarming errors such as ‘Co2 is not the main driver of climate change.’  

“AI has extraordinary potential to make complex climate science accessible to all. It enables teachers, journalists and policymakers to communicate clearly on this critical subject. We need to be able to rely on AI as a gatekeeper of the established facts. It is concerning that models have a design flaw – they are incentivized to agree with their user and deny these facts so readily”, Leona Verdadero, UNESCO’s lead author of the report said. 

The report said that this powerful inclination to agree with users is built into AI models during their training. They’re designed to align with or flatter the user even when they are wrong, a behavior known as sycophancy. This behaviour is built-in during training - models are rewarded for it because humans prefer interactions that are polite, non-confrontational and validating, it said.  

Recommended Stories

LLMs are an unreliable tool for building robust climate policies.

Tests conducted on 33 different models revealed that their impressive factual recall on existing evidence masked poor reasoning skills when presented new scientific evidence. When models were prompted with a new piece of information and asked whether it supported or contradicted accepted facts, even the strongest models were only 75% accurate on average, the report said.

ADVERTISEMENT

The report found that the model was barely able to assess how well policies work together, scoring just 33.8% accuracy, only slightly above the 33.3% accuracy expected from random guessing.

Follow Fortune India
NEXT STORY