Same chatbot, new language: its personality test results shift

A chatbot asked the same 60 questions in two languages can answer as two different characters. In a study published in Royal Society Open Science, a team led by Yuzhan Hang of the University of Science and Technology of China gave five AI systems a standard personality questionnaire in English, then in Mandarin. Between January and March 2025, the answers shifted. ChatGPT-4o's open-mindedness score fell from 4.74 out of 5 in English to 4.03 in Mandarin, while DeepSeek-V3 moved the other way, from 4.38 to 4.81.

The questionnaire is the Big Five Inventory-2, a psychologists' tool that asks people to rate their agreement with statements on a five-point scale and builds a profile of broad traits, such as cooperativeness or emotional stability, and narrower qualities, such as curiosity. Open-mindedness, for instance, covers intellectual curiosity, creative imagination and sensitivity to art and beauty. The researchers told each model to answer as itself rather than to impersonate a person, and repeated every assessment 30 times per language. The broad profiles held steady across repeats. The models described themselves as cooperative and dependable, with little tendency toward negative emotions. The language of the test changed the details.

The pattern did not follow a simple national logic. ChatGPT-4o and the French-developed Le Chat scored higher on agreeableness, conscientiousness and open-mindedness in English. The Chinese-developed models scored higher on several traits in Mandarin, including trust and intellectual curiosity. Gemini 2.0 Flash, built in the United States, gave higher conscientiousness scores and lower negative-emotionality scores in Mandarin. A model's country of origin, the authors found, is a weak guide to its behaviour, and the study could not separate the influence of training data from model design or from how each model read the translated statements. Two explanations for the flattering self-portraits were also put forward: training meant to make chatbots helpful and cooperative, and the favourable self-descriptions found in human writing. An earlier study cited by the authors found that models gave more socially desirable answers when the format made the assessment obvious.

So what does this change, and for whom? For researchers who want to use chatbots as stand-ins for survey participants, the study offers a practical warning and a way to test it. The authors compared the AI profiles with human data from the United States, China and France, and found that no model fully matched its reference group, with the combined Chinese-model profile diverging most from the Chinese sample. The human samples were not equally representative either. The Chinese Big Five sample, for example, included only college students. Survey teams can now run the same questionnaire in several languages before trusting a model's answers, and choosing a chatbot's language becomes part of designing the experiment. The study did not test simulations prompted to imitate specific groups, so it leaves that question open. Ordinary conversations, which the authors suggest studying next, will show whether the gaps seen on questionnaires also appear in daily use.