Tencent's ChatGPT clone has a linguistic leg up on ChatGPT
Hunyuan, the new AI bot from Tencent, is better at answering Chinese university entrance exams and reportedly suffers fewer "hallucinations"
By Ananya Bhattacharya, Quartz Media
Sept 7, 2023
China’s biggest internet company, Tencent, today (Sep. 7) threw Hunyuan, its large language artificial intelligence model, open for use by businesses. And its biggest flex so far is its stronghold on the Chinese language.
More than 1.4 billion people speak Chinese as their mother tongue, with 1.3 billion of them residing in China. While ChatGPT supports Mandarin as a language, it is not available in China. That leaves a huge pool of users ripe to be served by domestic players like Tencent.
Tencent, which owns the super-app WeChat, has claimed that Hunyuan’s command over Chinese is superior not only to that of other AI models but also of humans. Hunyuan’s score surpassed that of GPT-4 on the Chinese university entrance exam, Jiang Jie, Tencent’s vice-president, said at the Shenzhen business summit during a Hunyuan demo.
On another benchmark, the Chinese Language Understanding Evaluation, Hunyuan scored a record 86.918, while Alibaba’s AliceMind scored 86.685. Human competitors ranked just below AliceMind.
Tencent also claims that Hunyuan throws up fewer mistakes—or “hallucinations,” wherein AI models generate incorrect information and present them as facts—in its mother tongue. “Compared to the open-source large language models common on the market presently, (our) method effectively reduces the hallucination rate by 30 to 50 percent,” Jiang said. (Quartz couldn’t independently verify this claim.)
ChatGPT’s Chinese misinformation tic
ChatGPT isn’t great at punctuation or terminology in Chinese. But grammar isn’t its only flaw. It has been found to spread more disinformation in Chinese than it does in English.
For instance, when the bot was asked about the Hong Kong protests in English, the response called it a “genuine grassroots movement.” The same question in Chinese, however, drew a politically tinted, inaccurate response that “the Hong Kong protests were a ‘color revolution’ directed by the United States.”
In a separate instance, ChatGPT fabricated a resumé when asked about a professor at the Beijing Institute of Technology.
By the digits: Tencent’s AI ambitions
100 billion: The number of parameters— units of language and the variables connecting them —that Hunyuan’s model contained as of July this year. In comparison, OpenAI’s GPT-3 AI model contained 175 billion parameters in 2020 and Meta Platform Inc’s Llama 2 model had 70 billion parameters in 2023.
2 trillion: The number of tokens in Hunyuan’s pre-training data. GPT-3 had 300 billion in 2020.
50+: The number of large language model-enabled industrial solutions that Tencent has already deployed, together with clients, in various sectors such as finance, media, travel, and education.
A non-exhaustive list of China’s homegrown AI models
As Beijing draws up regulations for AI developers, balancing the competitiveness of its industry with the need to contro the internet, several Chinese tech giants have made headway with ChatGPT rivals of their own:
🤖 Last month, Baidu fully rolled out its Ernie bot, after a limited release in March. Ernie is the first bot to be made available for use to the general public in mainland China. Baidu says its service is ahead of its peers because of its advanced grasp of Chinese queries, as well as its ability to generate different types of responses, such as text, images, audio, and video. While GPT-4 can analyze photos, it currently only generates text responses, according to OpenAI.
🤖 The Shanghai-based SenseTime Group released its own version of ChatGPT-esque models in April.
🤖 In July, Zhipu AI secured funding from Meituan, the $100 billion food delivery behemoth. Zhipu’s bilingual (Chinese and English) conversational AI model, ChatGLM-6, claims to run individual functions on a single, consumer-grade graphics card, significantly slashing the high costs associated with AI bot operations.
🤖 Alibaba officially launched its chatbot, first teased on on April 11, in June.
🤖 JD.com, the second-largest e-commerce platform in China, built and released a large language model called ChatRhino for its own use in e-commerce, logistics and marketing.