The bearable mediocrity of Baidu’s ChatGPT competitor
China Report is MIT Technology Review’s newsletter about technology developments in China. Sign up to receive it in your inbox every Tuesday.
Did you stay up late last week to watch the release of Ernie Bot, the first Chinese rival to ChatGPT? It felt like the most anticipated event in China’s tech world so far this year, but I couldn’t force myself to stay awake till 3 a.m., so I watched a recorded version of the Baidu press conference the following morning.
By that time, the launch had been met with an almost overwhelming wave of disappointment. Chinese publications with testing access ridiculed the chatbot’s performance, social media users mocked it with memes, and Baidu’s stock dropped by 6.4%. (If you missed the news, don’t worry: I wrote a story summarizing the day’s highlights and letdowns.)
But a curious thing has happened since last week’s launch: Ernie Bot’s reputation seems to have bounced back. Baidu’s stock price rebounded by 15.7% on Friday. More Chinese reporters gained access to the chatbot and published more moderate reviews.
“The market rationalized and realized: Even [though] Ernie didn’t wow us, it’s probably good enough for the Chinese market,” says Jennifer Zhu Scott, a Hong Kong–based deep-tech venture capital investor and founder of IN. Capital.
I’ve warmed up to it too. I was pretty disappointed at first, primarily because Baidu only showed pre-recorded demonstrations of the chatbot at the event, which didn’t suggest a lot of confidence in the technology. But the more I’ve read about other testers’ interactions with Ernie Bot, the more it seems like a decent upgrade on Baidu’s previous models and certainly one fair-size step toward ChatGPT.
It actually performs very much the way ChatGPT does: Ernie Bot also likes to talk in a weirdly formal tone and list answers in bullet points and numbered lists. It has a basic command of historical facts, works of literature, and internet trends but sometimes gets the details wrong. When asked questions about harmful information or politically sensitive topics, it awkwardly shies away from giving an answer. But it also has image-making capabilities, unlike ChatGPT. On that score it may not be as sophisticated as Stable Diffusion or Midjourney, but it does seem much better than first-generation models like DALL·E.
Is this a big enough step for the Chinese market?
There’s a popular phrase in China’s science and tech world today: 弯道超车, to overtake another car on a bend. While it’s clear that the US is still the world leader in science and innovation, the Chinese government and companies often hope that with the advantage of a large market, more accessible data, and direct government support, they can quickly bridge the technology gap in a short amount of time and even overtake the US. Artificial intelligence is one area the Chinese tech industry is targeting, and even the US side has become worried about competition.
But Ernie Bot didn’t overtake ChatGPT. It fails in many of the same areas where ChatGPT failed: It too makes up facts and makes errors in grade school math. The same questions can befuddle both bots. (I love this example: When you ask “Can my dad and mom get married?” in Chinese, both bots tell you that they can’t legally marry because they are first-degree relatives.) Ernie performs marginally worse than ChatGPT, making more mistakes and understanding less about complex questions. At the same time, ChatGPT could be getting better quickly: last week Open AI unveiled GPT-4, an even more advanced version of the large language model that can be used to power the chatbot.
Rather than becoming a source of national pride, as many observers had hoped, the release of Ernie Bot confirmed that Chinese companies are still trailing behind their American peers by quite a distance. It’s a sobering reminder that Chinese AI companies and researchers still have a lot to catch up on, even as they become increasingly important in this space.
So what’s next? Many of Baidu’s domestic competitors, like Alibaba and Tencent, have confirmed that they are working on similar products, but there’s no indication that they’re close. “I would not bet on any [other consumer-facing] applications coming out anytime soon,” says Zhu. Enterprise products, on the other hand, may come sooner.
At the end of his presentation on Thursday, Baidu’s CEO, Robin Li, tried to downplay the theme of an AI arms race between the US and China. “Ernie Bot is not a tool for China-US technology confrontation,” he said.
But in the current geopolitical climate, it’s inevitable that people on both sides will continue to use the Ernie Bot vs. GPT comparison as a proxy for the tech gap between China and the US. Baidu just announced that there will be another press conference on Monday, when it’s expected to explain more about how other companies can adapt Ernie Bot for their own businesses. And by then, there’s no doubt people will compare it with how GPT-4 is being used by Microsoft, Duolingo, or Morgan Stanley.
Are you satisfied with Ernie Bot’s performance so far? Tell me your reaction at [email protected].
Catch up with China
1. TikTok has had a tumultuous week. First, it was reported that its parent company ByteDance is considering selling the app if it can’t reach a deal with the US government over national security questions. (Bloomberg $)
- But that plan could also be rejected by the Chinese government, China-based lawyers and investors say. (The Information $)
- It was reported this week that the US Justice Department started investigating ByteDance as early as late 2022 for surveilling American TikTok users, including two journalists. (New York Times $)
- And a group of Silicon Valley executives, including Peter Thiel, are secretly mobilizing in Washington against TikTok. (Wall Street Journal $)
2. Chinese researchers uploaded genetic samples from Wuhan in 2020 that show links between the coronavirus and raccoon dogs, boosting the likelihood that covid had a natural origin. But the data was quickly scraped from the database after international academics reached out to investigate it further. (New York Times $)
3. President Xi Jinping traveled to Russia to meet with Vladimir Putin this week. The economic relationship between the two countries has weakened in recent years. (Wall Street Journal $)
4. A year after the China Eastern Airlines crash that killed 132 people, the Chinese government still doesn’t have a conclusion about what went wrong. (Associated Press)
5. Jiang Yanyong, the Chinese doctor who exposed the cover-up of the SARS outbreak in 2003, died at the age of 91. (NPR)
6. Someone keeps cutting the undersea cables connecting a Taiwanese archipelago to the internet. Taiwanese authorities blame accidental damage from Chinese ships. (Vice)
7. Guo Wengui, a controversial Chinese billionaire with close ties to Steve Bannon, was arrested in New York on Wednesday for a $1 billion fraud scheme. (NBC News)
- “The New Federal State of China,” an entity Guo and Bannon launched in 2020, greatly exaggerated its role in helping to rescue Ukrainian refugees in 2022 and used it for political promotion. (Mother Jones)
Lost in translation
During the first two years of the pandemic, Chinese insurance companies popularized “covid insurance”—people can pay a one-time premium of a few bucks and get thousands of dollars back if they catch covid. But as journalist Yu Meng wrote in the Chinese publication Connecting, it can be extremely hard to get that payout.
Yu bought covid insurance at the beginning of 2022 and tested positive on an at-home antigen test in December, during a national wave of infections after China loosened its pandemic control measures. The insurance company gave her a number to call, but no one answered. Yu reports there are at least 60,000 more people who filed a claim with the same company. Some called dozens of times a day, and some sued the company. Some filed complaints with China’s insurance regulator. But very few people actually got paid in the end.
At one point when she finally managed to reach the company, a customer representative told Yu: “Do you know how many claims we have? You think the people above me haven’t calculated the costs? Of course, they did. It can reach billions and will cause the company to go bankrupt. Do you think the state will allow a state-owned company to go bankrupt? Can you imagine that?” In the end, Yu, who had expected to get 20,000 RMB ($2,900), accepted 5000 RMB. Her parents, who bought the same insurance, gave up on getting any money back.
One more thing
Even kids can’t escape the AI craze now. Recently, the local government in China’s eastern province Zhejiang announced it would incorporate more artificial-intelligence education into the grade school and middle school curricula. How intense the lessons will be is still unclear, but I’m wondering: will we come full circle and see Chinese kids using Ernie Bot to do their homework on Ernie Bot?
