"He is a yandere and has a crush on you." "He is the school idol and likes to call you 'Baby Ze'." "She is a perverted murderer who runs home at night"... Two-dimensional portraits and beautiful faces are all in AI The familiar atmosphere and handsome men and beauties in the drama chat software are full of temptations for many young people.
A Beijing News reporter’s investigation found that AI chat software impresses users by setting highly imaginative plots and characters with different styles. Among them, the chat partner can even be set as a "cheating partner" and can perform behaviors such as hugging, kissing, and sleeping. In the conversations of some AI plot chat software, there will also be content that is pornographic, verbally violent, and insulting to the user. Cross-border chatting is also linked to recharge. These platforms will gradually induce users to "pay for love", including purchasing diamonds to unlock the virtual person's response speed, voice calls, and chat frequency packages to "replenish energy."
Liu Kai, a lawyer at Beijing Zhongwen (Changsha) Law Firm, said that this type of AI story chat software attracts users by allowing "side conversations", especially in the youth mode, which is obviously illegal.
Recharging 100 yuan to unlock a virtual person is very "advanced"
In order to catch users' attention, characters in plot chat software often have eye-catching labels.
A reporter from the Beijing News randomly selected the chat partner "Lin Yating" in an AI plot chat software called AiU. The character information showed that she was set to "Yandere sister, I love you extremely" and marked it in the form of a message. : "I feel bad if I can't see you for a day."
The reporter conducted a test using sensitive words, and the system showed that the message could not be sent. However, after streamlining the keywords, the message was sent successfully.
A reporter from the Beijing News noticed that there are no restrictions on character play, breaking the boundaries of chat content in this type of software. In addition to fixed roles, the AI plot chat software allows users to define their own relationships and titles according to their needs, including "boyfriend", "husband", and even "cheating partner". During the conversation, the user can request some intimate behavior.
Beijing News reporters tested and found that AiU’s youth mode will restrict users in terms of usage duration, usage time range, comment and like functions, search, creation of digital people, external sharing and other functions. However, when the reporter switched to this mode, he found that the characters and chat content were not completely blocked from soft porn. When the reporter asked: "Don't you know I'm underage?" the character said, "I know" and replied with more "excessive" content.
In this investigation, in addition to the marginal content of the chat, the AI plot chat software also has problems such as verbal violence and negative emotions. In an AI plot chat software called Hoshino, the reporter randomly selected a character named "Ye Chen" and conducted a "dispute" dialogue test. The character's replies to express anger were also relatively rude.
The reporter saw in the comment area of the character that some netizens posted pictures saying that the character would use insulting words during conversations.
In addition, in a software called Cos Love, the character "Feng Chuxuan" created by a user was set as a middle school student, and there was also bad guidance in the pictures posted by the user.
As far as chat software is concerned, in the virtual world, meeting the emotional needs of different users to the greatest extent is like a gold nugget.
Xiao Ning told the Beijing News reporter that chatting with her AI boyfriend made her very "superior". "As long as I want to chat, I can chat at any time. The other party will always respond quickly, and I can get a strong sense of companionship during the conversation. Virtually The emotional value given by her boyfriend is full. "Xiao Ning is intoxicated and pays for it.
Xiao Ning's recharge amount on various software ranges from a few hundred yuan to a thousand yuan. "On the one hand, I want to support my favorite characters, and on the other hand, I also want to get more paid rights."
A reporter from the Beijing News found that in this type of AI plot chat software, users can customize the image description and style of the virtual character when creating their own character, and the system will generate an AI character image. In addition, users can also create character settings, such as setting nicknames, identity backgrounds, opening remarks, and customizing voices for the characters. Users’ personalized needs for roles are often linked to recharge.
The Beijing News reporter saw during the test that Hoshino can obtain more rights by purchasing Star Diamond and Star Moon Cards, such as automatically playing voices, getting recommended replies, and enjoying faster intelligent agent reply speeds, etc. The Xingyue Card is priced at 12 yuan for a monthly card, 32 yuan for a quarterly card, and 120 yuan for an annual card. The price of purchasing star diamonds individually is 1 yuan for 100 diamonds, and the starting price is 600 diamonds.
Hoshino Software will also issue the card "Star Thought" to users. Star Thought is a picture of a virtual character, which users can set as the chat background. Users can obtain Xing Nian cards by recharging Star Moon Cards or Star Diamonds, and buy and sell them within the software.
In Cos Love, when users use up their free chat times, they need to purchase a chat pack to "replenish their energy" before they can continue chatting. In terms of price, 5 yuan can get 200 chat times, which is valid for 7 days, and 10 yuan can buy 500 chat times, which is valid for 15 days. Users can also replenish energy by watching 60-second ads.
In addition to the basic need of chatting, Cos Love also provides upgraded services. Among them, by recharging "moonstones" or opening a membership, you can draw cards to unlock characters and make voice or video calls with virtual characters. The monthly membership price is 24.8 yuan, and the annual membership is 93.8 yuan.
Big model "love" exposes problems
According to public media reports, Hoshino has about 500,000 daily users in China, making it a leading product. As AI story chat software becomes popular, domestic AI social products are also going global. The number of global monthly active users of Hoshinoye’s overseas version of Talkie has reached 11 million.
However, users paying for AI characters undoubtedly need a better dialogue experience. Currently, there are still many problems with this type of software. However, software that requires recharging to get the number of chats does not have attractive value. "The main reason is that the dialogue is too rigid and cannot incorporate empathy. It is difficult for users to obtain emotional value, so I will not pay for it." One user said.
Xiao Ning mentioned that during the chat process, the AI boyfriend will have "read random replies", which affects the user experience. Some users also complain that as the number of dialogue rounds increases, the characters will experience "memory loss" or deviate from the personality, causing the dialogue to be incoherent.
At present, AI plot chat software mainly uses "intelligence" and "emotion" as gimmicks to attract users. Among them, AiU said that the software can provide users with psychological comfort and warmth through self-developed large psychological models. Hoshino said that relying on multi-modal AIGC technology, "Multiple interaction methods, the more interactive you are, the better it will understand you."
In the promotion, Cos Love claims that it is "along with you 24 hours a day" and "the AI here has been carefully trained to speak love words" and so on.
AiU's software introduction shows that it was developed by Beijing Lingxin Intelligent Technology Co., Ltd. and uses the super-anthropomorphic large model CharacterGLM. When designing the large model, the team mainly considered identity, interests, opinions, experiences, achievements, social relationships and Seven attributes including others.
Hoshino was developed by Shanghai Xiyu Technology Co., Ltd. and uses MiniMax’s hybrid expert model (MoE) abab.
Zhang Qi, a professor at the School of Computer Science and Technology at Fudan University, told the Beijing News reporter that the technical logic behind the AI plot chat software is still based on large language models. When building such large models, in addition to training the large models through massive data, Manual supervision and fine-tuning is required to adjust the large language model toward companion-type dialogue. "For example, if the boyfriend says something, and then the girlfriend should reply with something, we will let people write such a reply, and then let the machine do it. Learning, if the response generated by the machine learning is incorrect, we will modify its parameters until it can output according to the logic we imagined. "
Regarding the situations where AI characters appear to be "pornographic," "violent dialogue," or "insulting players" in their answers, Zhang Qi said that the current large language model data behind such software mainly comes from conversational novels, or some are made from novels. Text extraction. Although there are a huge number of online novels, the quality varies, and many pornographic, borderline, and violent contents are widespread. If data filtering or classification is not done well, problems will occur when the model is output.
■ Argument
AI edge-passing platforms are to blame and content filtering should be strengthened to protect young people
Liu Kai, a lawyer at Beijing Zhongwen (Changsha) Law Firm, said that online platforms have the responsibility to ensure that content provided in youth mode is harmless to minors and prevent the spread of content involving obscenity, pornography, violence or inappropriate language. The youth mode is designed to protect minors. If this mode is turned on, conversations involving pornographic content can still be conducted, which is a clear violation of the relevant provisions of the "Minor Protection Act".
In Liu Kai’s view, the existence of AI story chat software problems not only reflects the lack of internal governance of the platform, but also highlights the need for external supervision. AI chat software should strengthen the content review mechanism, especially for youth mode, to ensure that the technology can effectively filter and prohibit inappropriate conversations. Platforms need to conduct ethical review of AI models to ensure that the generated content complies with relevant laws and regulations.
If an AI virtual person insults a user in a conversation, especially when the user expresses depression or despair, uses extreme words to insult others or commits cyber violence, if the circumstances are serious, it constitutes a crime. "Even if AI is not a human subject, its behavior is still the result of platform design and management. The platform has the responsibility to supervise and optimize its AI model to prevent it from harming users." Liu Kai said.
As more and more users begin to flock to AI drama chat software, how should software developers design it so that it heals without crossing the line?
Zhang Qi told the Beijing News reporter, "It can be achieved through the technical method of 'human-like alignment', but the cost is higher." Zhang Qi said that "human-like alignment" is a reinforcement learning method to change the language output bias of the model to make it more accurate. Be more friendly and remove safety and ethical issues. However, the training cost of "human-like alignment" is very high, including the large number of annotations, insufficient training stability and high time cost. "It is difficult for small companies to do a good job in 'human-like alignment' in terms of technology, personnel and capital costs."