Online underwear brand Adore Me, owned by Victoria’s Secret, has introduced a new AI feature that allows users to customize lingerie sets with images of swastikas, corporate logos, and copyrighted characters like Disney’s Snow White and Aladdin. This feature also enables users to add phrases like “Heil Hitler” and “White Power” to the underwear’s waistband. Despite attempts to block certain terms, a phrase using the n-word was allowed to proceed, showing some flaws in the system.
Forbes tested the AI tool and was able to create designs featuring living people, including politicians like President Joe Biden and former President Donald Trump, as well as logos of popular sports teams. While the tool does have some limitations, such as blocking terms related to suicide and self-harm, it still allows for the creation of controversial designs. Adore Me’s senior vice president for strategy, Ranjan Roy, stated that the company reviews individual designs for brand safety before a purchase can be made.
Despite concerns about hate speech and inappropriate content being created with the AI tool, Adore Me and its parent company Victoria’s Secret suggest that they are not responsible for what users generate since they are not actually selling the offensive designs. However, Forbes was able to purchase a panty set featuring terms associated with a neo-Nazi hate group, raising questions about the company’s oversight of the platform. The AI technology behind the tool is provided by Leonardo AI, an Australian company that has raised $31 million in funding and has faced criticism for creating nonconsensual pornography.
Leonardo’s terms of use prohibit depictions of real people and content that promotes hatred or discrimination. The company has stated that they are developing more advanced detection mechanisms to prevent hate speech and associated imagery. However, the AI tool was found to be less likely to block terms related to suicide and self-harm in certain languages, highlighting the challenges of content moderation in different linguistic contexts. Additionally, the tool generated a disturbing image in response to pro-anorexia search terms, raising concerns about the potential for harmful content creation.
The introduction of generative AI tools on retail websites is becoming increasingly common, with companies like Puma and Rembrand also incorporating AI features into their platforms. These tools allow for personalized experiences for customers but also raise issues around content moderation and toxicity detection. Analysts warn that AI-powered offerings require careful oversight and monitoring to prevent the dissemination of harmful or inappropriate content. Adore Me’s AI tool is just one example of the challenges faced by companies utilizing AI technology for customization and personalization.
