Apple’s iPhone has not been seen as an AI-infused smartphone, but new research from Cornell University working with Apple suggests that this may be about to change. The research paper titled “Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs” introduces a multimodal large language model that can understand what is displayed on a screen, specifically the elements in a mobile user interface like an iPhone display. This upgrade to the existing Ferret model could potentially enhance the capabilities of Siri and open up new possibilities for accessibility apps.
Ferret-UI, released in October 2023, is designed to parse photos and images to identify what is being shown. The ability to describe the screen, regardless of the app being used, could help users perform complex tasks or find hidden options on their iPhone by simply asking Siri for assistance. This advancement in AI technology could also be beneficial for developers, who could utilize Ferret-UI as a testing tool to simulate various user interactions and scenarios.
Recent discoveries by Nicolás Álvarez point to new server-side tools for individual iPhones, including features such as “Safari Browsing Assistant” and “Encrypted Visual Search.” These features align with the AI capabilities described in the Ferret-UI research, although it is important to note that these findings are based on server-side code and may not necessarily indicate the final product features. Apple’s approach to AI technology is slowly emerging, with hints of future advancements in their products.
Google was the first to publicly embrace AI-first smartphones, setting the stage for other tech giants like Apple to follow suit. While Apple did not make any significant announcements regarding AI improvements in their devices at the time, they are now gradually revealing their plans for incorporating AI into their products. The upcoming Worldwide Developer Conference in June will likely shed more light on Apple’s AI strategy as they prepare to launch the next generation of iPhones later in the year.
As Apple continues to innovate in the field of AI technology, users can expect enhanced features and capabilities in future iPhone models. The integration of advanced AI models like Ferret-UI could revolutionize how users interact with their devices, offering a more intuitive and personalized experience. With ongoing research and development in AI, Apple is poised to make significant strides in this area, potentially reshaping the landscape of smartphone technology.
