Apple iOS 27 Visual Intelligence searches screen content on supported iPhones

Apple has introduced an expanded Visual Intelligence feature in iOS 27 that allows iPhone users to search for objects, ask questions, and perform actions based on content already displayed on their screens. This update extends the capability of Apple Intelligence beyond camera-based identification to include screenshots of videos, messages, and digital documents.

The primary function of this tool is to recognize items within a user's current view. For example, if a user pauses a video to capture a screenshot of a lamp, the system can identify the object and offer options for visual search or similar item recommendations. Apple states that the feature can also process text from digital flyers or invitations sent via Messages, potentially converting that information directly into a Calendar event without manual data entry.

Availability for these specific Visual Intelligence tools is limited to devices supporting Apple Intelligence on iOS 27. Users must have an iPhone 15 Pro, iPhone 15 Pro Max, or any iPhone 16 model or later. While iOS 27 is compatible with older iPhone models, those devices will not support the advanced screen-analysis features if they lack the necessary hardware. Additionally, access to the feature may vary depending on the user's language and region settings.

Users can access these tools by taking a screenshot of the desired content. The interface then presents context-specific buttons, which may include visual search, text summaries, Read Aloud, or direct actions like adding calendar events. The specific options available depend on what the system recognizes in the image. For camera-based usage, users with Camera Control can press and hold the button to initiate a search of physical objects in front of the device. A newer Siri AI mode in the Camera app also allows users to ask questions about the camera frame, though this feature remains in beta.

Samsung Galaxy and Google Pixel owners utilize a similar function called Circle to Search, which has been available on Android devices prior to the iOS 27 release. On Galaxy devices running Android 14 with One UI 6.1 or later, users can activate Circle to Search directly over on-screen content without taking a screenshot. This is enabled through Settings under Display and Navigation bar. The feature also works while the Camera app is open, allowing users to identify physical objects in stores or other environments.

Google Pixel phones, specifically models from Pixel 6 onward, support Circle to Search with comparable capabilities. Users enable the feature by searching for "Circle to Search" in Settings and ensuring Google is selected as the default digital assistant with screen data permissions enabled. Like its counterpart on Galaxy devices, Pixel users can select on-screen elements to search for more information, translate text, or identify landmarks and products.

Both Apple and Google provide privacy controls for these visual search features. Apple states that many Apple Intelligence requests run directly on the device, while more complex requests use Private Cloud Compute, where data is not retained after fulfilling the request. Users can manage Visual Intelligence settings under Settings and Siri. Google notes that Circle to Search activity may be saved with other Google Search activity, subject to standard search history and personalization controls.

Reporters note that visual search tools may not work within applications that block screen capture or analysis for security or copyright reasons. Users are advised to verify results before relying on them, as visual identification can occasionally produce incorrect or similar-but-different matches. These tools are designed to streamline initial searches rather than guaranteeing absolute accuracy in object identification.