How to use your smartphone to identify objects
You don’t need us to tell you how smart the smartphone has become: from recognizing our voices to plotting complex routes in seconds, this device is a real tricks box.
With the right app, they can also help identify what’s in the world around us, whether it’s the breed of the dog that just befriended you or information. on a landmark you are visiting.
These apps work as visual search tools, using information from your phone’s camera and adding artificial intelligence-based object recognition magic to determine what is currently in sight.
Live visual and text search
If you have an Apple device with the A12 Bionic or a newer chip inside (iPhones launched in 2018 or later, or iPads launched in 2019 or later) and it is running the latest software, you can enjoy of two features called Visual Lookup and Live Text.
Visual Lookup works on photos you’ve already taken and stored on your phone or tablet, and can identify things like artwork, monuments, flowers, books, and pet breeds. Open the image in your Photos app and you’ll see a special glitter effect around the info button (an “i” in a circle) at the bottom if your phone can identify something in the image.
[Related: Organize your catastrophic photo library]
Press the button, then the icon in the center of the image on the next screen. This icon will indicate what has been identified (it will be a locator pin for a landmark, for example, or a petal for a flower). More information will appear on the screen, although the content will vary depending on what’s in the image. You may be shown details of the breed of dog you are looking at, or the book cover that is in the photo, to name just a few potential results.
Live Text works in the Photos app and the Camera app. If you’re using the latter, just point the camera at text on a page or panel without taking a photo. If your phone recognizes text, a Live Text icon (three horizontal lines in a frame) will appear in the lower right corner, and you can tap it to start highlighting the text in the image.
Long press the text to select it, then drag the blue handles to expand the selection. Once you’ve selected a few, a pop-up list of options will appear, including the ability to search the web for the selected text (To look for) or translate it from a foreign language (Translate).
Google Lens is available as a standalone version Android app, and as part of the Google app on ios and iPadOS. Some of its skills include the ability to recognize plants, animals, and landmarks, and it works much like an image search on Google’s main website.
On Android, open the Google Lens app and you can browse the images already on your phone or tap Search with your camera. Choose the latter option and point your phone’s camera at anything to see if Google Lens can identify it. We’ve had success with dog breeds, book covers, famous buildings, and American presidents.
The search function works similarly on photos already stored on your phone. Open one of them in Google Lens, and after a few moments, the bottom of your screen will display all the results that Google Lens can find. By tapping on any result you will get more information on the web.
In the Google app for iOS or iPadOS, tap the camera icon in the search box to launch Google Lens, which works just like on Android. You’ll also see a Google Lens button (a small camera icon) in Google Photos to Android and ios when you open an individual image, use it to search using a photo you’ve already taken.
Google Lens can also copy and translate text, shop for items online, and even solve math equations. or whatever is on the photo you selected).
Various other apps are able to recognize objects that you point your phone’s camera at or that have already been photographed. Snapchat for Android and ios can do this, for example, through a feature called Scan: open the camera screen, then tap and hold whatever you’re looking at on the screen (on iOS, you’ll also need to tap To analyse).
If Snapchat Scan can find a match with what you’re watching, you’ll see it appear at the bottom of the screen, along with any relevant Snapchat content (such as themed lenses and backgrounds). Snapchat says the feature can identify cars, dog breeds, plants, and wine bottles, and also solve math problems. Expect more features and capabilities to be added over time.
Depending on what you are looking at, more specific apps may be available. The Merlin Bird ID app for Android and ios works through your phone’s microphone rather than its camera, but it can help you identify birds from their songs with impressive accuracy. If you prefer to take a photo, try Picture Bird for Android and ios.
[Related: How to start birding in any US city]
Avid gardeners will want to take a look at PictureThis for Android and ios. Open the app, point it at a plant and the program will instantly tell you what type of plant you are looking at. The app is even smart enough to diagnose plant diseases by leaf shape and color, and give you tips on restoring diseased plants.
If you are going outdoors, you can use your smartphone’s camera to identify mountain peaks you encounter. Install PeakVisor for Android Where ios, point your phone at a mountain range, and the app will examine your location and the shape of the peaks to tell you what they are called and how high they are.