Five cool things you can ask Google’s improved Gemini models to do from your Android phone

Google I/O 2024 kicked off with a significant update for Android users. Google has introduced the new Gemini AI models into the Android system, enhancing the intelligence of mobile devices. These models are built to understand what you need by analysing the context, making it easier to get help with everyday tasks.

Now, your Android phone can help you plan trips, find information, or reply to messages without the need to jump between apps. For students, the standout feature is the homework help added to Circle to Search, which can quickly solve word problems.

This article will highlight some handy ways to use the updated Gemini models on your Android phone.

1. Solve complex problems with Circle To Search

Circle To Search now offers a helping hand with math and physics problems. If you’re stuck, circle the problem and let Gemini’s AI take over with step-by-step guidance. It’s available on Android devices that support this feature, allowing you to highlight math and physics word problems onscreen for immediate assistance. 

This capability is powered by the latest LearnLM AI models, specialized for educational purposes.

While yesterday’s update helps with text-based problems, solutions for questions involving diagrams, graphs, or symbolic formulas are expected later in 2024. 

It’s uncertain whether this will coincide with the release of Android 15 or if it’s pending further AI advancements. But later this year, Circle To Search aims to address even more complex challenges, including those with symbolic formulas, diagrams, and graphs. 

Availability: Already aiding over 100 million devices, this feature will soon be upgraded to tackle even more complex academic challenges.

2. Seamless Gemini Integration across apps

Gemini’s AI is now more ingrained in Android phones, streamlining your digital experiences. With the new Gemini overlay, you can effortlessly drag and drop images into emails or messages from any app. The “Ask this video” feature lets you pinpoint details in YouTube videos. Google’s also enhancing the overlay for easier use, like pulling images into apps or, with Gemini Advanced, digging into PDF content. For example, Gemini can clarify the two-bounce rule from a pickleball tutorial video.

Availability: The new overlay will make additional dynamic suggestions too, but all of these improvements are only rolling out "over the next few months," according to Google.

3. Multimodal understanding with Gemini Nano

Gemini Nano AI model will introduce a new dimension to Pixel phones, processing more than just text. It now understands visuals, sounds, and spoken words, providing a richer context for every interaction. That means the AI will be able to interpret sights and sounds, including spoken language, much like OpenAI's new GPT-4o model. 

A simple toggle lets you switch between text, images, and audio. 

Google hasn’t disclosed if this feature will require the upcoming Pixel 9 or how it connects to Project Astra, an effort to deliver AI results based on real-time video.

Availability: Pixel users will be the first to experience Gemini Nano later this year.

4. Enhanced accessibility with TalkBack

TalkBack is getting a significant upgrade with Gemini Nano’s multimodal capabilities. Visually impaired users will receive more detailed information about what’s on their display, making the digital world more accessible. 

The multimodal support in TalkBack will help better describe images with or without a network connection.

Availability: Significant updates to TalkBack is expected to roll out later this year. 

5. Real-Time Scam Identification

Reflecting on a study by the Global Anti-Scam Alliance, Google highlights the staggering $1 trillion lost to fraud in just one year. To combat this, they’re introducing a feature to identify scams as they happen. During a call, if Gemini Nano spots patterns typical of scams, it’ll alert you right away. This means if someone posing as a bank official asks for a quick money transfer or your card details, you’ll know something’s off.

Your conversations stay private, processed directly on your device, much like how current Live Captioning works. Google assures us that opting in is your choice, addressing concerns about AI listening in on calls.

Availability: Scam detection is in the testing stage, and we’ll get the full scoop by the end of 2024. Likely starting with Pixel phones running Gemini Nano.

Google’s set to share more about Android 15 this Wednesday. We’re expecting talks on new features, including satellite messaging, and a launch this fall with the new Pixel lineup. 

The Gemini models are ready to make Android phones more helpful and intuitive. So, when it’s time, go ahead and just ask Google— to experience your phone in a smarter way.

Pricing

While many of Google’s AI features are available for free, some advanced capabilities will come with a price tag:

Gemini Advanced: Gemini Advanced is available at $19.99 per month, complete with a 2-month free trial.

Stay tuned for more updates on the release dates and availability of these exciting new features.