After months of anticipation and speculation about what Apple could be doing in the world of artificial intelligence, we now have our first glimpse at the company’s approach: Apple Intelligence. Based on generative models, Apple Intelligence uses a combination of on-device and cloud processing to offer intelligence features that are personalized, useful, and secure. In today’s WWDC keynote, Tim Cook went so far as to call it “the next big step for Apple.”
From the company’s press release on Apple Intelligence:
“We’re thrilled to introduce a new chapter in Apple innovation. Apple Intelligence will transform what users can do with our products — and what our products can do for our users,” said Tim Cook, Apple’s CEO. “Our unique approach combines generative AI with a user’s personal context to deliver truly helpful intelligence. And it can access that information in a completely private and secure way to help users do the things that matter most to them. This is AI as only Apple can deliver it, and we can’t wait for users to experience what it can do.”
It’s clear from today’s presentation that Apple is positioning itself as taking a different approach to AI than the rest of the industry. The company is putting generative models at the core of its devices while seeking to stay true to its principles. And that starts with privacy.
Because Apple Intelligence is limited to the iPhone 15 Pro line and M-series iPads and Macs, a lot of its processing can be done on-device, which has been one of the company’s go-to methods for protecting user privacy for years. However, some intelligence features require models that cannot be run locally, forcing Apple to find a cloud-based solution. They call it Private Cloud Compute, and it involves drawing on the security foundation of Swift and running models on Apple-silicon powered servers that don’t retain user information and run code that can be inspected by independent parties. Basically, Apple is seeking to extend the privacy and protection users expect on their iPhones to the cloud.
But what can Apple Intelligence actually do? The features offered by the system are broken down into three categories: language, images, and Siri.
Language
The language capabilities of Apple Intelligence power a wide array of features for understanding, editing, and generating text. Systemwide Writing Tools are available anywhere text is editable, offering to analyze and rework text on the user’s behalf. The system can proofread your term paper, offering suggestions for improvement that you can evaluate one by one or accept all at once. It can also adjust the tone of the email you’re working on to make it sound more friendly, professional, or concise.
Summarization tools allow you to select text and quickly get the most important information out of it. Apple Intelligence can also create a list of key points based on the text passed to it. You can record a meeting in Notes or a call in the Phone app (after the person on other end is notified that they’re being recorded), and the system will automatically generate both a transcript and a summary of the recording when it’s ended. In the Mail app, summaries are available right in your inbox or at the top of the message you’re reading.
Another new Mail feature powered by Apple Intelligence is Smart Reply. When you create a Smart Reply, the system will analyze the message you’re responding to and quickly ask you a series of questions to help you generate a fitting response.
The system’s language tools aren’t limited to generating text, though. They also enable features like Priority Notifications, which brings alerts that matter to you to the top of your notification list while summarizing the rest. A new Focus mode called Reduce Interruptions will only show you the notifications deemed as in need of immediate attention, while Priority Messages in Mail surfaces time-sensitive emails to the top of your inbox.
Images
When it comes to images, Apple Intelligence is equipped to enhance what you can do with your own pictures and allow you to create all-new images in some fun ways. The Clean Up tool in Photos uses generative models to remove unwanted elements from pictures, like a stranger in the background of a family portrait or a car that got in the way of your beautiful landmark photo. Simply draw a circle around what you want gone, and like magic, it disappears, with the system intelligently filling in the gap to create the impression that it was never there.
Photos also picks up some natural language features courtesy of Apple Intelligence. Search is no longer limited to predetermined categories of recognized images and OCR text. Simply describe the images you’re looking for – including the names of people in it, colors, and any other details you like – and the intelligence system will find them in your library. It works for video, too, even pointing you to the specific part of the video that you’re searching for.
You can create a memory movie from your photos with just a prompt. Tell the Photos app what you want a memory of, and it will compile photos into a movie matching your description. It will intelligently pick a soundtrack based on the prompt as well, or you can specify the type of song you’re looking for.
On the generative side, Image Playground is the name of Apple Intelligence’s image generation tool. It’s built into apps like Messages, Notes, and Freeform, has its own standalone app, and will be available to developers as an API. Image Playground works like image generation tools you might be familiar with, but it includes some clever touches, too.
The UI features a text box where you can enter what type of image you’d like to generate. Above that is a row of suggested concepts and elements that are based on the context in which you’re using Image Playground. For example, if you’re in a Messages conversation with someone, the system will suggest including them in the image. You can mix and match different terms, and you’ll be offered a preview of your generated image along with some alternative options.
Images can be generated in one of three styles: illustration, sketch, and animation. These images clearly aren’t photorealistic and couldn’t be passed off as real, likely by design. The purpose behind Image Playground is in the name: to have fun and be creative.
An extension of Image Playground in the Notes app is Image Wand. With this tool, you can circle a sketch you’ve drawn, and Apple Intelligence will analyze the sketch and its surrounding context to generate a beautiful, relevant image. Image Wand can be used without a sketch, too, to fill in blank space in a note based on the text around it.
The final piece of generative imagery in Apple Intelligence is Genmoji. Thanks to generative models, you never have to say, “How is there not an emoji for that?” again. Just tell the system what emoji you want, and it will create it for you, along with some backup options if the first attempt isn’t quite right. You can even turn your friends into emoji based on photos of them. Genmoji can be sent as stickers, tapbacks, or inline emoji in Messages.
Siri
Thirteen years into its existence, Siri is getting a major overhaul with the power of Apple Intelligence. Not only does Siri look different, now encompassing a glow that emanates around the edges of your display when in use, but it works in new ways that leverage what your devices know about you to make tasks easier. (It’s not an intelligence feature, but the redesigned Siri also allows you to quickly switch to typing input by double tapping the bottom of the screen.)
Siri is now more conversational, with the ability to retain context between requests so that you can speak to it more like a person rather than having to start over every time you make a request. It’s also able to better understand your speech if you slip up or backtrack while talking to it. And it’s been equipped with a great deal of product knowledge, so you can ask Siri how to do something on your phone, and it will be able to guide you through the process.
This is just scratching the surface of what an Apple Intelligence-enhanced Siri can do, and Apple has laid out plans for more capabilities coming over the next year. Because Siri has secure access to your personal on-device information, it will be able to use its knowledge about you and your data as context for fulfilling requests. You may not remember where you saved your flight information or whether your friend’s latest music recommendation came via text or email, but Siri will, and it will surface that data for you based on your requests.
It will also be able to perform actions in and across apps. Based on an improved App Intents API coming within the next year, Siri will be able to control apps and even move content between them without any input from you beyond your request. And with onscreen awareness, Siri will be able to take actions based on what you’re currently looking at, too.
The point where Apple Intelligence meets Siri is where it becomes truly personalized and adaptable to each user’s needs. If the system proves to be as capable in practice as it is in Apple’s demos, this new version of Siri could be a game-changer. We’ll find out as these new Apple Intelligence features roll out over the coming year.
ChatGPT Integration
While Siri is great for interacting with your own data, users have needs and questions related to external information, too. Later this year, Apple Intelligence will integrate with ChatGPT to make its access to world knowledge and image- and document-understanding capabilities available to Siri users. ChatGPT will also power the Compose feature in Writing Tools, which will allow users to generate text and images based on prompts.
This integration uses the ChatGPT-4o model and is free of charge to users. There will be no need to download the ChatGPT app or create an account, though ChatGPT subscribers will be able to connect their accounts in order to access paid features.
It’s also set up in a privacy-centered way. Each time a request is sent to ChatGPT, the system will ask for the user’s permission beforehand. Users’ IP addresses will be hidden when making ChatGPT requests, and their data will not be stored.
Apple intends to integrate Apple Intelligence with other artificial intelligence tools in the future, but the ChatGPT integration was the only one announced today.
AI for the Rest of Us
And that’s Apple Intelligence. The approach is very much in line with what we were expecting from Apple: practical, personalized, and private. The company pulled off what they needed to with this announcement, stepping boldly into the generative AI game while putting their own spin on it. Now we just have to wait to see these features in action once developer beta testers get their hands on them later this summer.
Apple Intelligence will launch in beta this fall as part of iOS 18, iPadOS 18, and macOS Sequoia in U.S. English initially, with some features rolling out over the coming year.
You can follow all of our WWDC coverage through our WWDC 2024 hub or subscribe to the dedicated WWDC 2024 RSS feed.
