This morning at Apple’s second fully-remote WWDC keynote address, Craig Federighi introduced iOS and iPadOS 15. This year’s updates include significant improvements to core first-party apps, new controls for maintaining focus, system-wide text and object recognition in images, and much more.

On the iPad-only side of things, Apple has announced a variety of new multitasking interface elements, feature parity with the iPhone’s Home Screen, quick note capturing available at any time in any app, and an overhauled Swift Playgrounds which supports building and shipping complete SwiftUI apps to the App Store.

As usual, developer betas are available today, with final versions scheduled to ship to all users this fall. Let’s take a look at all the details that Apple has in store for us this year.

FaceTime

Last year the pandemic caused an explosion in video chatting, and it was quickly made clear that FaceTime did not support the features that users needed. Apple must have recognized this as well, because iOS and iPadOS 15 are patching many of these holes, and should make FaceTime a viable alternative to the now-ubiquitous Zoom for many more situations.

FaceTime in iOS 15 is finally receiving a proper grid mode in calls with multiple participants. The previous automatically-moving layout, which still exists, felt like it didn’t work well when calls had more than a few participants. Grid mode should improve upon that feeling. FaceTime will also support Apple’s Portrait mode, allowing you to blur your background while keeping yourself in focus.

Most importantly for users familiar with Zoom, FaceTime in iOS and iPadOS 15 will allow users to generate web links for scheduled FaceTime calls. Send out the links and all invitees can use them to join the call together at the specified time. This even works for Android and Windows users, because Apple is making FaceTime available from a web browser for the first time ever. Web-based FaceTime calls remain end-to-end encrypted.

iOS 15 will also bring Apple’s Spatial Audio to FaceTime calls, allowing you to hear voices of participants like they’re coming from a direction relative to their position on your screen. There is a new Voice Isolation feature too, which can block out surrounding background noise and amplify just the sound of your voice. If you’re looking for the opposite effect, switch to Wide Spectrum mode to boost the music or other sounds around you so that your friends can hear what you’re hearing.

Finally, the new FaceTime is introducing a feature called SharePlay. This enables users to expand their video chats with movies, TV shows, music, and screen sharing. Just switch to a participating video app and press play to stream the video to everyone on your call. Playback controls are available to all participants, and play state is synced so that everyone can be confident that they’re watching the same moment together.

Music (via Apple Music only) and screen sharing work the same way, although music adds a shared queue which all participants can add songs to over time. Since FaceTime supports picture-in-picture, participants can keep using their devices while they watch the video stream in a floating window. The picture-in-picture panel can be tapped at any time to reveal playback controls.

You can pass SharePlay streams off to your Apple TV to watch video content there if you’d prefer. While third-party apps will need to add support for SharePlay, Apple has provided a new API to make adoption easy. Big-name video services including Disney+, Hulu, HBO Max, Twitch, TikTok, ESPN+, and more have already committed to supporting the feature.

Messages

iOS 15’s changes to the Messages app revolve mostly around sharing content. Groups of photos that you send will now be shown as stacks instead of in a long list, allowing you to more easily swipe through the images without losing track of the conversation around them. Tap on a stack to open a grid view where you can see and select multiple photos at once.

A more in-depth new feature is called Shared with You. This stretches across the system in iOS 15, allowing users to share various types of content with each other and have it show up in its associated app. For instance, if you share a song from Apple Music to one of your friends in the Messages app, when that friend opens Music on their device they’ll see the song that you sent in a new ‘Shared with You’ section. This section will include a button allowing your friend to respond directly to your Messages thread once they’ve listened to the song.

The updated Messages app will let you “pin” content, which elevates it into your own Shared with You sections and also exposes it more easily in Spotlight searched.

Shared with You works in News, Photos, Safari, Podcasts, and the TV app. We’ll need to get our hands on iOS 15 to properly explore the intricacies of this feature, but it seems quite extensive and embedded throughout the operating system.

iOS 15 also includes improvements to Memoji, such as choosing outfits, customizing headwear with multiple colors, and new accessibility customizations like cochlear implants and oxygen tubes.

Focus

With iOS 15, Apple is introducing new controls to manage notifications and interruptions, and to generally help you stay more focused on whatever task is at hand throughout each day. The marquee feature here is named ‘Focus,’ and it grants users the ability to create custom modes which correspond to a set of rules. For instance, you might create a Focus named ‘Work’ which triggers Do Not Disturb to be on, but still allows notifications to come in from a selection of work-related apps. Maybe you need Slack notifications, but don’t want to hear anything from Messages until you’re finished working.

Focus can set a status message for you which shows up when friends or family try to send you messages. This lets your contacts know when you’re busy, but also allows them to override your notification settings and send a message that bypasses Do Not Disturb if they have something urgent that needs to get through to you.

Focus modes can also include Home Screen customizations, so when you switch to your Work focus you’ll only see work-related apps and widgets on your Home Screen. You can set up these Focus modes to trigger based on your location too, in which case you’ll get a notification when you arrive somewhere which you can tap to enable the corresponding Focus.

You can easily switch your current Focus via Control Center:

Notifications

Apple has slightly refreshed the design of Notifications in iOS 15. The app icons which each notification belongs to have been enlarged, and now sit to the left side of the notification content. Shifting the app icon position and removing the app name allowed each notification to get a bit shorter, so you can see more of them on a single page. For Messages notifications, you’ll now see the contact photos for whoever sent the message displayed on the notification. If you don’t want to hear from an app for a while, you’ll be able to mute specific apps so that they can’t send you notifications for the next hour, or for the rest of the day.

The new notification summary feature delays the delivery of less important notifications. Instead of delivering immediately, they get packaged up and sent to you together in a summary block. You can choose what time of day you want to receive your notification summaries, and Apple will sort the included notifications by priority as determined by on-device machine learning. Apple says that “urgent” messages will always be delivered immediately, but it’s not yet clear how this urgency is determined. During the keynote it was mentioned that messages directly from people (presumably texts, emails, maybe Slack or Discord notifications?) will never be included in the notification summary, but we’ll have to dig into the betas to see how granular the controls are around this feature.

Live Text

Live Text is a new system-wide feature in iOS 15 which allows you to access text recognized within images. We’ve seen earlier iterations of this idea in the past with text recognition in Notes, but now Apple is expanding it to a more ambitious level.

The most obviously impressive aspect of Live Text is in the Camera app. You can now point your camera at any text in the real world and your iPhone or iPad running iOS or iPadOS 15 will automatically recognize and parse it. You can then tap on an icon that appears in the bottom corner and the text will jump out of the image and allow you to copy it, select within it, or drag and drop it into another app on your device. This works great paired with cross-app drag-and-drop (which is making its debut on iPhone in iOS 15).

Live Text also works in the Photos app on existing images, so you can pull the text out later if you don’t want to do it while you’re in the middle of taking a picture. Data Detectors are at work here too, so if the text you’re tapping is a phone number, address, or other recognized item, you’ll be able to automatically call it, open it in Maps, etc.

Live Text doesn’t stop at text though. Using machine learning, its sister feature Visual Look Up can recognize objects such as dog breeds, flowers, landmarks, and more. Tapping on recognized objects will bring up more detailed information about them, and allow you to perform web searches to learn even more.

Live Text and Visual Look Up seem to be the (significantly improved) successors to the smart search feature that has been present in the Photos app for a few years. All of the keywords from Photos search can now be used in Spotlight to run system-wide searches for metadata recognized by Live Text or Visual Look Up.

Live Text supports seven different languages at launch, and will work on all Apple devices running the latest operating systems once they launch later this year.

Photos

The Photos app has expanded its Memories feature even further. Memories can now be set to your favorite songs from Apple Music, and can be customized with color filters. Setting different filters will result in different song choices and transition effects to nail a wider variety of vibes on Memories videos.

While watching a video generated by Memories, you can tap and hold at any time to freeze a photo so that it doesn’t transition away. The song playing over the video will not pause when you do this, but when you let go the remaining video transitions and timings will be automatically altered to match back up with the song’s beat.

If you don’t want to go with the song that was chosen automatically, you can tap the new Music button to get a pop-up interface into Apple Music, allowing you to choose a song manually. This interface will include smart suggestions for other songs that Apple thinks you’ll like which would also fit the vibe of your video.

Wallet

Apple is continuing its quest to make the Wallet app replace all need for a physical wallet or keys. In iOS 15, they’re adding virtual keys for physical locks. This will allow users to lock and unlock their homes, apartments, offices, and hotel rooms using their iPhone.

On top of the above, Apple is working with states in the US to include drivers licenses and other official identification types in the Wallet app. They’re also working with TSA to support identification via the Wallet app at airport security checkpoints.

Finally, car keys in the Wallet app are adding support for Ultra Wideband technology, allowing users to unlock and start their vehicles without removing their phones from their pockets.

All of these new features in Wallet will be rolling out with select partners (and only in the United State for some), so it will probably be years before the majority of iPhone users have access to this technology in their homes, offices, cars, and hotel rooms. While that fact puts a bit of a damper on the excitement here, it’s still great to see Apple building out Wallet with such enthusiasm. The ball is rolling now, albeit slowly, so eventually the dream of a wallet-less and key-less life is going to become a reality.

Weather

Last year Apple acquired Dark Sky, one of the best weather apps around. It’s still not clear how much of that technology has been integrated into iOS at this time, but iOS 15 is launching a refreshed Weather app with some advanced features similar to what Dark Sky offered.

The new Weather app will support more granular weather animations based on conditions of the location that you’re viewing. Apple is boasting that it has thousands of background animation variations to match every type of whether, and these backgrounds will adjust as weather conditions change.

On top of the new backgrounds, Weather in iOS 15 is adding high-resolution weather maps which you can view to see detailed weather conditions across large areas of land. There will also be new notifications available to warn you of impending snow, rain, or other conditions.

Maps

Maps is launching an all-new experience for cities in iOS 15. This will include extraordinarily increased levels of detail, such as individual lanes rendered with dotted or solid lines to indicate passing lanes, crosswalks and bike lanes, trees, elevation levels, and high-quality 3D landmarks. A new nighttime mode will render everything as though the illumination is coming from street lights instead of the sun.

As you approach complicated interchanges, Maps will render them in 3D so that you can see at which points you need to go underneath overpasses or exit onto a particular offshoot of a multi-lane highway. These features are launching on iOS and iPadOS only, but will be coming to CarPlay later this year.

New features for transit riders include pinning favorite routes, and notifications to indicate when it’s time to disembark. In the new advanced city interface, you’ll be able to scan the buildings around yourself using your iPhone or iPad to get an augmented reality view with floating arrows to show what direction you need to walk. These highly advanced city views will only be available in a handful of select cities to start.

The final new addition to Maps is an interactive globe interface, reminiscent of Google Earth. You can pan and zoom in this interface to see detailed maps data for mountain ranges, oceans, rain forests, and more.

Safari

Safari’s design has been refreshed to move the navigation bar to the bottom so that it’s more easily accessible using only your thumb. This new bar floats above your content, and disappears when you scroll. You can swipe side-to-side on the bar to switch between Safari tabs, or swipe up on it to access to the redesigned tab view.

The new Tab Groups feature lets you sort tabs into saved groups, then access them on any of your devices. Safari on iOS and iPadOS 15 have a redesigned start page which features favorites, reading list items, Shared with You links, and Tab Groups.

Finally, the new Safari adds support for web extensions for the first time on iOS and iPadOS. These extensions use the same code as those on macOS, so transitioning Mac extensions to iOS and iPadOS should be fairly straightforward for most developers.

iPadOS 15

Home Screen

iPadOS 15 brings the same features to the Home Screen that we saw on the iPhone last year: widgets and the App Library. There’s nothing too special here, but a few minor changes can be seen. On iPadOS, widgets include a new size which is even bigger than the max size available on iPhone. The App Library is accessible from an icon in the iPadOS Dock rather than by swiping all the way past the leftmost Home Screen page.

Multitasking

Multitasking is getting another refresh in iOS 15, starting with the brand new multitasking icon that sits in the top-center of each app in the status bar. Tap this icon to open the new multitasking menu, which shows a few icons that you can tap to switch the app between full screen, Slide Over, and Split View modes.

When you select Split View from the multitasking menu, and you aren’t already in Split View, your app will slide completely off the edge of the screen and reveal your Home Screen. This allows you to select a second app which you want to pair with the current one in order to create the dual Split View. Once you’ve tapped another app, the first one slides back in so that you can see them both together.

In some cases you might want to bring full-screen focus onto a particular element even when working in Split View. iPadOS 15 supports this use case. For instance, if you’re working in Split View between Mail and another app, but want to take some time to read through an email without distraction, you can tap and hold on that email. This will trigger the new ‘center window’ view, in which the email pops out of Mail and is displayed as a modal window on top of both of your Split View apps. The background apps will get a dark overlay so that you can focus in on the modal. When you’re done, dismiss the modal and you’ll be back in your previous Split View.

To a certain editor-in-chief’s surprise and delight, Apple added a shelf to iPadOS 15. This new interface is used to display all open windows for a particular app so that you can quickly switch between them. The shelf shows up automatically at the bottom when you switch to an app that has multiple windows active. If the app only has one window, the shelf won’t exist until you create a second one. When viewing the shelf you can tap to switch between windows, swipe up on windows to close them, or use the ‘New Window’ button to create a new one.

Another nice new addition to multitasking in iPadOS 15 is the ability to create new Split View app groups directly within the App Switcher. Just open the App Switcher and drag one app on top of another one to create a new Split View, then tap to open it.

When you have a Split View active, you’ll see the multitasking menu icons in the top center of both app windows. You can tap either of these icons to access the multitasking menu for that window, or swipe down from one of the icons to remove its app from the Split View.

Keyboard shortcuts are available for all of the new multitasking commands, and these join a host of other new keyboard shortcuts to access system-level tasks across iPadOS 15.

Notes

The Notes app in iOS and iPadOS 15 supports mentioning other people in shared notes, and tagging notes via the common hashtag method (type a hashtag anywhere in the note to add that tag). A new activity view tracks changes to a note over time.

On iPadOS 15, there’s an all-new system-wide feature called Quick Note, which brings up a miniature Notes interface and allows you to rapidly jot down a note and save it without leaving your current context. If you create a Quick Note from certain apps, such as Safari or Mail, you can easily capture a back link to the particular mail message or webpage to keep reference of it in the note.

You can also highlight and capture text from a webpage into a Quick Note. Next time you visit the webpage, a Quick Note thumbnail will show up so you know that you have notes here. You can then open the note and tap on any text that you saved and Safari will automatically navigate the webpage to that text. This is a great way to pick up where you left off when taking notes on long articles.

Quick Notes can be created on iPadOS 15 via Control Center, via a new Globe Key-Q keyboard shortcut, or by dragging diagonally out from the bottom-right corner of the screen using your Apple Pencil. Each Quick Note is saved into a special folder in the Notes app, from which you can find them on your iPhone and Mac as well.

Translate on iPad

The Translate app has come to iPad in iPadOS 15, and features improvements including Auto Translate and handwriting practice for various languages. Auto Translate recognizes languages automatically and translates text on the fly, allowing you to have a smooth conversation with someone using two languages, with the iPad translating each of your speech into text for the other to read.

Translation is also now a system-wide feature, allowing you to select and translate text in any app. You can use the Translate app in Split View to get easy translations right next to text documents. System-wide translation and Auto Translate in the Translate app are both available on iOS 15 as well, and you can download languages to enable on-device translations.

Swift Playgrounds

Swift Playgrounds has received an enormous upgrade in iPadOS 15. Apple still isn’t changing the name to Xcode, but Playgrounds can now be used to create fully functioning apps for iOS and iPadOS using SwiftUI.

The new Swift Playgrounds can run live previews of apps as you build them, and includes helpful tools to teach you how to write apps in SwiftUI as you go. Once you’re finished with your new app, you can submit it to the App Store directly from Swift Playgrounds.

The update also includes improved code completion for Swift, helping developers move even faster.

Building and shipping apps is a groundbreaking addition to iPadOS, although we’ll have to wait to hear from developers to see exactly how restricting the new environment feels for professional development. Regardless, it’s exciting to see the capability of developing apps come to the iPad. Even if things end up being too restricted this year, now that the feature exists Apple is sure to make it more powerful in the years to come.

Miscellany

• Siri requests can now occur on-device for select queries, significantly increasing the speed and privacy of Siri responses, and allowing requests to be processed with no Internet connection.
• You can now ask Siri to share the current item on your screen, which works automatically for photos, webpages, news, and more. If the item isn’t shareable, Siri will capture a screenshot and share that instead.
• Spotlight Search has been updated with expanded information cards for contacts, as well as for actors, TV shows, Movies, and more.
• The iPhone setup process has been improved, including allowing existing users to back up to iCloud for free regardless of their subscription level so that they can seamlessly restore their new phone via iCloud. These temporary iCloud backups will be stored for three weeks.
• The text magnification loupe is back for improved text selection.
• You can now enable separation notifications which are sent when you leave behind items that are tracked with Find My.
• Apple’s built-in password manager now supports two-factor authentication codes.
• Custom smart lists in Reminders.
• A new location sharing permission allows developers to request your location only a single time, rather than requesting it permanently even if they only need it for a one-off occasion.
• Copying files on iPadOS now shows a progress bar.
• EXIF data can be viewed and edited from the Photos app in iOS and iPadOS 15.
• Shortcuts updated with collapsible blocks and new ‘Stop and Output’ option.
• When your iPhone is powered off, it can now still be located via the Find My network. This intriguing feature definitely requires some further investigation, so we’ll have more information on how it works as the summer progresses.
• An improved date and time picker brings back the old spinning-wheel interface when you tap into the time, but preserves the advancements from last year (including the ability to type in dates and times if you prefer that).
• Android users moving to iOS can now access a “Move to iOS” feature which will help them transfer their photo albums, files, folders, and accessibility settings to their new iPhone.
• A new option in Settings allows users to choose to stay on a previous version of iOS or iPadOS and just receive security updates, opting them out of the nagging notifications to upgrade to the next OS if they aren’t ready to do so.
• New first-party widgets for Find My, Game Center, App Store Today, Sleep, Mail, and People.
• Follow family and friends’ locations more granularly with continuous streaming updates in the Find My app.
• New accessibility features include expanded VoiceOver details, per-app display and text size settings, and background sounds to help minimize distractions.
• The Made for iPhone program now supports bidirectional hearing aids.
• Headphone Accommodations can now import and parse audiograms to show and adjust for the results of hearing tests.
• Developers can now offer subscription management and refunds within their own apps via new APIs, rather than users having to go through the Settings app or contact Apple.

Conclusion

iOS and iPadOS 15 are both significant updates to two of Apple’s flagship operating systems. The new multitasking changes in iPadOS look like promising improvements to a system that was fairly confusing to handle in its previous iteration. FaceTime’s play to gain ground back against Zoom seems helpful to Apple users, although it remains to be seen how effective the browser-based versions will feel. Focus, Live Text, and Quick Note are all very clear wins, and I’m excited to get my hands on them as the beta season progresses.

These updates are full of such a variety of new features that it’s going to take some time to fully digest them all. As usual, the MacStories team will be digging into all of them and more over the summer, so stay tuned for our continued coverage.

iOS and iPadOS 15 are both available as developer betas today, and will be shipping to all users this fall.

---

You can follow all of our WWDC coverage through our WWDC 2021 hub, or subscribe to the dedicated WWDC 2021 RSS feed.