Messages, Animoji, and FaceTime

In a perfect segue from digital well-being and notification overload, let’s talk about how you can turn yourself into an Animoji and torment your friends with sticker-laden selfies in iOS 12.

Messages

Aside from new types of media you can share over iMessage, the design and structure of the Messages app remains mostly unchanged from iOS 10 and iOS 11. The only exception is a highly contentious change to one of its built-in sharing features.

In iOS 12, sending photos and videos over iMessage is done via a new standalone Photos app inside Messages. Sending an item from your library has been completely ripped out of the camera experience and now exists as the first, default app across the Messages app strip. While there is still a camera button next to the App Store logo flanking the compose field, the camera is now dedicated to capturing media and applying effects; tapping it doesn’t bring up a grid of recent photos anymore.

Photos and camera are now two separate features in Messages for iOS 12.

For the first few weeks of iOS 12, you will not adjust easily. You’ll constantly hit the camera button, thinking that you can then tap on the bottom left corner to open your photo library, only to discover that you’re about to capture a selfie of your perplexed face. Trust me – I’ve been there. For many weeks. Since iOS 5, we’ve been trained to share photos on iMessage with the same Tap-Tap-Choose-Send dance. You’re going to see articles from folks who install iOS 12 and complain about the worst new iOS feature of the year. There will be tweets and opinions. Allow me to share mine.

Apple is mitigating the transition to Photos as an iMessage app with a library button in the camera view that dismisses the camera and loads the Photos app in compact mode instead.¹⁰ The company obviously knows that users will try to share pictures the old way, and I think the button is a clever solution¹¹ to teach users about the existence of the new Photos app.

The new Messages camera flow in iOS 12.

More broadly, switching Photos to be a standalone iMessage app takes a while to understand, but I believe it is a necessary medicine for the photo sharing experience to become more powerful and integrated with iOS. Here’s why: not only does the new iMessage Photos app replicate the feature set of iOS 11 (you can share recent photos and videos or view the complete library with an ‘All Photos’ button), but sharing suggestions have been integrated with it as well. By expanding the Photos app to its full layout¹², you’ll see a subset of sharing suggestions inside Messages, ready to be sent to participants in the conversation.

Sharing suggestions are available by expanding the Photos app.

A slightly tweaked layout for expanded iMessage apps in iOS 12.

These sharing suggestions are the same ones built into iOS 12’s Photos app. They are generated with on-device machine learning that uses facial recognition and environmental triggers (location, time of the day, etc.) to create collections of photos you can share with people recognized in them. In the Photos app for iMessage, you can preview sharing suggestions, which lets you remove individual photos then share them in a conversation as a rich “album bubble” that others can preview, add to their libraries, and expand with the new Share Back feature (see Photos chapter).

Even better though, iOS 12’s intelligence prioritizes certain sharing suggestions over others depending on the conversation you’re in. So while a ‘Home’ suggestion is at the top of the Photos app in the iMessage thread with my girlfriend, a different ‘London’ album is the first suggestion in my conversation with Myke, Stephen, and John. According to Apple, this analysis happens locally, on-device, without further cloud processing required.

Data detectors and the QuickType keyboard have also been updated to recognize when someone asks you to share photos with them. If you receive a message with a request to share photos with another person, the word “photos” will be underlined in the transcript, and you may see a ‘Choose Photos’ button in the QuickType keyboard. Tapping either will load the Photos app for iMessage in compact mode.

iOS 12 may decide to underline words (on the right, “video”) or embed a photo picker in the QuickType prediction bar.

Given Apple’s investment on smart sharing suggestions in iOS 12, it’s not surprising to see them as the fulcrum of Messages’ revamped photo sharing flow. Apple wants – nay, needs – to ensure that people use this feature: it can act as further lock-in for iMessage as well as compensate for the lack of true shared libraries among family members. From a strategic standpoint, it’s smart to rebrand the traditional photo picker as a full-fledged extension of the Photos app that carries its iCloud-only perks over to iMessage.

Once you get used to opening the Photos app first and toggling between typing suggestions and the app strip¹³, the new Messages structure starts to click: tapping the camera now instantly takes you to a full-featured camera mode; when you’re typing, toggling the app strip doesn’t take over the keyboard anymore but simply alternates between predictive suggestions and app icons. As a result, taking pictures is faster and choosing from existing ones is easier and more powerful thanks to sharing suggestions.

In hindsight, the old camera button felt like a compromise between taking pictures and browsing them; with Messages in iOS 12, these features are two distinct modes, each with their own space and room for growth.

Messages Camera

At a high level, Apple’s goal with the new Messages camera was to let users be taken into the real iPhone camera experience, not a lesser version in a tiny preview screen. They were successful in this transition. When you tap the camera button in Messages, the regular camera view now comes up in full-screen, offering the complete set of features from the main Camera app. You can take portrait shots, record video, and adjust options without having to switch to a “full camera mode” as was the case in iOS 11.

You’ll run into changes when you either take a picture or tap the star button in the lower left corner. Once a picture is taken, the photo preview “freezes” in the middle of the screen and reveals three buttons along the bottom (Effects, Edit, and Markup), plus a large blue send button to share the item in the current conversation. ‘Edit’ and ‘Markup’ are the exact same features you can access in the Photos app or elsewhere in iOS.

Taking pictures in Messages has a new editing flow in iOS 12.

The Effects button is new in iOS 12: it combines existing photo filters, iMessage stickers, Animoji, text annotations, and shapes in a new media context that lets you creatively modify pictures and videos with an experience comparable to Instagram or Snapchat.

James, why did you have to do this to me?

Please don’t judge me.

Messages offered a selection of fun editing tools before, but the concept is more cohesively implemented in iOS 12 – particularly thanks to stickers from iMessage apps, which can now become part of a photo or video. With enough patience and dedication, you can turn your pictures into hilarious monstrosities by mixing and matching shapes or text and attaching stickers to any point of the scene. It reminds me of Apple’s Clips app, and it’s clearly designed for a younger generation of users who find it perfectly normal to complement their snaps with stickers and shapes. Even though I’ll seldom use these tools, I’m sure millions of other people will.

The similarities with Clips are more apparent when you tap on the Effects button before taking a picture or recording a video.¹⁴ This mode is Apple’s showcase for the TrueDepth camera system and powerful hardware in the latest iPhones. When selected beforehand, Effects lets you take pictures and record videos with Animoji, Memoji (more on this in a bit), Clips-like real-time effects, and stickers that support face tracking.

Playing around with these features can either lead to the creation of modern masterpieces or go dramatically wrong. In any case, the camera’s Effects mode is the most fun and jovial Messages has been in years. Forget about Digital Touch¹⁵ – iOS 12’s Messages is all about giant Animoji heads and over-the-top filters. It’s messy, it’s silly, and my girlfriend hates it when I ask her questions as a deadpan T-Rex. I love it.

I want to cover the technical aspect of face tracking stickers, which I find to be an interesting addition for developers. In iOS 12, Messages extensions (both apps with stickers and standalone sticker packs) can support two contexts: Messages, for user interaction in a conversation, and Media, which applies to the Messages camera and FaceTime. The Media context only supports a subset of features provided by the Messages framework; for instance, when shown in the camera, extensions can’t bring up text fields. Interactions in the Media environment are limited to placing stickers in the viewfinder with support for rotation, resizing, and removal of stickers.

When you start dragging a sticker that has been updated for iOS 12’s Media context, you’ll see the familiar yellow rectangle for face recognition and tracking. This allows you to attach a sticker to your face and have the system maintain its position as you move your head around. After you’ve placed a sticker on your face, iOS 12 takes care of tracking it and scaling it appropriately, which includes automatic resizing as you move backwards and forwards. Sticker developers don’t have to do any additional coding for this to happen; it’s all managed by the system and its built-in face tracking APIs.

In addition to Apple’s default sticker packs, I was able to play around with Lemonade, a third-party sticker app that received support for iOS 12’s Media context earlier this summer. The technology underlying this relatively minor feature is remarkable: Lemonade’s stickers, which include a gorilla, politician heads, and other creatures, can be attached to a recognized face and are continuously tracked as you move around in videos or tilt your head back and forth while talking.

Even more impressively, iOS 12 makes it possible to stack and track multiple effects at the same time: in my very professional tests, I combined Animoji, hat stickers placed on the top of my cartoon head, and backdrop effects while recording videos that I shared with my friends for their enjoyment.

I now fully expect all sticker packs to be updated for iOS 12 and its Media context in Messages and FaceTime. While they may not be as technically impressive as the real-time, TrueDepth-enabled face masks of Snapchat¹⁶, they’re fun to play with and can liven up conversations quite a bit.

Effects and stickers, however, aren’t the real stars of the Messages experience in iOS 12. While Apple has done a solid job at reinvigorating the camera and photo sharing flows, Animoji are once again stealing the show.

Memoji and the Expansion of Animoji

As I argued in my iPhone X story from last year, Animoji are more than a simple 3D adaptation of standard emoji: under the hood, they’ve always been a technical showcase for the TrueDepth camera system – an array of sensors and other technologies that is expanding to new iPhones and, quite possibly, iPads this year. In addition to widening TrueDepth’s reach to more devices, in iOS 12 Apple is also spreading Animoji across the system and enhancing their basic capabilities.

As a first order of (very serious) business, there are four new Animoji in iOS 12: tiger, koala, T-Rex, and ghost. The last two are hysterical given the dinosaur’s cartoonish look (in contrast with Hollywood’s celebrated representation of the creature) and the ghost’s suspicious presence of lips. I never thought I would be typing this in an iOS review, but I have found myself attempting to capture a duckface as the ghost Animoji, and I’m happy to report that it can be done.

New Animoji characters in iOS 12.

In a welcome improvement from last year, Animoji now feature wink and tongue detection. This means you can, well, wink while recording an Animoji video and stick your tongue out as well. Alas, tongue detection isn’t real-time tongue animation: unlike head movements, TrueDepth can’t capture you moving your tongue around – it only supports a single animation to stick it out. Still, I found tongue detection to suffice when putting together videos and selfies with a wider range of expressions than before.

Speaking of video, while Animoji Karaoke might have been an entertaining fad, Apple is making it easier to share custom Animoji video by extending their duration. In iOS 12, you can record Animoji videos that are up to 30 seconds long – 20 seconds longer than before. Unfortunately, you still have to record Animoji video in Messages as there’s no dedicated Animoji app available.¹⁷

The most important addition to Animoji in iOS 12, however, isn’t the inclusion of new built-in characters. It’s the fact that you can now create an Animoji for yourself through a new mode called Memoji. I believe that this feature will singlehandedly convince existing iPhone X owners to upgrade to iOS 12. I also think it will turn out to be one of the most popular social functionalities bundled with iOS and Messages, perhaps second only to new emoji. Memoji are fun, expressive, not creepy, and integrated with the system in a bunch of clever ways.

Everyday life Ticci (left) and beach lifestyle Ticci.

Memoji have to be created manually with an avatar creation tool reminiscent of Nintendo’s Mii maker or the Xbox’s avatar editor. Unlike similar features by other companies, Memoji don’t use facial recognition to automatically assemble a 3D avatar by scanning your face.

Creating a Memoji character is reminiscent of Nintendo’s original Mii editor, albeit with fewer options.

The Memoji creation tool is included with the Animoji app in Messages and guides you through the process of selecting a skin color and facial features such as hairstyle, eye color and shape, nose, lips, and even eyewear and headwear. Creating a Memoji character happens in a full-screen view inside Messages, with a real-time preview at the top and controls for color pickers and different categories in the lower half.

For a 1.0 feature, Memoji are surprisingly packed with features and build atop emoji and Animoji with options that always veer toward a pleasantly cartoonish style. Skin colors, for instance, include the classic emoji yellow, plus purple, green, blue, and others; the eye color tool comes with realistic options such as green and blue but also includes a button for brown/aquamarine gradients and artificial colors like pink and purple; if you want, you can create a Memoji that has a green face and a long pink beard.

You can spend hours playing around with Memoji to replicate your favorite superhero or cartoon character – just like people did when Miis first launched on the Nintendo Wii console. And because multiple Memoji can be stored and duplicated in the Animoji app, I expect the most dedicated users to come up with wild reinterpretations of their faces as well as popular characters from videogames, movies, or TV shows. There’s enough freedom to go beyond simply turning your basic self into an Animoji; I wouldn’t be surprised if Animoji Karaoke naturally evolves into Memoji movie trailers thanks to iOS 12.

Once you’ve created a Memoji, using it in the Messages app follows the same interaction model of Animoji. You can tap on a Memoji to send it as a static sticker, or you can record a video of yourself making silly faces and send it to your friends over iMessage. Like Animoji, you can’t save a Memoji video to your camera roll directly, but, once shared in a conversation, you can open the video in a full-screen preview and save it locally.

You can share Memoji like regular iMessage stickers.

In addition to the iMessage app, Memoji and Animoji have also been integrated as effects in the Messages camera in iOS 12. By tapping the Animoji app inside the camera when in front-facing mode, you’ll be able to take selfies and record videos featuring a giant Memoji head¹⁸ instead of your real face. The result is hilarious because, as I mentioned above, you can stack Memoji/Animoji together with other effects and stickers while you’re also recording a video using your voice.¹⁹ One of my favorite Animoji videos created during the iOS 12 beta features me pretending to roar like a dinosaur against a cell-shaded background; my parents are also fans of my Memoji characters, although they insist I should get a haircut soon.

I would have liked to see more representation in Memoji for different types of accessories (nose ring and septum are absent from this release) as well as more inclusive options, such as hearing aids; perplexingly, not even AirPods or standard headphones made the cut. I suppose that new facial features (tattoos?), makeup options, and more accessories will be progressively added in future updates to the Animoji app. There are hundreds of missing options that Apple should implement before even considering full-body Memoji characters.

Memoji strike a good balance of generating realistic characters that aren’t horrifying and blending Apple’s emoji design sensibility with non-emoji avatars. By making every facial feature delightfully cartoonish, Apple eschewed the problem of users creating unsettling Memoji that look like tiny humans printed in a factory. Instead, Memoji feel like a successful cross-over of emoji and Bitmoji, powered by the TrueDepth camera and nicely integrated with iMessage. I bet we’re going to see a lot of these in our conversations going forward.

FaceTime

After neglecting its audio/video communication service for years, Apple wanted to rebuild FaceTime in iOS 12 from the ground up as a new app revolving around three key improvements: a new design, integration with camera effects, and group conversations. Unfortunately, only two out of three goals have been achieved in time for iOS 12’s initial release as group conversations have been delayed to later this fall.

The new FaceTime UI in iOS 12.

The new FaceTime UI follows the trend of moving essential interface elements toward the bottom of the screen where they’re easier to reach with one hand (an aesthetic already implemented in Apple Maps and coming to the Stocks app as well). FaceTime for iOS 12 features a new panel in the lower third of the screen with buttons to apply effects in real-time, end the current call, and access other settings.

In the new FaceTime, flipping the camera requires an extra tap.

The latter option is noteworthy from a design standpoint as it expands the panel upwards to reveal other call-related functionalities. These include buttons to flip the camera or turn it off, mute audio, and change speaker settings. The expanded panel also allows you to jump directly to the iMessage thread for the person you’re currently chatting with – a handy way to share pictures and links during a FaceTime call.

The new call management screen works well: it collects useful controls in a dashboard that removes complexity from the main call UI, allowing people to be front and center. The design change has been thoughtfully ported to iPad too, where the minimized card is a floating panel in the bottom left corner that can be expanded into a Maps-like sidebar with embedded controls. If this is the direction Apple is taking for a future (and long-awaited) new iOS design language, I like where the company is going.

Using the Effects camera mode in FaceTime can be a fun diversion from traditional video calls. When you launch into Effects mode, the view from your front-facing camera takes over the call and lets you preview effects in real-time. The design is the same as the Messages camera: there’s an app strip at the bottom for browsing the same effects available in Messages, including filters and stickers.

I’ve always wanted to FaceTime my friends as a dinosaur.

On TrueDepth-enabled devices, you can turn your face into a Memoji or Animoji and let other people talk to a monkey, a dragon, or your Memoji character. On older iPhones and iPads, you only get access to filters and stickers, which you can attach to your face or any other area captured by the camera. I don’t think people are going to regularly video-call one another with effects, but they’re fun to play around with for a few times regardless. As with Messages, it’s a neat technical showcase and performance is solid.

This section would have been longer, and more enthusiastic of FaceTime in iOS 12, had Apple managed to ship group conversations as they initially promised at WWDC. Instead, while I think Apple has done a good job with the new FaceTime app’s layout, its new simplified design is obviously waiting for a feature that isn’t here yet. Effects are fun, and the new call management screen makes FaceTime easier to use than before, but neither option is what FaceTime truly needs. The current version of FaceTime in iOS 12 is a product on hold, waiting for another shoe to drop. I’m glad that Apple is taking its time to make group conversations work well, but their absence can be felt in a redesigned app that feels simultaneously brand new and stuck in the past.

In iOS 12, Apple aims to provide a suite of communication services that integrate deeply with each other and the OS, across a wide range of platforms and devices, with an interplay of hardware and software as their cornerstone. This is apparent with Animoji and TrueDepth blending with FaceTime and the Messages camera; with FaceTime Audio now having consistent support on iOS, macOS, watchOS, and HomePod; and it’ll be evident once FaceTime can scale from individual to group conversations.

Communicating with friends and family on Apple platforms increasingly means turning to a rich ecosystem of services that yield a more personal and holistic experience. Looking ahead to iOS 12’s cycle through the rest of 2018, it’s clear that Apple doesn’t want to make simple communication apps anymore. We just need to wait a few more months for this vision to be fully realized.