This Week's Sponsor:

DEVONTHINK

Store, Organize, and Work the Smart Way

Posts tagged with "siri"

Notes on the Apple Intelligence Delay

Simon Willison, one of the more authoritative independent voices in the LLM space right now, published a good theory on what may have happened with Apple’s delay of Apple Intelligence’s Siri personalization features:

I have a hunch that this delay might relate to security.

These new Apple Intelligence features involve Siri responding to requests to access information in applications and then perform actions on the user’s behalf.

This is the worst possible combination for prompt injection attacks! Any time an LLM-based system has access to private data, tools it can call and potentially malicious instructions (like emails and text messages from untrusted strangers) there’s a risk that an attacker might subvert those tools and use them to damage or exfiltration a user’s data.

Willison has been writing about prompt injection attacks since 2023. We know that Mail’s AI summaries were (at least initially?) sort of susceptible to prompt injections (using hidden HTML elements), as were Writing Tools during the beta period. It’s scary to imagine what would happen with a well-crafted prompt injection when the attack’s surface area becomes the entire assistant directly plugged into your favorite apps with your data. But then again, one has to wonder why these features were demoed at all at Apple’s biggest software event last year and if those previews – absent a real, in-person event – were actually animated prototypes.

On this note, I disagree with Jason Snell’s idea that previewing Apple Intelligence last year was a good move no matter what. Are we sure that “nobody is looking” at Apple’s position in the AI space right now and that Siri isn’t continuing down its path of damaging Apple’s software reputation, like MobileMe did? As a reminder, the iPhone 16 lineup was advertised as “built for Apple Intelligence” in commercials, interviews, and Apple’s website.

If the company’s executives are so certain that the 2024 marketing blitz worked, why are they pulling Apple Intelligence ads from YouTube when “nobody is looking”?

On another security note: knowing Apple’s penchant for user permission prompts (Shortcuts and macOS are the worst offenders), I wouldn’t be surprised if the company tried to mitigate Siri’s potential hallucinations and/or the risk of prompt injections with permission dialogs everywhere, and later realized the experience was terrible. Remember: Apple announced an App Intents-driven system with assistant schemas that included actions for your web browser, file manager, camera, and more. Getting any of those actions wrong (think: worse than not picking your mom up at the airport, but actually deleting some of your documents) could have pretty disastrous consequences.

Regardless of what happened, here’s the kicker: according to Mark Gurman, “some within Apple’s AI division” believe that the delayed Apple Intelligence features may be scrapped altogether and replaced by a new system rebuilt from scratch. From his story, pay close attention to this paragraph:

There are also concerns internally that fixing Siri will require having more powerful AI models run on Apple’s devices. That could strain the hardware, meaning Apple either has to reduce its set of features or make the models run more slowly on current or older devices. It would also require upping the hardware capabilities of future products to make the features run at full strength.

Inference costs may have gone down over the past 12 months and context windows may have gotten bigger, but I’m guessing there’s only so much you can do locally with 8 GB of RAM when you have to draw on the user’s personal context across (potentially) dozens of different apps, and then have conversations with the user about those results. It’ll be interesting to watch what Apple does here within the next 1-2 years: more RAM for the same price on iPhones, even more tasks handed off to Private Cloud Compute, or a combination of both?

We’ll see how this will play out at WWDC 2025 and beyond. I continue to think that Apple and Google have the most exciting takes on AI in terms of applying the technology to user’s phones and apps they use everyday. The only difference is that one company’s announcements were theoretical, and the other’s are shipping today. It seems clear now that Apple got caught off guard by LLMs while they were going down the Vision Pro path, and I’ll be curious to see how their marketing strategy will play out in the coming months.

Apple Delays Siri Personalization→

Apple released a statement to John Gruber of Daring Fireball today announcing that it is delaying a “more personalized Siri.” According to Apple’s Jacqueline Roy:

Siri helps our users find what they need and get things done quickly, and in just the past six months, we’ve made Siri more conversational, introduced new features like type to Siri and product knowledge, and added an integration with ChatGPT. We’ve also been working on a more personalized Siri, giving it more awareness of your personal context, as well as the ability to take action for you within and across your apps. It’s going to take us longer than we thought to deliver on these features and we anticipate rolling them out in the coming year.

This isn’t surprising given where things stand with Siri and Apple Intelligence more generally, but it is still disappointing. Of all the features shown off at WWDC last year, the ability to have Siri take actions in multiple apps on your behalf through natural language requests was one of the most eagerly anticipated. But, I’d prefer to get a feature that works than one that is half-baked.

Still, you have to wonder where the rest of the AI market will be by the time a “more personalized Siri” is released and whether it will look as much like yesterday’s tech as some of today’s Apple Intelligence features do.

Permalink

“Everyone Is Caught Up, Except for Apple”→

Good post by Parker Ortolani (who’s blogging more frequently now; I recommend subscribing to his blog) on the new (and surprisingly good looking?) Alexa+ and where Apple stands with Siri:

So here we are. Everyone is caught up, except for Apple. Siri may have a pretty glowing animation but it is not even remotely the same kind of personal assistant that these others are. Even the version of Siri shown at WWDC last year doesn’t appear to be quite as powerful as Alexa+. Who knows how good the app intents powered Siri will even be at the end of the day when it ships, after all according to reports it has been pushed back and looks like an increasingly difficult endeavor. I obviously want Siri to be great. It desperately needs improvement, not just to compete but to make using an iPhone an even better experience.

I continue to think that Apple has immense potential for Apple Intelligence and Siri if they get both to work right with their ecosystem. But at this point, I have to wonder if we’ll see GTA 6 before Siri gets any good.

Permalink

Apple’s Commitment to AI Is Clear, But Its Execution Is Uneven

The day has finally arrived. iOS 18.1, iPadOS 18.1, and macOS 15.1 are all out and include Apple’s first major foray into the world of artificial intelligence. Of course, Apple is no stranger to AI and machine learning, but it became the narrative that the company was behind on AI because it didn’t market any of its OS features as such. Nor did it have anything resembling the generative AI tools from OpenAI, Midjourney, or a host of other companies.

However, with today’s OS updates, that has begun to change. Each update released today includes a far deeper set of new features than any other ‘.1’ release I can remember. Not only are the releases stuffed with a suite of artificial intelligence tools that Apple collectively refers to as Apple Intelligence, but there are a bunch of other new features that Niléane has written about, too.

The company is tackling AI in a unique and very Apple way that goes beyond just the marketing name the features have been given. As users have come to expect, Apple is taking an integrated approach. You don’t have to use a chatbot to do everything from proofreading text to summarizing articles; instead, Apple Intelligence is sprinkled throughout Apple’s OSes and system apps in ways that make them convenient to use with existing workflows.

If you don't want to use Apple Intelligence, you can turn it off with a single toggle in each OS's settings.

If you don’t want to use Apple Intelligence, you can turn it off with a single toggle in each OS’s settings.

Apple also recognizes that not everyone is a fan of AI tools, so they’re just as easy to ignore or turn off completely from System Settings on a Mac or Settings on an iPhone or iPad. Users are in control of the experience and their data, which is refreshing since that’s far from given in the broader AI industry.

The Apple Intelligence features themselves are a decidedly mixed bag, though. Some I like, but others don’t work very well or aren’t especially useful. To be fair, Apple has said that Apple Intelligence is a beta feature. This isn’t the first time that the company has given a feature the “beta” label even after it’s been released widely and is no longer part of the official developer or public beta programs. However, it’s still an unusual move and seems to reveal the pressure Apple is under to demonstrate its AI bona fides. Whatever the reasons behind the release, there’s no escaping the fact that most of the Apple Intelligence features we see today feel unfinished and unpolished, while others remain months away from release.

Still, it’s very early days for Apple Intelligence. These features will eventually graduate from betas to final products, and along the way, I expect they’ll improve. They may not be perfect, but what is certain from the extent of today’s releases and what has already been previewed in the developer beta of iOS 18.2, iPadOS 18.2, and macOS 15.2 is that Apple Intelligence is going to be a major component of Apple’s OSes going forward, so let’s look at what’s available today, what works, and what needs more attention.

Read more

The New York Times Declares that Voice Assistants Have Lost the ‘AI Race’→

Brian Chen, Nico Grant, and Karen Weise of The New York Times set out to explain why voice assistants like Siri, Alexa, and Google Assistant seem primitive by comparison to ChatGPT. According to ex-Apple, Amazon, and Google engineers and employees, the difference is grounded in the approach the companies took with their assistants:

The assistants and the chatbots are based on different flavors of A.I. Chatbots are powered by what are known as large language models, which are systems trained to recognize and generate text based on enormous data sets scraped off the web. They can then suggest words to complete a sentence.

In contrast, Siri, Alexa and Google Assistant are essentially what are known as command-and-control systems. These can understand a finite list of questions and requests like “What’s the weather in New York City?” or “Turn on the bedroom lights.” If a user asks the virtual assistant to do something that is not in its code, the bot simply says it can’t help.

In the case of Siri, former Apple engineer John Burkey said the company’s assistant was designed as a monolithic database that took weeks to update with new capabilities. Burkey left Apple in 2016 after less than two years at the company according to his LinkedIn bio. According to other unnamed Apple sources, the company has been testing AI based on large language models in the years since Burkey’s departure:

At Apple’s headquarters last month, the company held its annual A.I. summit, an internal event for employees to learn about its large language model and other A.I. tools, two people who were briefed on the program said. Many engineers, including members of the Siri team, have been testing language-generating concepts every week, the people said.

It’s not surprising that sources have told The New York Times that Apple is researching the latest advances in artificial intelligence. All you have to do is visit the company’s Machine Learning Research website to see that. But to declare a winner in ‘the AI race’ based on the architecture of where voice assistants started compared to today’s chatbots is a bit facile. Voice assistants may be primitive by comparison to chatbots, but it’s far too early to count Apple, Google, or Amazon out or declare the race over, for that matter.

Permalink

Apple’s Fall OS Updates Promise Deeper HomeKit and Entertainment Integration

Apple’s fall OS updates will include a variety of HomeKit and home entertainment features. Unsurprisingly, some of those changes can be found in the company’s Home and TV apps, but this year, those apps only tell part of the overall story. To get the full picture, you need to zoom out from the apps, where you’ll find an interesting mix of new smart home device and entertainment features sprinkled throughout each platform.

Let’s start with HomeKit devices. This year, many of the changes coming to Apple’s OSes relate to two important categories: video cameras and door locks. Controlling both types of devices will become easier this fall, thanks to deeper integration with the upcoming OS releases.

Read more

WWDC 2021: All The Small Things in Apple’s Upcoming OS Releases

WWDC keynotes cover a lot of ground, hitting the highlights of the OS updates Apple plans to release in the fall. However, as the week progresses, new details emerge from session videos, developers trying new frameworks, and others who bravely install the first OS betas. So, as with past WWDCs, we’ve supplemented our iOS and iPadOS 15, macOS Monterey, and watchOS 8, and tvOS 15 coverage with all the small things we’ve found interesting this week:

Read more

Siri Adds Two New English Speaking Voices and Lets Users Choose Among Them→

Matthew Panzarino, reporting for TechCrunch says the latest beta version of iOS and iPadOS 14.5 includes two new English Siri voices. The report elaborates that the existing female voice is no longer the default and that users will choose the voice they want to use with Apple’s voice assistant when setting up a device for the first time.

In a statement to TechCrunch, an Apple said:

We’re excited to introduce two new Siri voices for English speakers and the option for Siri users to select the voice they want when they set up their device. This is a continuation of Apple’s long-standing commitment to diversity and inclusion, and products and services that are designed to better reflect the diversity of the world we live in.

Panzarino says he’s heard the new voices and likes them a lot and will be embedding samples in his story once he has the sixth iOS 14.5 beta installed.

I’m surprised that Apple is adding new Siri voices this late in the iOS 14 cycle, but it’s a welcome change that eliminates bias and makes Siri a more diverse and inclusive service.

Permalink

Two Months with the HomePod mini: More Than Meets the Eye

As a smaller, affordable smart speaker tightly integrated with Apple services, the HomePod mini is a compelling product for many people. The mini is little enough to work just about anywhere in most homes. At $99, the device’s price tag also fits more budgets and makes multiple HomePod minis a far more realistic option than multiple original HomePods ever were. Of course, the mini comes with tradeoffs compared to its larger, more expensive sibling, which I’ll get into, but for many people, it’s a terrific alternative.

As compelling as the HomePod mini is as a speaker, though, its potential as a smart device reaches beyond the original HomePod in ways that have far greater implications for Apple’s place in customers’ homes. Part of the story is the mini’s ability to serve as a border router for Thread-compatible smart devices, forming a low-power, mesh network that can operate independently of your Wi-Fi setup. The other part of the story is the way the mini extends Siri throughout your home. Apple’s smart assistant still has room to improve. However, the promise of a ubiquitous audio interface to Apple services, apps, HomeKit devices, and the Internet is more compelling than ever as Siri-enabled devices proliferate.

For the past couple of months, I’ve been testing a pair of HomePod minis that Apple sent me. That pair joined my original HomePods and another pair of minis that I added to the setup to get a sense of what having a whole-home audio system with Siri always within earshot would be like. The result is a more flexible system that outshines its individual parts and should improve over time as the HomeKit device market evolves.

Read more