Julian Lepinski has a thoughtful response to last week’s story by Walt Mossberg on Siri’s failures and inconsistencies. In particular, about the way Siri handles failed queries:

Apple’s high-level goal here should be to include responses that increase your faith in Siri’s ability to parse and respond to your question, even when that isn’t immediately possible. Google Search accomplishes this by explaining what they’re showing you, and asking you questions like “_Did you mean ‘when is the debate’?_” when they think you’ve made an error. Beyond increasing your trust in Siri, including questions like this in the responses would also generate a torrent of incredible data to help Apple tune the responses that Siri gives.

Apple has a bias towards failing silently when errors occur, which can be effective when the error rate is low. With Siri, however, this error rate is still quite high and the approach is far less appropriate. When Siri fails, there’s no path to success short of restarting and trying again (the brute force approach).

The comparison between conversational assistants and iOS’ original user interface feels particularly apt. It’d be helpful to know what else to try when Siri doesn’t understand a question.

Walt Mossberg, writing for The Verge, shares some frustrations with using Siri across multiple Apple devices:

In recent weeks, on multiple Apple devices, Siri has been unable to tell me the names of the major party candidates for president and vice president of the United States. Or when they were debating. Or when the Emmy awards show was due to be on. Or the date of the World Series. When I asked it “What is the weather on Crete?” it gave me the weather for Crete, Illinois, a small village which — while I’m sure it’s great — isn’t what most people mean when they ask for the weather _on _Crete, the famous Greek island.

Google Now, on the same Apple devices, using the same voice input, answered every one of these questions clearly and correctly. And that isn’t even Google’s latest digital helper, the new Google Assistant.

It’s a little odd that Mossberg didn’t mention Siri’s new third-party abilities at all, but it’s hard to disagree with the overall assessment.

Like Mossberg, I think Siri has gotten pretty good at transcribing my commands (despite my accent) but it still fails often when it comes to doing stuff with transcribed text. Every example mentioned by Mossberg sounds more of less familiar to me (including the egregious presidential debate one).

Five years on, Siri in iOS 10 is much better than its first version, but it still has to improve in key areas such as consistency of results, timeliness of web-based queries (e.g. Grammys, presidential debates, news stories, etc.), and inferred queries (case in point). Despite the improvements and launch of a developer platform, these aspects are so fundamental to a virtual assistant, even the occasional stumble makes Siri, as Mossberg writes, seem dumb.

Steven Levy has a fascinating inside look at Apple’s artificial intelligence and machine learning efforts on Backchannel. Levy spent most of a day with Eddy Cue, Phil Schiller, Craig Federighi, Tom Gruber, and Alex Acero in a wide-ranging discussion of the products impacted by those efforts. Perhaps the most interesting parts of the interviews revolved around what Levy refers to as the Apple Brain inside the iPhone:

How big is this brain, the dynamic cache that enables machine learning on the iPhone? Somewhat to my surprise when I asked Apple, it provided the information: about 200 megabytes, depending on how much personal information is stored (it’s always deleting older data). This includes information about app usage, interactions with other people, neural net processing, a speech modeler, and “natural language event modeling.” It also has data used for the neural nets that power object recognition, face recognition, and scene classification.

And, according to Apple, it’s all done so your preferences, predilections, and peregrinations are private.

Levy also covers the replacement of Siri’s smarts on July 30, 2014 with neural-net system. The impact according to Eddy Cue was immediate:

This was one of those things where the jump was so significant that you do the test again to make sure that somebody didn’t drop a decimal place.

Many people have commented that Siri has improved over time, but without the context beyond one’s own experience or metrics from Apple, the perceived change has been largely anecdotal. According to Acero, however:

The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases.… That’s mostly due to deep learning and the way we have optimized it — not just the algorithm itself but in the context of the whole end-to-end product.

Levy also delves into whether Apple’s stance on privacy hobbles its ability to effectively implement AI and machine learning. According to Apple, it does not. The most personal information remains on-device in the ‘Apple Brain.’ Other data, which is transmitted to Apple uses techniques like differential privacy, which is coming in iOS 10, to obfuscate a user’s identity.

The entire article is worth a read to get a sense of the breadth and depth of Apple’s AI and machine learning efforts and the impact on its products. It’s also fascinating to see Apple continue to open up on its own terms as a way to rebut recent criticisms leveled against it.

Useful site by Sandro Roth (via Six Colors) to browse every command supported by Siri in Apple’s apps. I almost wish iOS had a similar interface to explore commands. I wonder if we’ll start seeing more sites like this pop up after iOS 10 and SiriKit.

Brian Roemmele makes some interesting points on VocalIQ, a speech/deep learning startup that Apple acquired last year (via Nick Heer):

It is not a secret that Siri has not kept up the pace that just about all of us expected, including some of the Siri team. The passion that Steve had seemed to have been waning deep inside of Apple and the results were Dag and Adam Cheyer moved on and formed Five Six Labs ( A play on V IV in Roman numerals) and Viv.

Tom Gruber, one of the original team members and the chief scientist that created Siri technology, stayed on and continued his work. During most of 2016 and 2017 we will begin to see the results of this work. I call it Siri2 and am very certain Apple will call it something else.

Roemmele has been following all this for a long time, and he adds:

If Apple utilizes just a small subset of the technology developed by VocalIQ, we will see a far more advanced Siri. However I am quite certain the amazing work of Tom Gruber will also be utilized. Additionally the amazing technology from Emollient, Perception and a number of unannounced and future Apple acquistions will also become a big part of Apple’s AI future.

Between these acquisitions and reports that Apple is indeed preparing a Siri API for developers, it sounds like we should expect some notable announcements at WWDC.

See also: this fascinating talk by VocalIQ CEO and founder Blaise Thomson from last June on machine learning applied to voice interactions.

Daniel Jalkut on Siri’s new behavior:

Apple “broke” the haptic feedback associated with invoking Siri, by “fixing” the problem that there had ever been any latency before. Have an iPhone 6s or 6s Plus? Go ahead, I dare you: hold down the home button and start talking to Siri. You will not escape its attention. It’s ready to go when you are, so it would be obnoxious of it to impose any contrived delay or to give taptic feedback that is uncalled for. Siri has become a more perfect assistant, and we have to change our habits to accommodate this.

Great little detail of Siri that I didn’t notice until today. Siri seems to agree, too.