Notes from Apple’s Siri AI Labs: Schemas, Tools, and Skills

Earlier this week, I attended Apple’s online Apple Intelligence Group Lab, much of which focused on Siri AI. Much of the ground covered is available in Apple’s documentation, but the presentation included additional detail, which is worth highlighting, although I encourage you to watch the two Apple Intelligence Group Labs for more context and details.

The big-picture change that Federico explained earlier this week is that Apple is greatly expanding the actions available to Siri. Today, Siri AI’s natural-language abilities are limited to tightly defined domains like calendar, reminders, and messages. If a developer’s app doesn’t fit a schema (or hasn’t implemented an App Shortcut), Siri won’t be able to respond to a user.

Apple isn’t dispensing with schemas, but it’s loosening the reins, giving developers far wider latitude than before. Apple noted during the lab that schemas are still developers’ best bet for getting Siri to respond because Apple’s already done the work to ensure they work.

What’s changing is that Siri will be able to directly access developers’ App Intents that have been defined as tools. That’s a big change from today where a developer’s main access to Siri is through App Shortcuts, using a defined phrase that names the app. Instead, if a developer marks their App Intents as tools, Siri will be able to access those App Intents directly when a user makes a natural-language request, much like how MCP works with tools like Codex and Claude Code.

Other interesting tidbits about tools mentioned include that they’ll use Private Cloud Compute to process the request and that apps are limited to 40 tools.

Skills are a third brand-new way to access Siri that’s designed for more complex, multistep actions that can chain up to 32 tool calls. Like the skills you may be familiar with from Codex and Claude Code, Siri’s skills are Markdown instructions with YAML front matter that explain what Siri should do with certain requests. Xcode 27.3 beta also includes coding-agent skills that help developers add Siri support to their apps and test them.

Finally, beginning with iOS 27.3, which will ship in January if history is any indicator, developers will be able to customize the look of snippets of information from their apps similar to how Apple’s own apps’ content can look in Siri. As for tools and skills, those will be available to ship to the App Store in the spring.

My Favorite iPhone Duo Productivity Apps, Vol. 1

As I hinted in my story about multi-window on the iPhone Duo, I find myself increasingly relying on Apple’s diminutive foldable as a pocket computer for getting work done with an external keyboard. It reminds me of the early days of iPad productivity: there’s something to be said about turning the multitasking limitations of the machine into a way to focus more and be less incentivized to switch between apps. Also, you know me: it’s fun to prove a point and show how you can get work done on an iPhone Duo if you want to.

As part of this, we’re dealing with the fact that we’ve received almost 600 submissions from developers about their iPhone Duo apps and app updates. I haven’t seen this kind of influx of new apps since the early days of the iPad (again, another parallel between these devices). John and I are working hard to make sense of all these app pitches, picking the ones that are worth our readers’ attention, time, and money. In the era of software abundance caused by AI, I think it’s more important than ever to test and curate software that wasn’t simply put together with two Claude Code prompts. This is why, rather than assembling an overflowing directory of iPhone Duo apps with no editorial curation, we’re going to write these old-fashioned roundups instead.

In the past two weeks since I’ve been testing the iPhone Duo, I’ve been impressed by the variety of productivity apps that have landed on the App Store. Most of these strike a fascinating balance of something in between an iPadOS and iOS experience – which, I guess, is the whole point of this new breed of pocket computer.

Below are some of my early favorites, which I’m using every day on my iPhone Duo. Expect more to follow over the next few weeks.

Read more

Apple: More Than 100,000 iPhone Duo Apps Are Now on the App Store

Source: Apple.

Source: Apple.

It’s been clear to us at MacStories that the iPhone Duo is a smash hit with developers. With a stack of over 600 app pitches to date, we haven’t seen this much developer interest in a new Apple gadget since the release of the iPad in 2010.

That’s a lot of app pitches, but it’s also just the tip of the iceberg, as Susan Prescott, Apple’s vice president of Worldwide Developer Relations, revealed in a statement today:

It’s been wonderful to see how quickly developers have embraced iPhone Duo, and we’re grateful for their creativity and innovation. Today there are well over 100,000 apps built for iPhone Duo already live on the App Store, and thousands of optimized apps are being added to the store each day. In the last month alone, thousands of developers have joined us for workshops in 19 cities around the world to experience iPhone Duo firsthand and bring updated apps to their users. The enthusiasm has been incredible, and we’re excited to see the pace of updates continue to accelerate as more and more apps optimize for iPhone Duo.

That’s a lot of apps. It helps that every developer I’ve spoken to immediately remarks on how easy it has been to adapt their apps for the iPhone Duo.

What continues to be striking to me is the breadth and depth of the apps that have been adapted for the iPhone Duo. It’s not unusual to see indie developers that make apps like Halide Mark III and Goodnotes jump on board with a device like the Duo, but it’s also found support among much larger players with apps like Microsoft Word and Outlook supporting the Duo, along with apps from Zoom, Etsy, the NBA, the NHL, and the many apps Apple showed off in New York last month.

With over 100,000 Duo apps and counting, keep an eye on MacStories and AppStories. We’ll continue to highlight the best of the best as we work through our backlog of pitches.

A Closer Look at Apple’s Upcoming Creator Studio Update

For the past week, I’ve been testing betas of the following updates coming to Apple Creator Studio later this month:

Final Cut Pro is being updated this month, too, but I haven’t tested it.

Logic Pro's new Liquid Glass design.

Logic Pro’s new Liquid Glass design.

I use Logic Pro to edit MacStories’ podcasts, and so far, I’ve edited four episodes using the updated app. For the limited set of Logic’s features that I use for podcast editing, the experience hasn’t changed much. The loops I’ve created over the years are now in the new Unified Browser on the left side of the window, instead of in the panel on the right, but every other aspect of editing an episode is the same as before.

But even though my editing workflow hasn’t changed much, the design of the app has. Logic Pro has been given a Liquid Glass makeover that emphasizes black and dark gray over its predecessor’s lighter gray treatment. I’m glad Apple decided to update Logic Pro’s look because it had really begun to look dated. The black palette is far more in line with modern professional apps, which makes it nicer to work in.

Some of the new features available to musicians in Logic Pro. Source: Apple.

Some of the new features available to musicians in Logic Pro. Source: Apple.

Now, just because little has changed for podcast editors doesn’t mean there aren’t substantial changes to Logic Pro for musicians. The app has an enhanced stem splitter that can separate drum elements from each other, giving producers greater control. There’s also a new Play Surface that exposes instruments, drum pads, and chord pads for playing music live. The Play Surface and other Logic Pro and MainStage controls can be accessed using Sidecar on an iPad, too.

The new Unified Browser, which now houses those loops I’ve made, includes other enhancements, too. You can search using natural language queries, drag a loop into the search field to find similar loops, and quickly narrow results with dynamic tag filtering.

Logic Pro's Intelligent Assistant helps you learn the app.

Logic Pro’s Intelligent Assistant helps you learn the app.

One of my favorite features is Logic Pro’s new Intelligent Assistant, a new beta feature that combines models from Apple and OpenAI to help guide you through the music creation process.

Apps like Logic Pro are complex, which can make it hard to know what to do when you’re new to the app. One of the first things I did last year with Codex and Claude Code was create a skill that draws on Apple’s extensive support documentation for Logic Pro to help me navigate the app. Now that’s built right into the app itself, allowing you to pose natural language questions and get answers tailored to the app’s features.

The Intelligent Assistant in Pages.

The Intelligent Assistant in Pages.

You’ll find similar Intelligent Assistants in Pages, Numbers, and Keynote. Each is great for getting help with how to use the apps, but you can ask general knowledge questions, too. When I asked the Pages Intelligent Assistant what it could tell me about Logic Pro, it went out to the web and returned with a synopsis of the app and links to Apple product pages. That makes the Intelligent Assistant an interesting option when you need to do research alongside the document you’re creating.

The More button at the top of the Intelligent Assistant lets you report a concern about the results and track your usage. Beneath your usage statistics is a note that monthly usage is increased for a limited time. I don’t believe Apple has said how long that will be and to what degree usage rates will be reduced after the introductory period, but there is already a button to buy more usage for $49.99. How much that buys you is also unclear, though I’m sure we’ll learn more when the apps are released publicly. The Intelligent Assistant can also be turned off in each app’s settings if you prefer not to use it.

Some of the videos available now in Content Hub.

Some of the videos available now in Content Hub.

Pages, Numbers, and Keynote have added stock video content to their shared Content Hub for the first time. The collection of landscape and portrait videos that range from around eight to 34 seconds long is large and can be incorporated into your documents royalty free along with the Content Hub’s collections of photos, graphics, backgrounds, and more. Creator Studio’s collection of themes and templates for Pages, Numbers, and Keynote has been extended with these updates, too.

Testing Pages editing in Claude Code using MCP.

Testing Pages editing in Claude Code using MCP.

Another big addition is beta MCP support for Pages, Numbers, and Keynote. In my testing, I was impressed to find a very complete set of features accessible via MCP, which runs a local server and uses a shared JavaScript API that adds a layer that’s exclusive to each app. That means, for example, that you could set up a workflow with an agent to generate invoices in Pages, automatically populate fields in Numbers with the details, from Pages and your payment processor or bank, and then generate slides of the data to report on your revenue results periodically.

The biggest gotcha is that the app you’re controlling with an AI agent must be open and commands must identify the window you’re targeting if more than one is open. I also found the initial MCP setup a little opaque, but not insurmountable with the help of Claude, which walked me through the setup process and created a usage guide.

Keynote has also added an open file format called Keynote Interchange Format, also known as Keynote X, that’s based on a JSON-based bundle. It’s designed to be more accessible to AI agents, developers, and other tools for automation, integration with third-party tools, and other use cases.

Pixelmator Pro has been updated, too. Generate Fill is a new tool that lets you select an area and describe a change. For example, you can trace the outline of a person’s shirt and change its color, preserving the shadows and lighting.

Original.

Original.

Edited with Generative Fill.

Edited with Generative Fill.

In my tests, where I didn’t attempt to be super precise with my selection, it did a good, though imperfect, job changing a shirt from black to red. More impressive was how it preserved the original texture and look of the shirt. Plus, the results can be tweaked to refine some of the imperfections I saw in my testing.

One of the biggest changes coming to Creator Studio is one I haven’t had a chance to try yet. Final Cut Pro is adding MotionVFX, image generation, editing inside Pixelmator Pro, keyframing improvements, more transcription languages, the ability to disconnect timeline clips, and more. For more on those changes, which Apple has said will be released later this month, be sure to read my story from last week.

Overall, I’m impressed with the updates to Creator Studio. It’s good to see the collection of apps updated already with meaningful additions that fill many of the gaps that existed. I hope Apple keeps up the pace.

I’m also glad to see Apple embracing MCP and an open format for Keynote. As AI agent use has risen, I’ve felt the pull of web apps that have extensive APIs and their own MCP integrations, even though I prefer Apple’s Creator Studio apps. As someone who already uses Logic Pro and Pixelmator Pro extensively and Final Cut Pro to a lesser extent, those types of integrations will make it easier to stick with the native apps I like best.

Apple Frames and Frames CLI Updated with iPad mini (A20 Pro), Apple Watch Ultra 4 Support→

Following this week’s update to my Apple Frames shortcut and Frames CLI with support for the iPhone Duo and 18 Pro, I’ve prepared another small update for more device bezels Apple released a couple of days ago: the Apple Watch Ultra 4 and the new iPad mini (which John reviewed earlier this week on MacStories).

If you’re using the Frames CLI, you can update to version 1.7.0 from the GitHub repo or ask your agent to do it for you. If you’re a Shortcuts user, you can get the latest version of Apple Frames (4.2.0) below or from the MacStories Shortcuts Archive. In either case, you’ll have to download an updated asset pack with the new bezels, which will automatically replace the older one.

As a general recommendation for Shortcuts users, before trying to run the shortcut from the Home Screen or Siri, you’ll want to run it manually at least once so you can grant the initial set of permissions.

Apple Frames

Add device frames to screenshots for iPhones (8/SE, 12-13, 15, 16, and 17 generations in mini/standard/Plus/Pro/Pro Max sizes, plus iPhone 18 Pro and iPhone 18 Pro Max), iPhone Air, iPhone Duo (inner and outer displays), iPad Pro (11” and 12.9”/13”, 2018-2025 models), iPad Air, iPad mini (including A20 Pro), iPad, Apple Watch Series 7/10/11 (Series 12 screenshots use the Series 11 frames), Apple Watch Ultra 2, Ultra 3, and Ultra 4, iMac M4, Studio Display, Studio Display XDR, MacBook Air (2020/M5 13”/M5 15”), MacBook Pro (13”/M5 14”/M5 16”), MacBook Pro with PROLED XDR Display (14” and 16”), and MacBook Neo. Apple Frames 4 supports colored device frames for iPhones 16 and 17, iPhone 18 Pro, iPhone Air, iPhone Duo, iPad mini (A20 Pro), MacBook Pro M5, MacBook Pro with PROLED XDR Display, MacBook Air M5, MacBook Neo, iMac M4, Studio Display, Studio Display XDR, Apple Watch Ultra, and Apple Watch Series 11.

Get the shortcut here.

Permalink

Our Favorite iPhone Duo Apps: Weird and Wonderful, Vol. 1

The iPhone Duo has ignited a resurgence of weird and wonderful apps that take advantage of its unique hardware and sensors. Some are frivolous and others are truly useful, but they’re all fun. These apps can be used on a regular iPhone, too, but the delight comes from using them on the Duo, which is what I’ll focus on here.

DuoBox

DuoBox is an accordion app for the iPhone Duo. With the Duo open, keys for the chords are arrayed on the left side and piano keys on the right, just like an accordion. Pick a key and start playing, folding the Duo as you press the keys. It’s tricky to get used to, but with a little effort, you’ll get the hang of it and be playing, with the music responding to the folding of the phone the way an accordion does when played.

The app is surprisingly deep. You can record what you play, pick a song to play and follow the app’s key-press prompts, add a metronome, adjust the room sound, delay, distortion, pitch, and other sound settings. There’s even MIDI Out to connect to an app like Logic Pro. On top of everything else, the design of DuoBox is excellent, adding to the fun and immersiveness of the experience.

DuoBox is free to download from the App Store, but some features require an In-App Purchase.

PocketMicrowave

You know that there are some things that should never be microwaved, right? Well, PocketMicrowave breaks the rules by letting you zap all kinds of things that have no business being anywhere near your microwave. Want to microwave a watermelon and see it explode? Go for it. The list of items available to microwave includes ordinary food items like pizza and hot dogs, along with no-nos like fireworks and electronics. This is not high-concept stuff, and it’s not meant to be. It’s just good, silly fun. And while the 3D graphics are rudimentary and could use a tune-up, for $0.99, PocketMicrowave is still an enjoyable little diversion that made me smile.

Ajar

Simulations are a big part of the Duo curiosities I’ve tried. Ajar is a soundscape app that’s also a window to other worlds. The app opens to a view of a neighborhood of rowhouses. It’s raining, and periodically, a car drives by. With a tap, you can close the window, semi-muting the sound of the rain against the window. Fold the Duo into a laptop orientation, and a desk appears on the lower half of the Duo with a notebook, pencil, and cup of coffee. The scene changes perspective when you move the Duo around, too.

Each of the app’s eight scenes follows a similar pattern but in a different setting. There’s a Japanese garden, a snowy country scene, a train ride, a beachside scene, and more. I love the peaceful landscapes and accompanying soundscapes, but I’d also like more control to do things like drag the windows closed in smaller increments instead of just tapping it to fully open or closed. I’d also like to interact with the items on the desk in front of me. But those are just small refinements. Ajar is an interesting, peaceful app that is free to download on the App Store and has a one-time $4.99 In-App Purchase to unlock all of its windows.

Duopaper

Duopaper is a wallpaper app that takes advantage of the iPhone Duo’s outer and inner screens. When your Duo is closed, the outer screen displays just the center portion of the wallpaper. Unfolding the Duo reveals the full wallpaper, which zooms out onto a bigger panoramic scene. The app is packed with excellent illustrations, often with a folded paper style that fits the app’s design well and adds a little whimsy every time you unfold your Duo.

Duopaper is available on the App Store as a one-time purchase of $3.99.

Fold & Squish

The name of this app says it all. Fold & Squish helps you recapture those days as a little kid in art class when you squirted paint onto a sheet of paper, and then folded it in half to make a symmetrical image. Instead of paper, Fold & Squish asks you to fold your Duo, rub the outer screen, and then reopen it to see your creation. You can create a painting with a blank sheet of paper or follow the app’s guides to create a butterfly, owl, heart, and more. The app has options to delete, save, and share your paintings too. It’s a simple and fun way to create colorful artwork.

Fold & Squish is available on the App Store as a one-time purchase for $1.99.

Popfold

Popfold is a clever photo app that simulates the effect of a popup book. After you pick a photo, the app pulls out the subject and gives it a sticker-like border. Fold your Duo to the laptop-like pose, and Popfold adds a shadow effect and blurred background to make it look like the subject is popping out of the screen. The way the effect is done makes it very hard to capture in a screenshot, but trust me, it looks cool and is an app that I’m sure would delight kids.

Popfold is available as a free download on the App Store.

Testing Audio Intelligence with Siri Recap in watchOS 27.3→

Victoria Song, writing at The Verge, tested the new Audio Intelligence features for Apple Watch that also rolled out yesterday as part of the iOS and watchOS 27.3 developer betas.

My first Siri Recap was for an editorial planning meeting for a story I’m working on. It was decent at identifying different topics discussed, and I was impressed that it stripped out some sensitive information. Any outsider looking at these notes would be terribly confused. Ironically, it meant that this summary wasn’t the most useful for my job.

What I will say is that I’ve been testing the Amazon Bee Frontier for the past few weeks. In the little time I’ve spent with the watchOS 27.3 beta, the differences between Audio Intelligence and the Bee are very clear. The latter has more of what I’d call an AI-forward presence. Bee, for example, offers insights about the emotional tone of a conversation. I’ve also found that the AI can be a bit judgy about insights and takeaways. Live Rewind is just a straight transcript. Apple’s said publicly that Siri Recap is meant to be highly factual notes to jog a person’s memory. That rings true based on my first one. And, as far as I can tell, there is no commerce angle for Siri Recaps, whereas Bee has an Amazon integration that surfaces buying suggestions based on conversations.

I forgot to link this, but a few weeks ago, Apple published a technical report (PDF link) that goes into more detail on how audio is recorded, transferred, and processed for Siri Recap:

The Secure Exclave on Apple Watch encrypts the audio and transmits it to the Secure Exclave on the paired iPhone. Once the transfer completes, audio is immediately deleted from the Secure Exclave on Apple Watch, and only the Secure Exclave of the paired iPhone can decrypt it.

And:

The encrypted audio is decrypted inside the Secure Exclave of the paired iPhone. On-device speech recognition transcribes the audio to text, and then an on-device language model generates a condensed version that is less than half the length of the original transcript. This is designed to distill the conversation to its core meaning, removing content suggestive of conversation tone, non-essential language, filler words, and redundancies, while preserving the topics discussed. Once processed, the raw audio is immediately and permanently deleted and no longer exists on either Secure Exclave. Only the condensed text remains. Finally, a safety model screens the text to omit potentially harmful terms.

The implementation of Siri Recap based on dedicated hardware and on-device processing sounds like quintessential Apple, and it seems like the feature does work as advertised. Alas, while I have access to Siri AI thanks to my American Apple Account and U.S. region setting, I live in Italy, and Siri AI does not speak Italian yet.

I do not recommend testing Siri Recap if your language isn’t officially supported by Siri AI – unless you’re looking for some laughs:

…sure.

…sure.

Permalink

“Introducing Adaptive Dark Mode”→

As part of the 27.3 betas released yesterday, Apple also introduced a new Safari feature:

We’re excited to announce Adaptive Dark Mode in Safari 27.3 beta. When enabled by a user, Safari adapts light websites and presents them as dark.

In the spring of 2019, Safari 12.1 became the first browser to support dark mode on the web. A lot of people love using dark mode, but not every website has a dark version. Almost 40% of web page loads are from users in dark mode, and yet only 10% of page loads support it. If you love dark mode, you’ve probably hit a site that felt particularly bright. Adaptive Dark Mode is designed to close that gap, providing a consistent dark experience across the web.

If this sounds familiar, it’s because so many third-party Safari extensions for this very task already exist, including the very popular (and a MacStories favorite) Noir. The difference, of course, is that Adaptive Dark Mode is built deep into the WebKit engine itself now, ships for free in Safari, and offers greater speed and performance since it’s not limited by an extension. It sounds like Apple thought of a lot of details here:

At the core of Adaptive Dark Mode is the transformation of color. Sometimes colors are simply decorative, while other times they have semantic meaning. Adaptive Dark Mode determines when to invert colors so elements remain visible or when to leave colors unchanged.

While creating Adaptive Dark Mode, we developed a color matrix to invert the luminosity of colors while preserving hue. It’s used to make light backgrounds dark, and dark text light. When transforming text color, Adaptive Dark Mode aims for sufficient contrast as defined by the WCAG 4.5:1 contrast ratio. Light CSS gradients are also transformed to dark.

And:

It takes the same care with images. Most are left unchanged, preserving their integrity. But many icons and logos can lose contrast when the background color behind them is inverted. So Adaptive Dark Mode inspects SVG images and transforms colors when needed. For bitmap images, it analyzes the pixel histograms and inverts simple black-and-white logos.

Noir and similar extensions had a great run. Between this, the MCP server added earlier this year, and vertical tabs in 27.3, it’s time for me to go back to Safari everywhere.

Permalink