Today, Apple announced on its Machine Learning Research website that iOS and iPadOS 16.2 and macOS 13.1 will gain optimizations to its Core ML framework for Stable Diffusion, the model that powers a wide variety of tools that allow users to do things like generate an image from text prompts and more. The post explains the advantages of running Stable Diffusion locally on Apple silicon devices:

One of the key questions for Stable Diffusion in any app is where the model is running. There are a number of reasons why on-device deployment of Stable Diffusion in an app is preferable to a server-based approach. First, the privacy of the end user is protected because any data the user provided as input to the model stays on the user’s device. Second, after initial download, users don’t require an internet connection to use the model. Finally, locally deploying this model enables developers to reduce or eliminate their server-related costs.

The optimizations to the Core ML framework are designed to simplify the process of incorporating Stable Diffusion into developers’ apps:

Optimizing Core ML for Stable Diffusion and simplifying model conversion makes it easier for developers to incorporate this technology in their apps in a privacy-preserving and economically feasible way, while getting the best performance on Apple Silicon.

The development of Stable Diffusion’s has been rapid since it became publicly available in August. I expect the optimizations to Core ML will only accelerate that trend in the Apple community and have the added benefit to Apple of enticing more developers to try Core ML.

If you want to take a look at the Core ML optimizations, they’re available on GitHub here and include “a Python package for converting Stable Diffusion models from PyTorch to Core ML using diffusers and coremltools, as well as a Swift package to deploy the models.”