Amidst the variety of announcements from WWDC 2014, Extensibility – a new set of technologies for developers to extend their apps – has been mainly regarded as Apple’s solution to the lack of inter-app communication on iOS.
Traditionally, iOS has been a closed platform in terms of software personalization and extensibility: due to a combination of design choices and strict enforcement of sandboxing rules, iOS users never enjoyed many of the benefits found on Google’s mobile operating system. Android users could, for instance, install system-wide replacement keyboards or pick documents from any app advertised as a storage location; iOS users, on the other hand, were forced to deal with unnecessary copies created by an outdated Open In system or stick with Apple’s dubious keyboard design in iOS 7.
Simultaneously, with Apple focusing on Maps improvements and a new design foundation for iOS, a few third-party developers took up on the task of creating apps and protocols capable of extending iOS as much as possible leveraging the tiny holes left by Apple in its sandbox.
We’ve seen a proliferation of apps that use URL schemes to facilitate the process of launching other apps and passing text to them; bookmarklets – pieces of JavaScript code executed in the browser – to let Safari communicate with third-party apps; developers creating their own SDKs and app ecosystems to solve document management; Fleksy – a popular Android keyboard – making an iOS SDK; a Python interpreter and a text editor with a workflow automation system, developed by a one-man shop in Germany.
The third-party iOS development community has been incredibly creative in spite of Apple’s longstanding limitations on iOS, but many of the devised solutions – especially URL schemes – were, ultimately, hacks and workarounds based on a protocol that wasn’t intended to let multiple apps communicate and exchange data.
With iOS 8, Apple wants to make iOS more flexible and powerful by letting developers extend custom functionality and content beyond their apps, making it available to users in other parts of the OS – and all while maintaining a secure design model, user privacy, good performance, and battery life.
As someone who’s invested in iOS as a productivity platform and uses the iPad as a primary computing device every day, I welcomed Apple’s move with excitement and optimism, but I also wanted to investigate the actual scope of the technology the company will ship later this year.
Extensions
By App Extensions, Apple means the ability for users to extend and customize the functionality of the operating system. Extensions won’t be side-loaded apps, they won’t be available for download from the web, and, generally speaking, they have been designed as a way to make a specific app feature available outside of its containing app and inside a separate “host” app.
The important aspect to consider about Apple’s Extensibility initiative is that it will apply to both iOS 8 and OS X Yosemite and that there will be several types of App Extensions. Some extensions will be shared across iOS and OS X, while others will be exclusive to a specific platform.
During the WWDC keynote and in dedicated sessions, Apple detailed the types of extensions that developers will be able to implement in their apps:
- Today (iOS and OS X): widgets for the Today view of Notification Center
- Share (iOS and OS X): post content to web services or share content with others
- Actions (iOS and OS X): app extensions to view or manipulate inside another app
- Photo Editing (iOS): edit a photo or video in Apple’s Photos app with extensions from a third-party apps
- Finder Sync (OS X): remote file storage in the Finder with support for Finder content annotation
- Storage Provider (iOS): an interface between files inside an app and other apps on a user’s device
- Custom Keyboard (iOS): system-wide alternative keyboards
The possibility to enrich the functionality of apps and customize the operating system will come with new guidelines and rules developers will have to follow. For one, extensions will always come embedded inside apps users download from the App Store: an extension will be bundled with an app, and developers won’t be able to release standalone extensions on the App Store or submit apps that only contain extensions.
Apple stresses that they want to keep users in control, reinforcing the principle that extensions originate from apps, that they can be activated and disabled at any time, and that they are removed when a containing app is uninstalled. Also, developers won’t be able to make their extensions available in every part of the OS: for the initial release of iOS 8 and OS X Yosemite, Apple has chosen specific areas called “extension points” where extensions will be summonable by the user. For example, Today widgets will only be available in the Today view of Notification Center (as opposed to Android, which allows widgets on the Home screen) and storage providers will be accessible from the iOS Document Picker.
More importantly, inter-app communication – a blanket term used in the past in discussions about the silo nature of apps on iOS – is only a part of what Apple is trying to accomplish with extensions. Contrary to rumors surfaced before WWDC, Apple’s primary goal is to allow users to personalize their experience and select which functionalities of installed apps they want to access when they’re performing a specific task. While there is an inter-app communication aspect to this (and it’s most apparent on iOS, which lacks the unifying filesystem and windowing layers found on OS X), key to understanding the potential of extensions is that they’re all about removing friction.
As I suspected, Apple chose an extensible, plugin-like architecture over the often rumored capability of seeing multiple apps at the same time – at least for iOS 8.0. “Apps as features”, as I wrote earlier this week and in the past – but this is only part of the App Extensions story.
What really matters is that Apple is prioritizing user efficiency and delight by setting a few base guidelines and then letting users choose how they want to use their devices. Whether it’s the keyboard, fast access to a widget, or saving a webpage from Safari, Apple’s goal is twofold: to make users happier and more productive, and to bring much-needed new capabilities to third-party apps.
Apple’s Demos
In Apple’s demos at WWDC ‘14, we got a glimpse at the kind of integration the company is envisioning between system apps and extensions. While not final, the extensions demonstrated during the keynote and in developer sessions offer an indication of how the system will work and what developers can look forward to implementing in their apps.
Extensions in the Today view of Notification Center, called widgets, will provide a way for developers to present glanceable information that’s relevant to the user right now, with the possibility of adding interactivity to the widgets. Like other extensions, widgets will generate from apps and the user will be able to add and remove widgets at any time, as well as reorder them in the Today view.
During the keynote, Craig Federighi showed a SportsCenter widget to stay updated on results from your favorite teams and an eBay widget to check on the status of ongoing auctions (with inline image thumbnails) and options to submit a new bid without opening the full eBay app. According to Apple, developers can decide if a widget should appear only when new content is available: in the example above, eBay could decide to display the widget only when an auction is about to end; a todo app could show interactive checkboxes only for due dates or overdue tasks.
Apple is advising developers to keep widgets simple, focused on a specific task, and not too big; unlike jailbreak tweaks developed since the company introduced Notification Center in iOS 5, iOS 8 won’t allow developers to create memory-intensive widgets with text input and other complex features. These system limitations, however, don’t mean that developers won’t be able to come up with clever ideas to help users save time: in Philips’ case, Today widgets will support basic controls for Hue lights to be available right in Notification Center, only a swipe away from the Home screen. In iOS 7, Apple scratched the surface of widget potential with a Calendar shortcut and interactive checkboxes for Reminders; it’ll be up to third-party developers this year to understand how widgets can serve as sources of essential information, gateways to full app experiences, and shortcuts to specific app features.
Share extensions, triggered from the system share sheet, will allow users to post content to websites or upload media to sharing services without Apple having to manually add social integrations to iOS as they did with Twitter, Facebook, Flickr, Vimeo, and others over the past three years.
Apple must have realized that continuining to update iOS’ sharing features with few options every year was not only unfeasible (Apple can’t support every sharing website in existence) – it would be detrimental to customers as they were forced to jump through hoops (and duplicate files) to, say, push a photo from the Camera Roll to Instagram. A standardized protocol in iOS 8 will allow any web service with a native app to build share extensions, which we’ll likely see adopted by hundreds of social apps this Fall.
Share extensions will work well in combination with Photo Editing extensions, also demoed by Craig Federighi as another friction-removal tentpole feature of Extensibility. Thanks to iOS 8, developers of apps with photo editing functionalities will be able to integrate lightweight editing features into Apple’s Photos app. Primarily meant for targeted and quick adjustments, Federighi showed extensions for VSCO Cam and Waterlogue filters in Photos, available alongside effects provided by Apple.
Developers will be able to use their own custom UIs on top of Photos, although Apple suggests to build photo editing extensions that don’t require prolonged interaction. So while simple filters or stickers may be okay for quick editing with an extension, embedding Photoshop as an extension may be a little too much for Photos.
I suspect we’ll see virtually every popular filter app on the App Store add a Photo Editing extension, but I would also expect to see watermarking utilities, custom cropping tools, or annotation apps (such as Skitch or BugShot) add extensibility support. Photo editing extensions will support non-destructive edits, so you’ll always be able to revert to the original version of a photo even after it’s been modified by an extension.
The even bigger reveal from Federighi’s presentation were Actions, app extensions that will allow users to view or manipulate content inside another app. Federighi used two examples to show the “view” and “non-view” nature of extensions: a Bing translate action, which used a feature from the Bing app to translate Japanese text in Safari without showing any additional interface; and a Pinterest web clipper, which presented a custom-designed clipping tool to save an image from a Safari webpage into the Pinterest app without actually launching the app.
Considered the spiritual successors to Services for OS X, Actions will let apps work together and help users avoid switching between apps or rely on complex workflows based on URL schemes or bookmarklets.
More importantly, Actions will nurture the creation of rich workflows that aren’t possible with URL schemes. In a demo shared with developers at WWDC, Apple showed an image annotation tool capable of editing an image from a Safari webpage and insert it back into the webpage when done. It was reminiscent of Markup, a feature of OS X Yosemite that also happens to work as an Action extension. This kind of visual, interactive inter-app communication will result in user-friendly and powerful workflows that will be akin to “mini apps” available right inside a host app without having to switch contexts or be limited by bookmarklets. And they will be especially impressive on iOS, which never enjoyed the (simpler) possibilities offered by a Services menu.
Actions will lead to a completely new class of productivity workflows on iOS because of the integrated and secure extension model Apple has designed. Imagine utilities like an Evernote web clipper for Safari, an Instapaper extension, or something as trivial as printing a webpage to PDF: until today, the limitations of iOS forced developers to create standalone apps unable to talk to each other. With Actions, app features become visual extensions that share resources with their containing app (i.e. an Evernote clipper would authenticate with your account from the Evernote app) and that let you do more without sacrificing the simplicity of iOS. It’s going to be a major change for iOS productivity, and casual users will reap the benefits of this new system as well. It’s no surprise, then, that Apple is advising to reconsider hardcoded workflows that switch user context by opening other apps: while this will still be possible, Actions will likely cover the majority of basic inter-app communication needs with a better system.
In the same vein, Storage Provider and Finder Sync extensions should put an end to headaches and annoyances caused by managing files with cloud services on iOS and OS X, respectively – although Apple hasn’t shared as many details and examples about these extensions as they did for others. Storage Providers should offer a way for apps to work in unison with documents stored in a cloud-based file storage app through the native Document Picker, while Finder Sync will allow apps that sync a local folder with the web to modify the Finder interface with badge icons and custom menus. The latter seems primarily aimed at services like Dropbox, which had to reverse-engineer the Finder to put sync icons and logos on top of folders.
Custom Keyboards for iOS were a huge surprise at WWDC: Apple’s always taken a lot of pride in the software keyboard they designed for the original iPhone, and the tech press and developer community assumed Apple would never allow third-party keyboards to replace the Apple-made one on iPhones and iPads. That’s going to change on iOS 8 – with limits.
The primary concern with allowing a third-party developer to control and monitor typing with a software keyboard is that keystrokes (and thus a user’s private information) may be stored somewhere in the originating app and possibly transmitted to a cloud service. With iOS 8, Apple is taking a series of precautions to ensure that custom keyboard extensions will only get the bare minimum of keystroke information unless a user consents to sharing more.
Custom keyboards won’t be able to type into secure text input fields – the ones where characters are replaced by dots that are used to conceal passwords. When tapping those fields, iOS 8 will switch to the default Apple keyboard. Custom keyboards are installed by downloading apps from the App Store (just like any other extension), which means that companies like Fleksy and SwiftKey that already have dedicated iOS apps will be able to simply update them to include a system-wide keyboard extension. To get approved, custom keyboards will have to include a way (for instance, a button) to switch to another keyboard available to the user, similarly to how Apple features a globe key to move across multiple system keyboards.
As for user privacy, custom keyboards will run with network access set to off by default; in this way, unless a user consents to sharing keyboard data with a third-party network, custom keyboards won’t be able to transmit keystrokes to any service. Developers will have to rely on local processing for predictive suggestions and other typing enhancements, as the cloud won’t be able to aid them. There’s no doubt that Apple is making the right call with the default setting, and I hope that developers of custom keyboards will strive for utility and rich functionality without always requiring network access.
Apple is advertising custom keyboard extensions as a way to support more languages, different input systems (such as swipe gestures), and predictive suggestions based on proprietary engines. Swype, SwiftKey, and Fleksy have already announced that they’re working on custom keyboards for iOS 8, and Fleksy has also shown a screenshot of an early version of the extension. It’s still early to tell, but because custom keyboards will be based on apps, it’s possible that other third-party developers will devise ways to enter macros of text or enable special buttons that go beyond the feature set of popular solutions like Swype and Fleksy.
Developer Reactions, Limitations, and iOS Automation
Since Apple’s Extensibility announcements on Monday, I’ve been discussing the implications and potential of Apple’s new technology with several developers of iOS apps that came with basic inter-app communication features, user-defined workflows, and custom keyboards with special keys and gestures well before iOS 8.
One of my questions was about Apple’s approach and whether it would be simple enough for developers to implement before this Fall. “I think so”, Ole Zorn, developer of Pythonista and Editorial, told me. “Apple has been working on the necessary infrastructure (remote views, XPC) for a while, and it seems that a lot of the building blocks were already in place when iOS 6 was released, so I’m pretty confident that the system is quite robust now”, he added.
Hon Cheng is the developer of Dispatch, an email client for iPhone that comes with hardcoded support for third-party app actions through URL schemes, and he notes that “it’s powerful when an app does not need to be aware of other apps or extensions installed”. Flexibits’ Michael Simmons, one of the creators of Fantastical, called Apple’s approach “pretty straightforward to implement”, while Drafts developer Greg Pierce told me that “Apple has provided a well thought out foundation for integration with App Extensions”. Developers seem to agree that iOS extensions should be relatively easy to implement; as nicely summed up by Zorn, “it looks like a great starting point”. Pierce also added that Share extensions appear remarkably easy to support, and that he was able to get a Share extension “up and running (in Swift, no less) in about 15 minutes”.
Dispatch and Editorial implement inter-app communication features often based on URL schemes and third-party SDKs.
At this point, however, there are questions about the way Apple designed extensions to be activated. Without getting into deep technical details, Actions – the kind of extensions Dispatch and Editorial could likely support – will only be displayed as icons inside the system share sheet with no option for developers to trigger or filter them programmatically. Users will be in control of which extensions are visible in the share sheet, but, for instance, it won’t be possible for a web browser to have a 1Password button in the toolbar that launches a (possible) 1Password extension – you’ll have to open the default Share menu, find the 1Password icon, and tap it. This is related to Apple’s decision to use carefully chosen extension points – areas of the OS that extensions can hook onto – and the Share button will be one of them.
According to Hon Cheng, “for most apps, this will be great” but in Dispatch the lack of customization for extension points will be “problematic” as extensions can’t be integrated in the app’s existing custom action menu. Zorn also noted that Apple’s design “limits the degree of automation you can achieve” as a developer can’t decide to show only some extensions; however, he added, “it’s also a lot less confusing for users, because they’ll always know exactly when they’re using an extension vs. a built-in feature”.
While developers would welcome some flexibility in regard to filtering extensions or launching them from anywhere inside an app, Apple is making a good move in starting with specific activation areas, as it’ll allow for an easier and smoother rollout. “That is not necessarily a bad thing”, Pierce told me, adding that most users expect additional functions to be inside a Share menu and “it’s wise for Apple to enforce controls on how extensions are used – because bad user experiences would likely derail many users from enabling any of these extensions”.
It’s important to understand that Apple’s initiative to extend iOS has little to do with iOS automation, at least in the sense of connecting multiple apps together in a “chain” of workflows. Extensions for iOS – whether they’re widgets or actions – are meant to extend the functionality of an app and make it available in other areas of the OS, and they don’t appear to be suitable replacements for scripts and workflows that execute multiple commands from multiple apps in a row. Pierce told me that “extensions are not intended to allow broad integration between multiple apps” because they “run in a limited environment, with only access to specific data shared by the host app, and limited ways to share data with the app which provides the extension”.
That doesn’t mean, though, that power users won’t find clear advantages in the approach chosen by Apple. “The whole idea of combining multiple apps to perform a task will become much more accessible for people who don’t necessarily consider themselves power users”, Zorn said, adding that thanks to extensions “even apps that weren’t designed with power users in mind can participate in workflows that the developers perhaps didn’t even think of”. In practice, complex automated workflows will keep working as they’ve always been with iOS 8, but Apple is providing a dramatically better solution for more basic aspects of inter-app communication that weren’t possible before. Most developers won’t have to worry about manually adding URL schemes to their apps or maintaning a list of URL handlers registered by other developers. As Simmons told me in our conversation, “extensions will enable us to build richer and more open functionality, which will allow us to be more creative”.
Syncing of files with cloud services is another Extensibility feature that will allow developers to worry less about writing additional code from scratch and focus more on the experience they’re building in their apps. For apps like Zorn’s Editorial, which uses Dropbox to sync text files with other devices and platforms, developers often need to write their own UI and logic for sync, and that is difficult to get right. “With the new document picker”, Zorn noted, “services like Dropbox can work in all apps that adopt it, and those apps don’t even need to be aware of Dropbox because they can handle all these services through one interface”.
The situation is quite different for custom keyboards, which are different types of extensions than actions or share extensions. I spoke with Ioannis Verdelis, founder and COO of Fleksy, the popular system-wide replacement for the stock Android keyboard that’s also available on iOS with a dedicated app and SDK for other apps to integrate with. Fleksy started as an app for visually impaired users, who could type efficiently thanks to the keyboard’s support for gestures, voice feedback, and powerful text engine with auto correction. Fleksy has announced their intention of offering a custom keyboard extension for iOS 8, which was shown in an early build put together in less than 48 hours by the Fleksy team.
“We are so excited, it’s hard to describe”, Verdelis told me. “We’ve watched every WWDC with bated breath, hoping for this announcement. If you were anywhere near our office, you would have heard the cheers when we saw the news. Our neighbors asked us what happened”.
With iOS 8, Fleksy will be able to offer an alternative keyboard with a technology they’ve tweaked and improved over the past few years, giving iOS users a typing experience considerably different from Apple’s keyboard. According to Verdelis, many of the big players that are currently Android-first or Android-only will be ready for the launch of iOS 8 later this year, but, he added, “the exercise of launching on iOS will depend on how portable they’ve built their technology to be in the first place”.
I asked about Apple’s implementation of custom keyboards and the limitations they’ve put around the technology – namely the decision of turning off network access by default and the lack of APIs for dictation, caps lock, auto-capitalization, and more. “Apple did not “half open” the keyboard space on its platform”, Verdelis said. “The implementation we’ve seen on iOS 8 is robust, well thought through, and structured to enable a genuine keyboard market to emerge on the App Store”. Verdelis mentioned Apple’s consistent implementation of the keyboard APIs and settings (which, according to him, isn’t always the case on Android) and he added that “it seems like it’s been in the making for some time, judging by the documentation and structure in place”.
Fleksy offers cloud-based enhancements for its Android keyboard, which include personalization of the “language algorithm with the user’s unique writing style using their email and social networks”, a feature similar to Apple’s new QuickType keyboard for iOS 8. With network access being opt-in on iOS 8, Fleksy won’t be able to provide these features by default, but Verdelis stressed that all processing will be done offline, on the device, with additional functionality requiring user’s consent as per Apple’s API. The company expects “many users to allow online access”, in which case Fleksy for iOS should offer the same enhanced experience of the Android version.
Zorn, who modified the default iOS keyboard with extra rows and buttons in his apps, says he is “thrilled” that Apple is enabling custom keyboards on iOS, but also “a bit disappointed by some of the restrictions”. For example, Apple’s won’t allow keyboards to have any kind of selection controls, so tweaks such as the popular Hooper Selection concept won’t be allowed on the App Store.
Overall, custom keyboards for iOS reflect Apple’s intention of letting users customize their mobile experience, which should create a new opportunity for developers on the App Store. Verdelis agrees: “On other platforms, keyboards are some of the most popular apps people download, and we are excited for the use cases that are being created here for iOS”.
After talking to developers for the past five days, their initial reactions include excitement for new possibilities offered by Extensibility and curiosity to start playing with Apple’s new SDK. As Hon Cheng put it, “This is great for productivity. There will be less interruption”, a sentiment echoed by Simmons: “The ability to have apps communicate and share information will hopefully open up new ways to be productive”.
Developers and iOS users I’ve spoken with seem to agree that a new class of apps will be possible thanks to extensions, and new interaction patterns will have to be considered. “When an app like 1Password can have a Safari extension to fill in passwords on the current website directly, I suspect this will become the primary way to access it”, Zorn said, adding that he thinks “the same applies to bookmarking apps, file upload/download tools, weather widgets, and much more”. And there’s also confidence in knowing that productivity tasks and communication between apps are now aspects that Apple is prioritizing on its mobile platform. “I think we’ll see more apps that do one thing well”, Zorn concluded.
New Possibilities
Today, it’s difficult to quantify the impact that extensions will have on the iOS app ecosystem, but I think it’s safe to say that, considering developers’ reactions to Apple’s announcement, we’re going to see plenty of cool new stuff this Fall.
Personally, I will probably be able to delete many of the workflows that I created to speed up text editing and inter-app communication. Actions will provide better ways to work with apps visually and interactively. But, again, communication with apps is only one side of the Extensibility story: how will consumers respond to widgets and keyboards? Will actions end up like Services on OS X – loved by power users but mostly forgotten by everyone else? Will big companies like Tumblr, Evernote, Facebook, or Dropbox adopt extensions by iOS 8’s launch, setting an example for smaller developers to follow? Will Apple with its iWork and iLife suites? Time will tell, but I’m optimistic.
There are plenty of potentially useful scenarios in which extensions will make using our devices faster and more enjoyable. Apple doesn’t need automated workflows or chains of apps: they needed to cover the basics of inter-app communication, and they’re doing that exactly as I hoped – going the extra mile with other extensions I wasn’t expecting to see this year.
With less annoyances, faster access to information, and better communication between the apps we use every day, iOS is growing up. Apple put the ball in the developers’ court, and now they have to figure out how they can improve the ways we use our devices every day with more flexible, powerful apps. As Verdelis told me, “extensions seem to open a world of possibilities”.
