Scriptable: Automating iOS with JavaScript

Simon Støvring’s app was one of my favorite app debuts of 2018 and, in terms of my daily iPad workflow, it’s turning out to be a pleasant acquired taste. I had no idea I would come to rely on Scriptable for key file management and writing tasks when I first tried the app last year, but Støvring’s creation is proving to be even more versatile than Shortcuts for specific functionalities missing from Apple’s app.

Let’s back up for a second here so I can contextualize what Scriptable is before explaining how I’m using it. Scriptable is a JavaScript IDE for iOS that lets you write and execute JavaScript code on an iPhone or iPad with an interactive environment. You get access to the standard JavaScript library, there is a console to log the output of your scripts, and you have an editor that supports syntax highlighting and code auto-completion.

What makes Scriptable unique is that in addition to “regular” JavaScript, it also comes with JavaScript bridges for native iOS APIs that allow you to write scripts that automate iOS-specific features. You can, for instance, access the contents of the system clipboard, view and create calendar events or reminders, open links in Safari, send rich notifications, and even display custom UIKit interface elements such as alerts, Quick Look previews, and table views.

The list of iOS functionalities that Støvring bridged to JavaScript for Scriptable is massive, and it’s reminiscent of another app that did it years ago for another programming language: Ole Zorn’s Pythonista and its iOS-Python bridges. Scriptable is, in many ways, a modern version of that original vision, supercharged by the addition of new technologies such as Siri shortcuts, Files integration, and customizable notifications.

Showing the Files picker via JavaScript in Scriptable.

What sets Scriptable apart from past efforts of blending a traditional programming environment with iOS automation, in fact, is the modern nature of the app combined with Støvring’s prioritization of new iOS APIs that are not yet supported by Pythonista or Apple’s Shortcuts app. While Pythonista hasn’t been updated in several months and Shortcuts is still limited to its own iCloud Drive folder and minimal integration with the Siri interface, Scriptable ships with features such as file bookmarks, custom UIKit-powered SiriKit snippets, file tagging, and custom arguments for Siri shortcuts. It would be unfair to categorize Scriptable merely as “Pythonista, but for JavaScript” because Scriptable, in less than a year on the App Store, has eclipsed Pythonista’s integration with iOS frameworks and is showing ideas that Apple itself should copy in their Shortcuts app.

It’s beyond the scope of this story to write a review of Scriptable or even attempt producing a manual for learning JavaScript via the app. There are plenty of excellent guides on the Internet for that, and Scriptable itself ships with a solid documentation tool for learning more about its iOS integrations. I can’t possibly cover them all in an exhaustive manner. What I can do, however, is highlight four of the areas where Scriptable has improved my work on the iPad, which also happen to be features that would fit Shortcuts well. Let’s take a look.

File Bookmarks

Unlike Shortcuts, Scriptable can view and modify tags assigned to documents in the Files app, and it can access documents located outside of its own iCloud Drive container thanks to file bookmarks. These two features are the main reasons I’ve found a place for Scriptable in my iPad workflow and they’re behind the scripts I use most on a daily basis.

File bookmarks can be created in Scriptable’s settings by selecting a file or folder from any location of the Files app and giving it a unique name. If the bookmark is a file, Scriptable will gain read/write access to that item alone; in the case of a folder, Scriptable will be able to list the entire contents of the directory and also have read/write access to it.

To create a bookmark, you have to pick a file or folder from Scriptable’s settings.

A list of file bookmarks created in Scriptable.

To do this, Scriptable uses a FileManager API that acts as a bridge between JavaScript and various Files-related technologies of iOS including file coordination and presentation, iCloud Drive, and the document picker. At a basic level, the FileManager API can be used to access Scriptable’s own files in iCloud Drive; however, when combined with the Files picker or file bookmarks, the API can be used to view and edit tags or, even better, allow Scriptable to access documents from any other iOS app.

Listing the contents of an iA Writer folder bookmarked in Scriptable.

I started my experiments with Scriptable and the FileManager API last year when I was working on my iOS 12 review. At the time, Scriptable didn’t support file bookmarks and was limited to accessing files and folders located in iCloud Drive/Scriptable. I wanted to speed up the process of finding out which screenshots of which chapters of the review I’d already taken, so I wrote a script that presented me with a native interface to pick Scriptable sub-folders, list screenshots contained inside them, and preview a selected image with Quick Look.

Here’s the code:

var fm = FileManager.iCloud();
let baseDir = fm.documentsDirectory();
// Present a list of the following folders, which have to be located under iCloud Drive/Scriptable. You can change these folders to whatever you like.

var folders = ['Setup', 'iPad', 'Well-Being', 'Communications', 'Shortcuts', 'Apps', 'Everything Else',]
var folderLength = folders.length;

var ask = new Alert
ask.title = 'Review Folders'
ask.message = 'Choose one of the folders below.'

for (var i = 0; i < folderLength; i++) {
    ask.addAction(folders[i])
}
 ask.addCancelAction('Close')

 var show = ask.presentAlert()
 var actionIndex = await show
 var selectedAction = folders[actionIndex]

var path = fm.joinPath(baseDir, '/Review/' + selectedAction + '/')

var list = fm.listContents(path)

var listLength = list.length;

var files = new Alert
files.title = 'Found Files'
files.message = 'Here are your files.'

for (var i = 0; i < listLength; i++) {
    files.addAction(list[i])
}
 files.addCancelAction('Close')

var showFiles = files.presentAlert()

var fileIndex = await showFiles
var selectedFile = list[fileIndex]

var filePath = fm.joinPath(path, selectedFile)

var img = fm.readImage(filePath)

QuickLook.present(img)

And here’s how the process looked in practice:

That was a very basic script¹⁶, but it got the job done as it helped me find and preview hundreds of screenshots more quickly than using the Files app.

Earlier this year, I took this concept a step further with file bookmarks by directly integrating iA Writer with Scriptable. iA Writer uses iCloud Drive to store documents, which means you can browse the contents of the iA Writer library from the Files app and create as many sub-folders as you want.

iA Writer’s folder structure in iCloud Drive.

Any of iA Writer’s folders (including the main one) can be picked from Scriptable and treated as a bookmark to let Scriptable read all the files contained inside it. So here’s what I did: I created a bookmark for the iCloud Drive/iA Writer/Image Assets/ sub-folder, which is where I store image files for screenshots I want to use in my stories, but which I haven’t framed or uploaded to our CDN yet. The benefit of this approach is that, thanks to iA Writer’s support for content blocks, I can reference these local files in the text editor and have them appear as full-sized images in the HTML preview of a document:

A local file reference in iA Writer…

…becomes a local image attachment when Markdown is previewed as HTML.

Now, while I love this method of writing in Markdown and referencing assets stored in iCloud Drive by path – which has proven essential in the making of this story – I often forget about which images I’ve already saved in iA Writer’s sub-folder and whether I’ve already tagged them or not. So instead of switching back and forth between the text editor and the sub-folder in iA Writer, and instead of using the Files app for this, I created my own presentation UI in Scriptable that shows me a visual preview of all the files contained in iA Writer’s Image Assets sub-folder.

The same folder being viewed in iA Writer (left) and in Scriptable via my custom script as a bookmark.

Thanks to file bookmarks, Scriptable can list the contents of the Image Assets folder; the folder belongs to another app, but because I picked it in Scriptable once and created a bookmark for it, now I get programmatic access to it via JavaScript. After gaining access to the folder and figuring out how to list files, I programmed a basic table view in Scriptable that would show files sorted alphabetically along with their file names and tags. Then, I programmed the table view so that it would allow me to select multiple files and preview them in full-screen using Quick Look. Lastly, I used Scriptable’s clipboard integration to copy the path of each chosen file so that I can paste it in iA Writer and generate a content block.

This process is more difficult to describe than it actually is in practice. Here’s what it looks like when I’m writing in iA Writer and want to insert a reference to a local image:

And here’s the code:

var fm = FileManager.iCloud()
var dir = fm.documentsDirectory()
var assets = fm.bookmarkedPath('Image Assets')

// Variables to store image data and paths
var images = []
var fileRefs = []

//List the contents of the folder bookmark
var list = fm.listContents(assets)

//Sort file names alphabetically
list.sort(function (a, b) {
  return a.toLowerCase().localeCompare(b.toLowerCase());
})

var table = new UITable()
table.dismissOnSelect = false
table.showSeparators = true

for (var i = 0; i < list.length; i++) {
  let path = fm.joinPath(assets, list[i])
  if (list[i] == 'Uploaded Assets') {
    //Do nothing for the Uploaded Assets sub-folder
  }
  else {
    //Download file from iCloud, then read the image 
    await fm.downloadFileFromiCloud(path)
    let row = new UITableRow()
    let image = fm.readImage(path)
    let title = fm.fileName(list[i], false)
    let imageCell = row.addImage(image)
    //Read file tags
    let tags = fm.allTags(path)
    let tagNames = tags.join(', ')
    let titleCell = row.addText(title, tagNames)
    imageCell.widthWeight = 20
    titleCell.widthWeight = 80
    row.height = 60
    row.cellSpacing = 10
    // Make sure a row can be selected
    row.dismissOnSelect = false
    table.addRow(row)
    row.onSelect = (number) => {
      var path = fm.joinPath(assets, list[number])
      images.push(path)
      // Add a selection checkmark to the row and reload the table
      var selCell = UITableCell.text('✔️')
      selCell.widthWeight = 10
      row.addCell(selCell)
      table.reload()
    }
    }
}
await table.present()

// Preview picked files with Quick Look and assemble local path for iA Writer
for (var i = 0; i < images.length; i++) {
  await QuickLook.present(images[i])
  fileRefs.push('/Image Assets/' + fm.fileName(images[i], true))
}

// Join local paths and copy them
var imageLinks = fileRefs.join('\n\n')
Pasteboard.copyString(imageLinks)

A lot of interesting system integrations are going on in this script that I’d like to call out. First, Scriptable can request read/write access to a bookmark, which happens to be an external folder, with four lines of code. Notice how line 22 and line 45 can construct the complete filesystem path to an iCloud Drive file (you can actually log it to the console and inspect it), which would be impossible in Shortcuts or Pythonista. The script can construct a native table view with custom sizes for cells and even reload the table after I make a selection, which is indicated by a checkmark emoji. Also, notice how lines 34-35 can read all the tags assigned to a file.

At the end, the script loads native Quick Look previews for each selected file and, using the FileManager API, reads the file name and extension of each item before copying it to the clipboard. It’s remarkable how much of iOS’ underlying frameworks Støvring was able to abstract in JavaScript, allowing an inexperienced JS tinkerer such as myself to put together a useful script that deals with native iOS APIs.

I used this script dozens of times when writing this story to view screenshots and reference them locally in iA Writer, but, obviously, it’s only one part of the automated workflow I put together for images. Using the same APIs mentioned above, I wrote another script that lets me save a new screenshot (previously copied to the clipboard) to iA Writer’s Image Assets folder, tag it, and copy its path to the clipboard. Here’s what this looks like when I’m writing:

And here’s the script behind it:

var fm = FileManager.iCloud();
var baseDir = fm.documentsDirectory();

// Get image from the clipboard
var clipboard = Pasteboard.pasteImage()

if (!!clipboard) {
  var ask = new Alert
  ask.title = 'Save File'
  ask.message = 'Rename your file.'
  ask.addCancelAction('Close')
  ask.addAction('OK')
  var fileName = ask.addTextField()
  // Present alert to rename file
  var show = await ask.presentAlert()
  var screenTitle = ask.textFieldValue(0) + '.jpeg'
  var path = fm.bookmarkedPath("Image Assets") + '/' + screenTitle
  // Write image data to the bookmarked folder
  await fm.writeImage(path,clipboard)
  if (fm.fileExists(path) == true) {
    await fm.addTag(path, 'Assets')
    let textRef = '/Image Assets/' + screenTitle
    // Copy local file path to the clipboard
    Pasteboard.copyString(textRef)
    var confirm = new Alert
    confirm.title = 'File Saved'
    confirm.message = 'The file ' + fm.fileName(path, false) + ' has been saved in Files and tagged with Assets.\n\n' + textRef + '\n\nThe file path has been copied to the clipboard.'
    confirm.addCancelAction('Close')
    confirm.presentAlert()
  }
 }
else {
  // No image found in the clipboard
  throw Error('You did not copy an image!')
}

Even more than the previous script, this one is the perfect example of the advantages of Scriptable compared to Shortcuts. If I were to use Shortcuts for this task, I would be limited to the Shortcuts sub-folder, which means I wouldn’t be able to reference files by their local path in iA Writer, and I couldn’t add tags to those images either. Without any kind of automation, I’d have to import screenshots in Files with drag and drop from Photos, rename them, tag them, and move them into iA Writer’s sub-folder. With Scriptable in Slide Over, everything can be done in a few seconds: images are saved into iA Writer’s container thanks to file bookmarks, a default tag is applied, I can type a filename in a native alert, and the local path is put in the clipboard, ready for pasting in the text editor.

The only downside of file bookmarks is that, due to API limitations imposed by Apple, they can only be used inside the Scriptable app itself. Because bookmarks can only be accessed by the process they were created in, extensions such as Siri and Scriptable’s action extension wouldn’t be able to resolve the bookmark’s path in the filesystem. According to Støvring, this is probably a restriction put in place by Apple for security reasons, but I think Apple should revise its stance here and allow extensions from the same app that has created a file bookmark to use it as well. As it stands now, it is not possible to visualize the contents of bookmarks in a Siri snippet, or to save files from the share sheet into a bookmarked folder using the Scriptable extension.

Despite these limitations, file bookmarks are an amazing technology around which I built my iA Writer/Working Copy writing and editing workflow, and which I learned to automate with JavaScript in Scriptable. I feel like I’ve barely scratched the surface of what’s possible with file bookmarks: for example, I’m already experimenting with ways to speed up how I file expenses and invoices in iCloud Drive by putting Scriptable in the middle of the process and letting it assign tags to PDFs and file them into the appropriate folders. The ability to work with any file and any folder is an obvious fit for automation, and Scriptable is leading the way here.

File Tagging

Scriptable’s other notable advantage in the Files department compared to Shortcuts is its full support for native tags, which can be read, modified, and removed entirely via JavaScript.

Let’s start with an easy example. Consider a file that has been assigned three different tags in the Files app. If you were to remove those tags manually from Files, you’d have to find the file, long-press it to reveal the contextual menu, open the Tags panel and remove each one by hand. There is no way to remove all tags from a file at once; if you have a lot of tags in your iCloud Drive account, you’ll have to scroll the entire list in the Files popup.

Now let’s take a look at how clearing all tags from a file works in Scriptable:

And here’s the code:

var utis = ["public.item"]
var fm = FileManager.iCloud()
var dir = fm.documentsDirectory()

// Bring up the Files picker
var files = await DocumentPicker.open(utis)
var processed = []

for (index in files) {
  // Download file, read all tags, and remove each one
    await fm.downloadFileFromiCloud(files[index])
    let allTags = fm.allTags(files[index])
    for (tag in allTags) {
      fm.removeTag(files[index], allTags[tag])
      }
    let fileName = fm.fileName(files[index], true)
    processed.push(fileName)
}
// Push file names to a list and present an alert
var items = processed.join('\n')

var dialog = new Alert()

dialog.title = 'Tags Cleared'

dialog.message = 'Tags have been cleared from the following files:' + '\n\n' + items

dialog.presentAlert()

Scriptable brings up a native Files picker, which can be used to select multiple files at once. Using a repeat loop, Scriptable will download each file from iCloud (so it can read its metadata in full), read all the tags assigned to it, and remove them one by one in a secondary repeat block. After all tags have been cleared, the app will display an alert with all the names of files that were processed and untagged by the script. If you find yourself constantly removing tags from entire groups of files, using Scriptable for the job is going to save you several minutes in aggregate over the course of a few weeks.

What about viewing which tags have already been assigned to one or multiple files? Again, that’s extremely easy to build in Scriptable with just a few lines of code. In my case, I created another script that constructs a table view with rows for each file selected from a Files picker; each row displays the file name, extension, and tag(s) assigned to a file. I find this script ideal for checking on the tags added to all the files contained in a folder – I can hit ‘Select All’ from the Files picker and Scriptable will quickly display details for all selected files in its table view.

// Optional: restrict file picker to images only
var utis = ["public.image"]
var fm = FileManager.iCloud()

var files = await DocumentPicker.open(utis)

var table = new UITable()
var header = new UITableRow()
var title = 'Selected Files'
var headerText = header.addText(title)
header.isHeader = true
table.addRow(header)

for (index in files) {
    await fm.downloadFileFromiCloud(files[index])
    let currentTags = fm.allTags(files[index])
    if (currentTags.length == 0) {
      // Add a special untagged label if the file has no tags
      var tags = 'Untagged'
      }
     if (currentTags.length > 0) {
       var tags = currentTags.join(', ')
      }
    let fileName = fm.fileName(files[index], true)

    let row = new UITableRow()
    let titleCell = row.addText(fileName, tags)
    titleCell.widthWeight = 90
    table.addRow(row)
}

QuickLook.present(table)

Of course, in addition to clearing and viewing tags, Scriptable also supports adding new tags to a file. In another script I created, called ‘Tag Files’, I took advantage of Scriptable’s ability to tag files and save them in iCloud Drive with the app’s action extension and Files picker.

The Tag Files script can be used in two ways: you can either pass any file from other iOS apps via the share sheet to the Scriptable extension, or you can pick a file manually from Files. In both cases, a table view will come up, allowing you to pick one or multiple tags to add to the document; once the file has been tagged and saved, you’ll get a confirmation message at the end. I’m quite proud of the fact that the colored circles for tags in the table view were drawn entirely in code (they’re not image files) using Scriptable’s Rect and DrawContext APIs.

var files
var fm = FileManager.iCloud();
var baseDir = fm.documentsDirectory();

// Check if file has been shared from the extension, otherwise pick it manually
if (config.runsInActionExtension) {
  files = args.fileURLs
  var originalPath = files.toString()
  // When shared from the extension, isolate the file name and use a new path inside the Scriptable container
  var finalName = originalPath.substring(originalPath.lastIndexOf('/')+1)
  var path = fm.joinPath(baseDir, '/Scriptable Assets/📥 Inbox/' + finalName)

}
else if (config.runsInApp) {
  var pick = await DocumentPicker.open()
  // If picked from the Files picker, save in the same location
  var originalPath = pick.toString()
  // Save two separate paths, even if they are the same, for starting location and destination
  var path = pick.toString()
  var finalName = originalPath.substring(originalPath.lastIndexOf('/')+1)
}

// Custom list of tags
var tags = ['Assets', 'Not Uploaded', 'Done']
var itemTags = []

var table = new UITable()
table.dismissOnSelect = false

var title = 'Pick a Tag'
let headerCell = UITableCell.text(title)
headerCell.centerAligned()
var header = new UITableRow()
header.addCell(headerCell)
header.isHeader = true
table.addRow(header)

table.showSeparators = true

// Create table view with custom color labels for each tag
for (var i = 0; i < tags.length; i++) {
  let shape = new Rect(0, 50, 100, 100)
  let draw = new DrawContext()
  draw.respectScreenScale = true
  if (tags[i] == 'Assets') {
    let shade = new Color('ffd700')
    draw.setFillColor(shade)
    }
  else if (tags[i] == 'Done') {
    draw.setFillColor(Color.green())
    }
  else if (tags[i] == 'Not Uploaded') {
    draw.setFillColor(Color.orange())
    }
  draw.fillEllipse(shape)
  let image = draw.getImage()
  let cell = UITableCell.image(image)
  cell.widthWeight = 10
  cell.centerAligned()
  let row = new UITableRow()
  row.addCell(cell)
  let tagCell = UITableCell.text(tags[i])
  tagCell.widthWeight = 80
  tagCell.leftAligned()
  row.addCell(tagCell)
  row.dismissOnSelect = false
  table.addRow(row)

  row.onSelect = (number) => {
    // Use -1 to account for the presence of a header row
    if (itemTags.includes(tags[number-1])) {
      itemTags.splice(itemTags.indexOf(tags[number]-1), 1)
      table.reload()
    }
    else {
      itemTags.push(tags[number-1])
      var selCell = UITableCell.text('✔️')
      selCell.widthWeight = 10
      row.addCell(selCell)
      table.reload()
    }
}

}

await table.present()

if (itemTags.length == 0) {
  var noTag = new Alert()
  noTag.title = 'No Tags Selected'
  noTag.message = 'You have not selected any tags.'
  noTag.addCancelAction('Close')
  noTag.presentAlert()
}
else {

// Save newly tagged files to the Inbox folder of Scriptable. This allows to use the extension, which cannot access file bookmarks for other external folders

var content = Data.fromFile(originalPath)
await fm.write(path, content)
// Add all the tags previously picked
for (var i = 0; i < itemTags.length; i++) {
  var saved = await fm.fileExists(path)
  if (saved) {
    fm.addTag(path, itemTags[i])
  }
}
var confirm = new Alert()

confirm.title = 'File Saved'
var itemName = fm.fileName(path)
confirm.message = 'The file ' + itemName + ' has been saved and tagged.'
confirm.addAction('OK')
confirm.presentAlert()
}

As you can see from the video, the script needs to save the input file in two distinct locations depending on how it was passed to the script. If a file was picked manually from Files, a tagged version will be saved back to the original location; it doesn’t matter from which folder the file was picked because iOS 12 pokes a hole in the Files sandbox the moment you pick a file, granting Scriptable temporary permission to write content anywhere. If a file was shared with the extension, however, the file gets saved into a sub-folder of Scriptable’s own container located under iCloud Drive/Scriptable/Inbox, which I added to my favorites in Files for easy access.

This decision highlights the limitations of file bookmarks, which cannot be accessed by the action extension process in iOS 12. Ideally, files shared from other apps to the Scriptable extension should be saved in the ‘Temp Files’ folder that I keep in the root of my iCloud Drive, and which I use for all kinds of temporary documents that need to go somewhere else. The only way to access this folder when I’m, say, sharing a PDF from Spark to the Scriptable extension would be a file bookmark, but bookmarks cannot be resolved from extensions due to sandboxing restrictions. For this reason, I had to account for the different ways a file can be picked from or passed to Scriptable. Given how bookmarks have to be created manually by the user, and given how the user needs to explicitly run a script in the extension, I don’t understand why this limitation exists.

I’ve mostly been using this script to save documents from other iPad apps and tag them upon saving them, which is not possible with Apple’s own Files extension. This has saved me a lot of time and allows me to keep a consistent tag-based organization in Files because everything gets tagged as soon as it’s saved to the Files app.

Arguments for Siri Shortcuts

Absent a native API for users to pass some input text when triggering a Siri shortcut via voice, Støvring came up with an ingenious workaround that lets Scriptable generate different Siri shortcut inputs for the same script using arguments.

Here’s how this works. You can assign a Siri activation phrase to a Scriptable script and run it as a Siri shortcut; if you use the QuickLook and Speech APIs, you’ll even be able to present data and custom voice responses in Siri. By default, you associate a personalized Siri phrase to a script like in other apps by tapping an ‘Add to Siri’ button and recording a phrase to trigger the script.

A script that shows custom voice and Quick Look responses in Siri.

A few months ago, Støvring added support for arguments to scripts triggered via Siri shortcuts. You can now record multiple phrases for the same script and each phrase will pass a slightly different piece of data to the script, which will read it as an argument. These small bits of data are stored as key:value options where key is the label and value is the input text you want to pass to the script when it’s running. In a script’s settings, you can create multiple arguments and record a unique Siri phrase for each; all the phrases associated with a script will be presented under a Siri Shortcuts screen in the app.

A list of multiple Siri shortcuts with arguments created for the same script.

Creating a new argument for a script to run as a Siri shortcut.

In your script, you can then access the contents of a specific key by name through a siriShortcutArguments property; if the key exists and holds value, it’ll return a text string to use as an input argument for the script. These arguments can then be used to dynamically alter the behavior of the same script as soon as a Siri shortcut is invoked. Rather than creating multiple duplicated and slightly-different versions of the same script (each paired with a different Siri phrase), you can just keep one script and associate multiple Siri shortcuts with it.

My perfect use case for this feature is a script that shows me a custom list of reminders due this week. By default, when it’s run without input, the script brings up a nice table view that aggregates all due reminders from all my Reminders lists along with due dates and emoji for overdue tasks. Obviously, all of this is fetched natively and securely using Scriptable’s built-in Reminders bridge.

By default, the script shows me all tasks due this week.

However, I also created multiple arguments to load reminders from specific lists, and I paired each with a different Siri phrase. This way, invoking the “podcasts this week” Siri shortcut will only display tasks from the ‘Podcasts’ list in Reminders, while “Club this week” will filter tasks belonging to the Club MacStories list instead. You get the idea. By combining a key named list with the value containing the name of a Reminders list, I was able to create five different Siri shortcuts for just one script in Scriptable, each presenting a different set of tasks in the Siri UI.

Tasks from a specific Reminders list, filtered by a Siri shortcut argument.

Thanks to this feature, I now have a fast way of viewing specific subsets of tasks in Siri, presented with an interface I designed with a few lines of code.

It’s not hard to imagine how Apple could implement a similar approach to Støvring’s Siri arguments in their Shortcuts app. Instead of just supporting one Siri phrase per shortcut, users could have the ability to associate a Siri phrase with a variable that holds data; when a shortcut runs, the value of the variable would change depending on which Siri phrase was used, thus altering the behavior of the shortcut without forcing users to duplicate the same shortcut multiple times if they want to have slightly different outputs in Siri. In a perfect scenario, the “argument” could even be interpreted directly by Siri (it’d extract a word from the Siri phrase itself) without creating fixed arguments beforehand, but I’d settle for a feature similar to Scriptable’s in the meantime. The idea is too good to pass up.

Custom UIs in Siri Responses

Scriptable’s integration with Siri doesn’t stop at presenting interfaces, generating custom voice responses, and supporting multiple phrases – features that already put Støvring’s app above Shortcuts’ own integration with the Siri UI. In addition to those features, Scriptable can also show arbitrary HTML, images, PDFs, and even video in the Siri interface. And it takes fewer than 10 lines of code to do it.

Let’s say that you have an image or document you need to refer to often for any personal reasons. Thanks to Scriptable’s QuickLook API, you can write a script that fetches the file from Scriptable’s iCloud Drive container and presents it inside Siri as a visual snippet. Any item that can be previewed using Quick Look on iOS – whether it’s a JPEG image or a PDF document – can be displayed inline within Siri by triggering a Scriptable script like the one below with a Siri shortcut:

var fm = FileManager.iCloud()
var dir = fm.documentsDirectory()
var path = fm.joinPath(dir, "/Image Assets/Codes.jpeg")
// Check if the file exists and has to be downloaded from iCloud before displaying it
if (fm.fileExists(path)) {
  await fm.downloadFileFromiCloud(path)
  img = fm.readImage(path)
  QuickLook.present(img)
}
if (config.runsWithSiri) {
  Speech.speak('Here is the codes image.');
}

I have a couple of image files that I need to view for quick reference on an almost daily basis, and I can’t even begin to explain how much time this automated technique has saved me.

But that’s not all. Among the various iOS frameworks bridged in Scriptable via JavaScript, there is also a WebView API that can be used to load HTML content or video files inside Siri. This web view uses the same technology that third-party apps typically implement to load webpages or other web content, only in this case it can be used in Scriptable and Siri to load and display files fetched from the filesystem. Whether you have some HTML content that you want to display as a single-page snippet or, for whatever reason, a video file that you want to load in Siri, it doesn’t matter: as long as you provide a file path and size your web view correctly, Siri will be able to display Scriptable’s content just fine via its Siri shortcuts.

Playing a video inside Siri thanks to a Scriptable shortcut.

I honestly don’t know why anyone would want to play a video inside Siri (as you might expect, the audio track will be muted by default), but it’s the underlying principle that is important here: even if use cases might be limited, a solid automation tool should be flexible enough to allow for all kinds of ideas to be possible – even the ones you can’t immediately justify – because there might always be someone with different needs than you who would like to control their computer in a unique, previously unforeseen way.

In my case, that meant writing a script that can use a web view and custom Siri responses to tell me and show me what the latest issue of our MacStories Weekly newsletter is. Using a handful of web requests to Mailchimp and Dropbox, the script gets the issue number from our latest newsletter campaign, turns the Mailchimp email to HTML, and displays everything as HTML. The web link to the newsletter is even placed in the clipboard when the script runs as a Siri shortcut.

Is this a common use case for automation on iOS? Absolutely not. It’s a custom script that I built specifically for me. But that’s exactly my point: automation frameworks should be designed for maximum versatility across all kinds of users, not for average use cases guessed by its creators. And from this standpoint, as I’ve demonstrated above, Scriptable is proving to be more flexible, and even better integrated with native iOS features, than Apple’s own Shortcuts app. I hope the company takes notice.

The state of iOS automation is strong and its future is bright.

With the Shortcuts app, Apple is democratizing visual automation by building a modern bicycle for the mind – a creation tool unlike any other that can help users craft solutions to problems in a few easy steps. There has never been anything as intuitive or powerful as Shortcuts on any other mobile OS. Without Shortcuts, my work on the iPad would be considerably slower, more limited, and maybe even less fun than it is today. Shortcuts is the glue that holds all my favorite apps and workflows together, and I strongly believe that the Shortcuts team at Apple is still just getting started. What we’ve seen so far is only the first phase in the transition from Workflow to Shortcuts.

For the next steps, I’d like to see Apple pay close attention to what Simon Støvring has created with Scriptable. But let me be clear: with my coverage of Scriptable in this story, I do not mean to suggest or imply that it’s only possible to get work done on the iPad if you’re a programmer and know JavaScript. That’s not the point I’m making, nor should it be your takeaway from this chapter.

As I mentioned above, automation should be primarily considered an addition to your favorite computing environment – an extra tool to speed things up. By covering Scriptable and its advanced capabilities, I hope I’m able to shine a light on the kinds of iOS integrations that third-party apps can already use, and which Apple should consider for Shortcuts’ visual automation environment as well.

From file bookmarks and tagging to advanced Siri shortcuts with support for input arguments and rich previews, there’s a bevy of iOS APIs launched in iOS 11/12 that Shortcuts is still not using, and which could be integrated with visual building blocks that completely abstract the programming underneath. Given the outstanding job Apple did converting Workflow to Shortcuts last year, I absolutely can’t wait to see where they take Shortcuts next.