Last month I wrote an article for MacStories on the extensibility and automation changes in OS X Yosemite. The second half was a basic overview of JavaScript for Automation (JXA) (the new addition to OS X scripting languages) joining AppleScript. Before writing that section of the article, I wanted to learn the basics of JXA in order to be sure that I understood what I was writing about and wasn’t just blindly summarizing the contents of the JXA release notes and WWDC Session Video.

Since JXA is so new, there obviously was not much information to go by. I’ve never gotten around to learning AppleScript, so articles based on the classic OS X automation language were not much help either, although I’m not sure how much they would apply anyway. The result was quite a few wasted hours trying to figure out some of the most basic parts of JXA, such as proper syntax of method calls, which method calls worked with which apps, and how to identify UI Elements in order to trigger them with UI automation. Eventually I was able to figure these things out, so now I’m back to share what I learned.

What Not to Expect from this Article

This article is not a comprehensive tutorial on the JavaScript programming language. If you read any further, I am assuming you already have at least a basic knowledge of JavaScript in its web implementation, and simply want to transition from scripting websites to scripting Mac apps.

If you do know JavaScript, this article is not meant to teach you every aspect of JXA either. I am not going to get into the JXA Objective-C bridge, as I do not know Objective-C, so that most powerful section of JXA is beyond my scope. Rather, the purpose of this writing is simply to give you the tools to get started with JXA and enable you to do some basic automating of applications using supported method calls or UI automation.

Getting Starting with JXA

The first thing to know is that JavaScript for Automation scripts are meant to be written in Apple’s Script Editor app. Script Editor comes preinstalled on every Mac: you can find it in the Utilities folder in your Applications folder.

Script Editor is not the best code editor. I’m disappointed by its half-baked syntax highlighting and complete lack of code completion, but the convenience of being able to run JXA scripts at any time within Script Editor by clicking the play button, see the output in the bottom pane, and have easy access to the scripting libraries of all the apps on your machine outweigh the app’s drawbacks. You can write your JXA in whatever editor you want, but I’ll be writing about doing so in Script Editor.

When you open Script Editor you will see two main sections, a large one on top and a smaller one beneath it. The big section is where you will write your code, the small section will display data about your script while it is being run. This small section defaults to the Description view (which switches to the Result view when you run your script), but I have found the Events view to be far more useful, as it outputs every event that happens while your script is running rather than just the end results. You can switch to the Events view by clicking the third button at the very bottom of Script Editor (the box with lines in it) and then selecting the Events tab.

Above the code section are four buttons. You only need to worry about the last three for most basic scripts. The button with the square in it will stop your script while it is still being executed (most useful for killing infinite loops), the triangle button will build and run your script, and the hammer button will just build your script (use this to make sure you don’t have any syntax errors without actually running the code if you do not, or to get Script Editor’s non-live syntax highlighting to catch up with any new code since your last build or run). Below the four buttons is a thin row with a dropdown menu that will likely be set to “AppleScript”. Change this to “JavaScript”. (You can tell Script Editor to default to JavaScript on new script files by going to Preferences and changing the default to “JavaScript (1.0)”). With Script Editor set up, all that’s left is to decide what apps you want to automate and then write your scripts.

There are two main ways to automate applications on OS X: using method calls built into apps by their developers, or UI automation. Of these, automating apps with their built in methods is significantly easier and less finicky.

All you need to know is what methods a particular app supports, and this information is easily accessible in the Script Dictionary. The Script Dictionary is available via the menus of Script Editor either by going to Window > Library, or Finder > Open Dictionary. The latter shows you every available app that can be scripted, but then disappears when you open a particular app’s dictionary. The Library option will show you a shorter list of scriptable apps to open dictionaries for, but will remain open for you to easily access multiple different dictionaries while writing a script. You can add apps to the Library by clicking the plus button on its small window. Once you have an app in mind that you wish to automate, open its dictionary and you will get a list of all available methods and properties that the app includes and supports scripting of. For automation via method calls, these dictionaries are your friends.

Example Scripts

The main reason I found it so difficult to get started with JXA myself was because there was a severe lack of sample code for me to compare to what I was reading in the release notes and script dictionaries. As such, for this article I built two scripts which we will follow along with to make my descriptions of JXA code more easily understandable. The JXA release notes (linked at the beginning of this article) are a great tool for learning JXA, and you should definitely refer to them while you’re getting started as well, but hopefully my scripts will give you some useful strategies and example code to refer to alongside the release notes to give you a more complete picture. For these scripts, I chose to automate iTunes, as it is an app that everyone will have installed which also supports a fairly extensive (although not all encompassing) scripting library.

As we all know, iTunes 12 has a terrible user interface. As such, I wrote these scripts in an effort to reduce the number of times I have to deal with it on a daily basis.

The most common thing I do with iTunes, not surprisingly, is play music from it. Personally, I keep a single main playlist of my 100 favorite songs (I update it often, moving old songs out and new favorites in, to keep things interesting) which I shuffle through. The vast majority of times that I want to play music in iTunes, I just want to hit play on this playlist. Unfortunately, if the iTunes interface is not already open to this playlist, I have to click a series of buttons to navigate back to the correct section of the app, open the playlist, and choose the shuffle button. I wanted a script that would do this for me.

The first script we’ll look at here does just that. Given the title of a playlist that exists in your iTunes, the script will use iTunes’ built in JXA methods to automatically play that playlist. The script does not specify shuffling, so it will play with whatever setting you had it set up for last. In other words, if you aren’t a shuffler like I am, you can still make use of this script too, if it interests you.

When the script is run, the first thing it does is identify the current player state of iTunes, which will either be “stopped”, “playing”, “paused”, “fast forwarding”, or “rewinding”. I can’t imagine why I would run the script when iTunes is in a state of fast forwarding or rewinding, so I do not account for these options. Rather, if iTunes’ state is playing, the script will throw up an alert which displays the title and artist of the current song along with the time remaining, and gives options to pause the song, restart the playlist (if you have shuffle turned on, this will reshuffle the entire playlist), or cancel and do nothing. If the state is paused, I get the same alert except the pause option is swapped for a play button and the message is tweaked to tell me the song is paused. If the state is stopped, the alert message tells me nothing is playing and gives only two options: play my playlist or cancel.

Now, let’s jump in.

Automation via Method Calls

The first JXA method you need to know is the Application() method. This method will return an application by name from your Applications folder, and once you have that application, you can call its own built-in methods. For us, we’ll start with:

iTunes = Application('iTunes')

With this line, we can now call any method available in the iTunes script dictionary on our new iTunes object.

The first thing our script needs to do is get the player state of iTunes so that it knows which alert to put up. To access this, open the iTunes script dictionary in Script Editor (Window > Library > Double click iTunes in the list). The top section of the dictionary window is divided into three panes, the first of which will have a list of “suites”. Each suite contains a variety of properties and methods accessible from our iTunes object. The Standard Suite will be present in almost every app, and generally contains the same set of properties and a few methods which are not particularly useful (print settings for iTunes?). We want the iTunes Suite. Select that and the second pane will fill with objects and methods available specifically for the iTunes application. Each list item has a small icon next to it. A “C” in a blue circle indicates a method you can call, a “C” in a purple square indicates an object. If you select an object, the third pane will fill with elements and properties of that object (yellow “E” icons are for elements and purple “P” icons are for properties).

Beneath the three small panes at the top is a much larger section which displays the details for each object or method that you select. You can scroll through all of these to find what you’re looking for, but it’s much easier to identify it from the lists at the top, where clicking any item will jump you directly to the detail view in the bottom section. One of the most useful objects is the Application object, which you can find a version of for every application dictionary in the app-specific suite. If we take a look at the Application object in our iTunes Suite, we’ll see this:

Within the list of properties for the Application object, we see the playerState property, which is what we are looking for.

The detail view displays this line for the playerState property: “playerState (“stopped”/‌”playing”/‌”paused”/‌”fast forwarding”/‌”rewinding”, r/o) : is iTunes stopped, paused, or playing?“ The items in quotes between the parentheses display every possible value that the playerState property can return, and the “r/o” at the end of these tells us that playerState is a read only property: we can access what the player state of iTunes is at any time, but we cannot change it by explicitly setting this property to a different value (rather, we would change it by calling a separate method supported by iTunes, such as play() or pause()). After the parentheses, beyond the colon, is a simple description of what the property is. This will be the layout of any property that you look at for any app, so you can always use the dictionary to determine what possible value will be returned when you access an object property.

Returning to our script, we’ll save the player state to a variable.

state = iTunes.playerState()

Next, we need a few if/else statements to check what the player state is and then display the corresponding alert dialogue. As I discussed earlier, we don’t really need to worry about the state returning “fast forwarding” or “rewinding”, so we just need three statements:

if (state == 'playing') {
    // Playing alert dialogue
} else if (state == 'paused') {
    // Paused alert dialogue
} else {
    // Stopped alert dialogue
}

Now we need the code to display an alert. Here’s where things get just a bit more complicated. iTunes cannot display system alert dialogues by default (neither can any app). Instead, these are wrapped up in a package of add ons known as “StandardAdditions”. StandardAdditions is a special library of methods that can interact with Mac OS X at a system level. StandardAdditions contains many useful methods such as converting ASCII characters, getting/setting the clipboard, reading and writing to files, and a multitude of mediums for user interaction. The last of these is where we can find the method for the alert dialogue we are looking for.

While the StandardAdditions dictionary is available amongst all the other scripting dictionaries when we look at the Library, it is not an app itself. Instead of accessing these methods in the normal way, with Application('StandardAdditions'), we instead enable these methods for an already existing application object to utilize by setting the includeStandardAdditions property to true (every app has this property and it always defaults to false). For our script, you probably assume (as I did when I first tried this) that we should send includeStandardAdditions = true to iTunes, because that’s the app we are automating. In practice, this will cause an annoying side effect.

The main goal of this script is to avoid ever needing to click around in the iTunes interface when we want to play music. It would be even better if we didn’t even need to look at the interface. What helps us achieve this is that JXA scripts can be set up as Automator workflows, just like AppleScripts have always been able to be. This means that we can bind our script to a keyboard shortcut and run it anytime from within any app (if you have an app like Keyboard Maestro, you can do this without Automator, but Automator workflows can be bound to keyboard shortcuts in System Preferences without necessitating a third party app). What we don’t want to have to deal with, because it would nearly defeat purpose of our script, is having the iTunes window thrust to the front of our screens every time we hit the keyboard shortcut. With that in mind, also note that any time an app displays an alert dialogue on OS X, the app in question starts bouncing around in your dock until you click on it, have its window moved to the front of your screen, and choose an option on the dialogue. We want to avoid that in our script, so we definitely do not want to bind the alert directly to iTunes, or to any specific app.

Thankfully, the JXA Application() method has a special way of handling this. Instead of passing an application to the method by name, we can call Application.currentApplication(), which will return an application object for whichever application is currently frontmost at the time the method is called. We can use this in our script to make sure that the alert dialogue is always tied to the app we currently have open when we want to start playing our music, so the script will never cause bouncing icons or necessitate switching apps.

So we want to create an application object out of the currently open app, then set the includeStardardAdditions property to true so that we can activate an alert in the app. Here’s the code:

currentApp = Application.currentApplication()
currentApp.includeStandardAdditions = true

Now our script has an application object for iTunes and one for our current frontmost app. The current app has StandardAdditions activated, and the player state of iTunes has been saved to a variable. We know the framework we’re going to use for the if/else statements to trigger different alerts depending on the player state, so now we just need the alerts themselves. First, take a look at the syntax for them in the StandardAdditions scripting dictionary; we’re looking for the displayAlert method (found in the User Interaction Suite).

This method looks so much more complicated than the last one we looked at because it includes parameters. The JXA syntax for a method with parameters like these will have the first parameter (the one not in hard brackets) right after the opening parenthesis, followed by a comma and an opening curly bracket. Then, on individual lines, the name of each parameter followed by a colon, a value for the parameter, and a comma. When you’ve set the parameters that you want, finish with the closing curly bracket and the closing parenthesis. You do not need to provide every parameter that the method can use, only the ones that you need. For our script, we want an alert with a title, a message, and three buttons (the maximum allowed). It’s also useful, although not necessary, to set a default button and a cancel button (the default button sets the button to be triggered if you press enter, and the cancel button sets the button to be triggered if you hit escape). Before we create our alert, we need to decide what message we want it to display.

I’ve been playing around with this script for a few weeks, tweaking the message to what I’ve found the most useful. The result is a message that tells me which song is playing, which artist it is by, and the time remaining in the song in mm:ss format. This requires a little bit of setup before it goes in the alert message. First, we need to get the title and artist of the currently playing song. There are methods to get this information from our iTunes object:

track = iTunes.currentTrack.name()
artist = iTunes.currentTrack.artist()

Now we have the song’s name stored in our track variable and the artist stored in our artist variable. Getting the remaining duration in mm:ss format is a bit more complicated. iTunes only provides the the player position of a song (time elapsed rather than time remaining), and it only gives this value in seconds. Thus, we need several lines of code to convert that value to a remaining duration in mm:ss format. Basically, we ask iTunes for the player position in the current song and the total duration of the track, then subtract to get the remaining seconds and do some simple math to convert that value in mm:ss format. Since we’re going to use this in both the play and pause alert dialogues, I wrapped it all up in a function so we can easily call it multiple times:

function calculatePlayerPosition() {
    playerPosition = iTunes.playerPosition()
    duration = iTunes.currentTrack.duration()
    secRemainder = (duration - playerPosition)
    minRemainder = Math.floor(secRemainder/60)
    secRemainder = Math.round(Math.abs(secRemainder) % 60)
    remainder = minRemainder + ":" + (secRemainder < 10 ? "0" + secRemainder : secRemainder)    
    return remainder
}

Now we have all the pieces to build our alert dialogues. The one for when the playerState returns playing will look like this:

action = currentApp.displayAlert('iTunes Playback Options', {
    buttons: ['Playlist','Pause','Cancel'],
    message: 'There is currently ' + calculatePlayerPosition() + ' left in \'' + track + '\' by ' + artist,
    defaultButton: 1,
    cancelButton: 3
})

When that alert gets called, it will look something like this:

The code for the paused alert will be almost identical, and the stopped alert will have a simpler message (there’s no song playing, so we don’t need to include name, track, or remaining time) and only two buttons (no play/pause because there isn’t a song to call these on).

Almost done now. The final piece to add to the script is some code to handle the response to the alert. Alerts return an AlertReply object, which has a buttonReturned property. buttonReturned is a string object consisting of the text of the button that was selected from the alert dialogue. We saved the response to a variable named action, so we just need to check the value stored there and perform an action based on it. We can do this in just a few lines:

playlist = 'Good Songs'
if (action.buttonReturned != 'Cancel') {
    if (action.buttonReturned == 'Playlist') {
        iTunes.playlists.play()
    } else {
        iTunes.playpause()
    }
}

The first thing this does is check whether you hit “Cancel” on the alert. If you did, nothing happens. If you did not, it checks whether you chose the “Playlist” option. If so, it runs the play() function on the variable playlist. (I defined playlist right above this section in that code fragment, but in the actual file I have playlist = 'Good Songs' on the first line of the entire script, so that I can easily change the playlist that gets played by the script if I want to. For you, since the playlist you are going to want to play is most likely not titled “Good Songs”, this is the only line you will need to change to make the script work for you). If you did not choose “Playlist” or “Cancel”, that means you either chose “Play” or “Pause”. Since the “Play” option would only have appeared if iTunes was currently paused, and vice versa for the “Pause” option, we can take care of both cases with the playpause() command, which just toggles the play state from playing to paused or from paused to playing.

That’s it! All together, the final script looks like this:

playlist = 'Good Songs'

currentApp = Application.currentApplication()
currentApp.includeStandardAdditions = true

iTunes = Application('iTunes')
state = iTunes.playerState()

if (state == 'playing') {
    track = iTunes.currentTrack.name()
    artist = iTunes.currentTrack.artist()   
    action = currentApp.displayAlert('iTunes Playback Options', {
        buttons: ['Playlist','Pause','Cancel'],
        message: 'There is currently ' + calculatePlayerPosition() + ' left in \'' + track + '\' by ' + artist,
        defaultButton: 1,
        cancelButton: 3
    })
} else if (state == 'paused') {
    track = iTunes.currentTrack.name()
    artist = iTunes.currentTrack.artist()
    action = currentApp.displayAlert('Play music in iTunes?', {
        buttons: ['Playlist','Play','Cancel'],
        message: 'Currently \'' + track + '\' by ' + artist + ' is paused with ' + calculatePlayerPosition() + ' remaining',
        defaultButton: 1,
        cancelButton: 3
    })
} else {
    action = currentApp.displayAlert('Play music in iTunes?', {
        buttons: ['Playlist','Cancel'],
        message: 'Currently nothing is playing.',
        defaultButton: 1,
        cancelButton: 2
    })      
}

if (action.buttonReturned != 'Cancel') {
    if (action.buttonReturned == 'Playlist') {
        iTunes.playlists.play() // Play the playlist defined on line 1.
    } else {
        iTunes.playpause()
    }
}

function calculatePlayerPosition() {
    playerPosition = iTunes.playerPosition()
    duration = iTunes.currentTrack.duration()
    secRemainder = (duration - playerPosition)
    minRemainder = Math.floor(secRemainder/60)
    secRemainder = Math.round(Math.abs(secRemainder) % 60)
    remainder = minRemainder + ":" + (secRemainder < 10 ? "0" + secRemainder : secRemainder)
    return remainder
}

Update: We’re having a temporary issue with our code previews, so the above script will throw an error at line 38 when you try to play a playlist. I’ve put the working script up on GitHub, so in the mean time you can download it there.

UI Automation

The second script we’re looking at covers UI automation. UI automation enables us to automate tasks that apps do not have built in support for via methods. It utilizes the Accessibility framework that nearly all Mac apps include to mimic the actions of actually clicking around in the app with a mouse. This kind of automation is not particularly difficult, but it can be time consuming because it involves a lot of guess-and-check style programming. That said, it’s worth it to be able to automate anything we want within any app.

The first step in any UI automation script is to create a System Events object. System Events, similar to the StandardAdditions that we used in the last script, is a script library of methods that are not tied to any particular app, but rather to the entire system. Instead of using Application objects like we did in the first script, for UI automation we will only be using a System Events object. This is because the Accessibility frameworks are manipulated by the system itself rather than individual applications. To the system, applications are considered “processes”, so if we want to target iTunes and manipulate its user interface, we need to create a System Events object that handles the iTunes process.

system = Application('System Events')
iTunesController = system.processes['iTunes']

Now that we have an object that controls iTunes’ process, we can start sending it events. To see what methods we can call on a process, and what objects exist to call them on, look in the System Events script dictionary, under the Processes Suite.

The main method that most UI automation scripts will use is the click() method. Not surprisingly, this method simulates what would happen if you clicked on something within an app. All that our script needs to do is identify the object in the app’s interface that we want to click on, which is a lot easier said than done.

Application interfaces are made up of a wide variety of elements. You can see all elements that an app can be made up of in the Processes Suite of the System Events dictionary. Specifically, look at the elements that make up the UIElement object. These elements exist in some sort of hierarchy that the Accessibility framework can see, but we cannot. In order to click on an object within an interface, we first need to drill down into that interface’s hierarchy until we identify the position of the object we’re looking for, then run the click() method on it. I like to think of this hierarchy like a file system: if you want to get a file located in a folder on your Desktop, and you’re currently at the top level of Finder, you first need to go to Users, then open your user folder, then open the Desktop folder, then open the folder on the Desktop, then double click the file. This is the exact same way that finding an object in a user interface works, except instead of navigating through folders we are navigating through uiElements. ¹

The best way to understand all of the pieces of a user interface that we might be looking at is by searching through the Processes Suite of the System Events dictionary. The top level object that we’re manipulating with UI automation is the Application object, so click on that object in the second pane of the dictionary window. Within the description view will be a section labeled “Elements”, and right under that is the word “contains”, followed by a comma separated list of elements that inherit from the Application object. In this list you’ll see “uiElements”, which you can click to show you the UIElement object. This is the object that holds every possible piece of a user interface that you may want to manipulate, or search through to find the object you want to manipulate. You can see every one of these objects under UIElement’s Elements list.

That list probably looks daunting right now, and it is. There’s really no way to know how your app might be laid out by looking at that list, because nearly all of those UIElement objects can inherit from each other, so they could be located in nearly any order in the “file system” of an app’s user interface. When you’re dealing with Finder on any Mac computer, you know that the Desktop folder will be located within your user folder. With app interfaces, we know no such thing. A Group object could be located within a ScrollArea object or a ScrollArea could be inside of a Group, etc. The only thing that can generally be assumed is that any type of Button object (this includes radioButtons, popUpButtons, and images) will be at the end of the hierarchy, because these are the objects that might receive a click event.² So how do you find the right path to follow through an app’s UI hierarchy? The best way I’ve found is by using Terminal.

With Terminal (also installed on every Mac, in your Utilities folder), we can start up an interactive JXA prompt to easily test from the command line without writing out a full script. To start this process, open a new Terminal window and type:

osascript -l JavaScript -i

You should then get a “»” symbol as your input prompt, which indicates that you can start entering JXA commands. First make sure to set up the app you want to start navigating through by entering the same lines that our UI Automation script has started with. Just type each line like you would in the normal script, except when you hit enter, that command will actually be run, then the Terminal will wait for the next one. Your Terminal window should look something like this:

Now you can start calling commands on the iTunes process to puzzle out its user interface. One important thing to note: UI Automation is only possible on apps that are open in the active desktop. If you, like me, keep multiple desktops open with different apps on each one, make sure that your iTunes window exists on the desktop you are trying to automate its UI in. In this, case, make sure your iTunes window is open in the background of the desktop you are running Terminal commands in.

To start figuring out the UI, it’s useful to know that if you send your iTunes object a UIElement in the form of a command (add opening and closing parentheses after it), Terminal will return every instance of that UIElement that currently exists in your iTunes window. Since you are most likely trying to automate something within the main window of iTunes, start by calling this command:

>> iTunes.windows()

Which will return this:

=> [Application("System Events").applicationProcesses.byName("iTunes").windows.byName("iTunes")]

That return tells us a lot. The first part, [Application("System Events").applicationProcesses.byName("iTunes"), just defines what our iTunes object is: an ApplicationProcess named iTunes that is an element of the System Events application. After that, the information is much more useful: .windows.byName("iTunes")] tells us that there is a single Window object, and its name is “iTunes”. Since there is only one window, and everything we want to automate is going to be within that window, we now have the second piece of our iTunes “file path”. Instead of just trying to find elements in iTunes, we will now look within iTunes.windows['iTunes'].

The next piece is not so easy. It’s obvious that we will be looking inside of the main iTunes window, but from there the UI could be organized in a nearly infinite number of ways. Thankfully, there is an incredibly useful command we can call to help us: the entireContents() command. This command will return every UIElement within an app’s user interface hierarchy. It basically reveals a map of the app’s current window, which we can then follow to isolate the elements that we want to “click” on.

If you put iTunes.windows['iTunes'].entireContents() into your Terminal window, the result is an intimidating wall of text. Every element in this giant list is separated by a comma and a space, and I’ve found the list significantly easier to parse by copying it into a text editor and doing a Find-and-Replace to switch out every instance of , (a comma followed by a space) with two newlines. ³ That will separate every element in the iTunes interface onto its own line, with blank lines above and below it. This makes it far easier to find what you’re looking for.

As you scan through the list, you’ll see increasingly large entries as you get deeper into the app’s hierarchy. Here’s a random entry from my list:

Application("System Events").applicationProcesses.byName("iTunes").windows.byName("iTunes").radioGroups.at(0).radioButtons.at(2)

This “file path” targets the RadioButton with an index of 2 within the RadioGroup with an index of 0 within the Window named “iTunes” within the iTunes ApplicationProcess of the System Events Application. That’s great, but it doesn’t really help in determining what that button does. Thankfully, there’s a command for that (multiple, actually).

Take a look back at the UIElement entry of the System Events script dictionary. Below the list of contained elements is a Properties list, and this includes every property that is attached to a UIElement. Whether or not all of these properties are defined is up to the app developer, and in the case of iTunes, if you ask for the name of most elements, the return will be null. Thankfully, they did define the description property for every UIElement that I have looked at in the app. So if we want to find out what the mysterious RadioButton at index 2 that we identified above is, we need to convert that entry from the list into dot notation, then just add .description() to the end of it. Converting to dot notation is fairly easy. First, Application("System Events").applicationProcesses.byName("iTunes") refers to the iTunes system process in the System Events application. We already created an object for that earlier, so that can be replaced by iTunesController. The rest is even easier: replace any instances of .byName() or .at() with hard brackets around whatever is between the parentheses. Thus, .windows.byName("iTunes") becomes .windows['iTunes'], .radioGroups.at(0) becomes radioGroups[0], etc. Here’s the result:

>> iTunesController.windows['iTunes'].radioGroups[0].radioButtons[2].description()
=> "TV Shows"

Now take a look back at the iTunes interface. Looking through the buttons that exist, we see that the third button in the row of buttons beneath the title bar is labeled “TV Shows”. That is the button that the path is targeting. So if we wanted to automate navigating to the TV Shows section of the iTunes interface, we could just call the click() command on that path, like so:

>> iTunesController.windows['iTunes'].radioGroups[0].radioButtons[2].click()

Run that in your Terminal window and you will see your iTunes window switch over to the TV Shows section. And there you have it, the basics of finding your way through an application’s user interface hierarchy.

An Example Script for UI Automation

So to put this information to use, let’s do a quick run through of the second script I prepared for this article – a UI automation script for iTunes. This script does something that iTunes, for some reason, does not seem capable of doing with its built-in methods: playing a station in iTunes Radio. Specifically, I want to automate the process of hitting play on my first custom iTunes Radio station.

To begin, we’ll set up an object for the iTunes system process, the same way that we did earlier:

system = Application('System Events')
iTunesController = system.processes['iTunes']

With UI automation, particularly in an app like iTunes with so many different views, you can’t assume which view you will be in at any given time. Thus, the first thing to do is make sure we are in the iTunes Radio view. However, if we are in, say, the TV Shows view at the moment, the iTunes Radio button actually does not exist. This is UI automation, so we cannot call commands on any buttons that are not explicitly visible in the UI. This means that before we can get to iTunes Radio, we need to make sure we are in the main “Music” tab. Earlier, we already identified the TV Shows button, and the Music button is clearly in the same group of buttons in the interface. Since we know the TV Shows button had an index of 2, it would follow that the first button in that group would have an index of 0, and indeed, we can navigate iTunes to the Music tab using that path and the click() command, like so:

iTunesController.windows[0].radioGroups[0].radioButtons[0].click()

Next, we need to to get to the iTunes Radio tab of the Music interface. Looking through the list of UIElements in the iTunes window, we’re lucky in that iTunes makes it easy for us. Instead of only giving a button index for the iTunes Radio button, the entry actually displays it by name:

Application("System Events").applicationProcesses.byName("iTunes").windows.byName("iTunes").radioGroups.at(1).radioButtons.byName("Radio")

So next in our script, we’ll convert to dot notation and then pass the click() command to that element, navigating iTunes to the iTunes Radio tab. The result is as follows:

    iTunesController.windows['iTunes'].radioGroups[1].radioButtons['Radio'].click()

The iTunes Radio interface is made up of a series of images, organized in rows. The top row is for featured stations, the bottom row is for custom stations. I want to play my first custom station, so I need the index of the first item in the second row of images.

If you were in a view in iTunes other than the iTunes Radio view when you created your entireContents() list, your list will not include the elements in the iTunes Radio view, as they did not exist in the interface at the time. So if this is the case, call that command in your Terminal again, this time with the right view open, and now your list will include the elements that we are looking for, we just have to find them.

We know that the iTunes Radio station buttons are actually images, which means that we are looking for Image elements rather than Button elements. There will be two groups of Image objects, one for the row of featured stations and one for the row of custom stations. I only have two custom iTunes Radio stations, so I also know that my custom station list will only have two elements, an Image at index 0 and an Image at index 1. The featured station list will have five images in it. ⁴

Looking through my list of UIElements, I do find two places where there are image elements, one is a list of five image elements and the other is a list of two. Calling the description command on the element at index 0 of the list of two images returns this:

>> iTunes.windows['iTunes'].scrollAreas[1].uiElements[0].images[0].description() 
=> "Good Songs"

The iTunes Radio station that I want to target is titled “Good Songs” (yes, the same as my playlist. I’m not very original, I just want to listen to good music), so I know that I’m in the right place. Now I just add that to the script and give it a click command:

iTunesController.windows['iTunes'].scrollAreas[1].uiElements[0].images[0].click()

When you run this through your Terminal, your first iTunes Radio station will expand in your iTunes window in the background, the same thing it does if you actually click on it. Now we just need to target the small play button within the expanded station view and click it to begin playback. Don’t waste your time looking through the previous list of uiElements to find this play button though, because it won’t be there. When we first called the entireContents() command on our iTunes window, the iTunes Radio station was collapsed, meaning that in the eyes of iTunes’ accessibility framework, none of the elements within the expanded view existed. Now that the station is expanded, we can call iTunesController.windows['iTunes'].entireContents() again, and the new list will include the play button we are looking for.

The new list is even bigger than the last one, and since we are looking for a Button instead of an Image, it’s harder to narrow down the choices. However, if we do a quick scroll through the list (after expanding it with a find/replace to make it easier to navigate), we’ll see that iTunes had made it easy on us. Within the interface of the expanded station view are lists of songs, one for the history of songs you’ve listened to on the station, one for the songs you’ve flagged for the station to play more of, and one for the songs you’ve flagged to never be played. Within our list of elements, we’ll see that iTunes kindly displays all of those song elements by name instead of by index. In my particular interface, the top song on my “Play More Like This” list is “All Eyes on You” by St. Lucia. A quick search for “all eyes” in my TextEdit window where I pasted my entireContents() results jumps me right to the UIElement I’m looking for. The play button is just slightly above that song on the interface, so I can scroll up through my list of items and assume it is not too far above. A little way up my list, I see a Button element with the name “Good Songs”, the title Button for my station. Above this are two unidentified Button elements, which I can venture to guess correspond to the only two other buttons near the station’s title: the play and share buttons. Calling description() on the first of these buttons returns => "play", and with that I’ve found what I’m looking for.

Regardless of what your station looks like, the path to the play button will be the same, so the last line in our script, which will start playback on the first iTunes Radio station in your My Stations list, will be as follows:

iTunesController.windows[0].scrollAreas[1].uiElements[0].groups[0].buttons[0].click()

There’s one last thing to do, and it’s because of a drawback with UI automation. If you watch the iTunes UI when you click on an iTunes Radio station, you will see that the station expands with an animation. What this means to our script is that there is a fraction of a second between clicking the iTunes Radio station Image and having the station fully expanded. During this small amount of time, the uiElements within the expanded view are still not available to the accessibility framework. While in the majority of my tests this did not cause issue, there were some times when the station would not expand fast enough, and the command that was supposed to click the play Button would end up clicking the Button at the index before the view expands. Thankfully, there is an easy fix for this, a fix that should help if you find this happening in any of your scripts: the delay() command. We can add one more line to our script and tell it to delay for half a second between clicking the station Image and clicking the play Button, so that we can be certain the animation has completed.

With that, our final UI automation script looks like this:

system = Application('System Events')
iTunesController = system.processes['iTunes']
iTunesController.windows[0].radioGroups[0].radioButtons[0].click()
    iTunesController.windows['iTunes'].radioGroups[1].radioButtons['Radio'].click()
iTunesController.windows['iTunes'].scrollAreas[1].uiElements[0].images[0].click()
delay(0.5)
iTunesController.windows[0].scrollAreas[1].uiElements[0].groups[0].buttons[0].click()

Closing Notes

Hopefully walking through the creation of these two fairly simple examples gave you some strategies to apply to any scripts you want to build with JXA.

I would definitely recommend using application automation via supported methods over UI automation wherever possible. While UI automation is powerful in its ability to automate nearly anything you could wish to in any application, it is not as solid of a method as using built-in methods. A couple of other issues to think about with UI automation: first, the application should be open before you try to run any UI automation scripts (or if not, you should open it at the start of your script, then run a delay() command to make sure it has loaded) so that you aren’t trying to automate a nonexistent UI, and second, if an application gets an update which breaks your scripts, it might be because the developer has changed the position of some of the uiElements you were targeting and you need to relocate them and update your script. Neither of these problems needs to be worried about with automation through methods (assuming an update does not completely remove a method you use, which is fairly unlikely).

Automating applications on OS X is an extremely powerful ability, and can save you huge amounts of time. My simple iTunes automation scripts are just conveniences, but the ideas behind them can be appropriated to other, more powerful apps to great effect. If you take the time to evaluate some of your common Mac workflows and take a look at the built-in methods in the productivity apps you use most, you may find that you can use JXA to greatly enhance your efficiency at certain tasks.

Automation can be used for less prestigious goals as well. Finder has built in some excellent support for various methods, and if you need to do some simple task a multitude of times (such as renaming a large batch of folders or moving a bunch of files), perhaps you can write a script to do the work for you programmatically.

If you think that taking the time to write a script to do something for you will just slow you down, I’ll leave you with this chart from Bruno Oliveira.

Happy scripting.