I live in Italy, but because of my work I communicate with friends and colleagues in English. All my devices are set to English, including Siri on our HomePods. As a multilingual iOS user, the addition of multilingual typing to the QuickType keyboard was, by far, one of my favorite changes in iOS 10. Besides being aware of the language used in each iMessage conversation, since 2016 the QuickType keyboard has been able to jump between multiple languages on the fly without switching keyboard layouts – which is amazing when you have conversations with people who live in different countries. I can’t believe there was a time when I was constantly switching between the Italian and English keyboards hundreds of times each day. For international users, it was as bad as not having copy and paste before iPhone OS 3.0.
With the release of Shortcuts and the ability to send iMessages in the background (without showing the Messages composer), I had an idea: what if instead of typing I could use dictation (also improved in Shortcuts) to quickly send a message from a widget? A shortcut to accomplish this seemed relatively easy to build, so I got to work. However, after a few minutes of tests, I realized that Shortcuts’ dictation didn’t support automatic language recognition – which meant I had to consider a more creative approach.
Multilingual Dictation Widget for iMessage
As usual, let’s start with the finished product and work our way back to the details of the shortcut itself. Here’s my Multilingual iMessage Dictation widget in action, featuring messages sent both in English and Italian:
Sending messages in English and Italian from a widget. (Tap volume button for audio.)Replay
The video shows a complete dictation workflow for iMessage that executes entirely within the widget without opening the Shortcuts or Messages apps at all. This is made possible by the ‘Dictate Text’ and ‘Send Message’ actions, which have been updated for Shortcuts to run inside the widget (or action extension). Furthermore, unlike the same action in the old Workflow app, the updated Dictate Text action (based on the native iOS speech recognition API) can now stop recording automatically after a short pause, which explains the natural interaction with the widget; you don’t have to press a stop button to finish dictation.
You may be wondering, though, how I managed to only pick the recipient of the message without also picking a language for dictation. Or how the shortcut has a single Dictate Text action instead of multiple ones for each language. This is where the aforementioned creative solution comes in.
By default, the Dictate Text action contains a Language field that needs to be manually configured to choose a dictation language beforehand. You can tap the language button to open a list of supported languages for dictation and select a different one. If you scroll all the way to the bottom of the list, you’ll also see an option to pick a language as a Magic Variable.
A little known fact about the Dictate Text action is that you can use language IDs as variables to load dictation mode in a specific language and thus ensure it actually understands what you’re saying. Language IDs supported by iOS are listed here, and they’re usually presented in language_REGION format, such as en_US (American English), it_IT (Italian, Italy), or fr_CA (French spoken in Canada). Even though the Dictate Text action doesn’t communicate this in its interface or documentation, using one of these locale codes as input for the action will change the language it’s listening for when it runs.
With this in mind, I set up the shortcut so that it would only ask me to pick a recipient from a list of favorite contacts then automatically set the dictation language without asking me to pick from a second list. To do this, I used location details stored in a contact’s entry in the address book.
The key idea at the core of this shortcut is that country information for a specific contact is matched against a dictionary of countries and locale codes. The Dictionary action is the very first one in the shortcut; you’ll have to modify it with countries and associated IDs based on the languages you’ll want to use with dictation.
The second action in the shortcut is the step that defines your list of favorite contacts. You have to fill this action manually by tapping on ‘+’ button to pick people from the Contacts database on iOS.
For this shortcut to work properly, you have to make sure your favorite contacts have a full address for the country they live in, which, in theory, should reflect the language they speak. In the examples below, you can see that my girlfriend Silvia lives in Italy, which is matched against the ‘Italy’ entry in the dictionary. All you have to do is pick your favorite contacts, make sure they have an address, and fill the list of countries and language IDs in the dictionary. I left some examples in the action to give you an idea of how it works.
After picking a contact from a list, Shortcuts uses a series of native Contacts and Location actions to turn a street address into a location object and, finally, a country name. This also happens to be a perfect example of the power of the Content Graph and Magic Variables in Shortcuts – you don’t have to care about any of these conversions.
At the end of the shortcut, the country name is matched with a language ID, which becomes the language variable for the Dictate Text action.
Shortcuts won’t visually alter the action in any way, but you’ll be able to speak in a different language, which will be correctly interpreted by iOS’ speech recognition. Finally, you’ll be able to confirm the recognized text in an alert (which can be cancelled) before the message is sent to the selected contact in the background, directly from the widget.
This shortcut could have also been possible with multiple Dictate Text actions, each manually adjusted to load speech recognition in a specific language based on a contact’s name. However, that approach would have resulted in a long and convoluted shortcut with lots of duplicate actions; it would have lacked the flexibility of a single Dictate Text action that dynamically changes language based on location details fetched from a contact.
At the same time, this shortcut is also a good demonstration of improvements to existing Workflow actions in the new Shortcuts app. Both ‘Dictate Text’ and ‘Send Message’ were supported before, but they were not as smart as they are in Shortcuts, and couldn’t take advantage of the deep integration with iOS that being a native Apple app entails. If you’re a multilingual iPhone user and often communicate with friends from different countries, I think you’re going to like what this shortcut can do for you.
Multilingual iMessage Dictation
Dictate an iMessage to a friend from a widget. The dictation language is set automatically based on the recipient’s address stored in their contact card. iMessages are sent in the background, but you’ll be asked to confirm the message before sending it.
