I’ve been thinking about app scalability a lot lately – most recently in the context of TestFlight, which I find is incredibly frustrating to use, at best, and, on the Mac, often unusable. This isn’t a new problem for me, but I haven’t mentioned it much in the past because I’ve suspected that my experience is colored by the fact that I’m an outlier. But, outlier or not, the app deserves more attention than it’s been given.

You see, I test a lot of apps, so TestFlight is a little crowded. There’s no count in the app to tell you how many betas you’re on, so I never knew how many until I recently took the time to count them all. As it turns out, I’m currently testing 200 apps:

• 176 cross-platform apps
• 4 iPad-only apps
• 20 Mac-only apps

Of those, 93 have been updated within the past three weeks. The rest are somewhere closer to their 90-day expiration date.

I have automatic updates turned on for every beta, but except for once or twice, it has never worked.

Whether it’s that app load or something else that’s going on, I’ve had to contend with a lot of TestFlight usability issues and bugs for a long time. On the iPhone and iPad, it’s simply too hard to find an app to see whether I have the latest version installed. Why am I checking to see if an app is installed when TestFlight offers automatic updates of betas? Good question. You see, other than once or twice for short periods of time, automatic updates have never worked for me. They’re turned on for every beta, but unless I manually update each app, they refuse to download.

That’s a hassle in and of itself, but because TestFlight has no sorting, filtering, or search, finding a particular app can be like looking for a needle in a haystack. Even if the app I’m looking for has an active beta, it gets buried in TestFlight’s reverse chronological list quickly because it’s not uncommon for me to get a dozen beta updates or more each day, even in relatively slow development periods like now. The situation is far worse in the fall as new OS releases approach. But no matter the circumstances, the only option I have is a lot of careful scrolling.

TestFlight needs a master switch to turn off all email alerts.

A related problem is TestFlight email messages. I once went to the trouble of turning off email notifications for every active beta I’m on, but they quickly reappeared. I’m not sure if that’s because email notifications are turned on by default for every new beta, or they’re rising from the dead somehow, but I was fighting a losing battle, so I gave up and created a Gmail rule to shuffle them off under a separate label where I periodically mass delete them instead. I get so many email messages alerting me to beta updates that I also had to create a separate rule to alert me to new betas to make sure I don’t delete new invitations by accident.

Most days, this is all I see in TestFlight for a few minutes before it suggests I try again later.

However, as painful as it is to use TestFlight on the iPhone and iPad, the situation on the Mac is exponentially worse, where the app refuses to load betas at all for long periods of time. That rules out quick checks to be sure I have the latest version of an app before I finish a review. Instead, I open the app and let it sit in the background, periodically checking to see if it’s loaded any content. The good news is that, eventually, the betas show up, although sometimes missing icons and often disappearing again later.

The good news is that betas eventually load on the Mac, but not always with icons.

I’ve always suspected that TestFlight isn’t built to handle anywhere near this many betas, and in one sense, why should it be? Who else out there is beta testing almost 200 apps? I’d wager not many. That’s not some sort of flex. There just aren’t that many people writing about apps regularly.

To test my theory, I asked around on Mastodon to see how many betas people were managing in TestFlight and whether they’d had any issues with the app’s performance. Several of the people who responded have between 20 and 40 betas installed, and one person had about 70. Of those people, most reported no issues with TestFlight, although a few mentioned that betas weren’t auto-updating reliably or that the Mac version was buggy. The issues people mentioned are in line with my experience, although my issues with TestFlight are worse by an order of magnitude, which tends to corroborate my theory that the app isn’t designed to handle the load of almost 200 betas.

Eventually apps appear in TestFlight for Mac without having to use the Try Again button, which as far as I can tell doesn’t do anything.

So let’s assume I’m correct. I’ve overloaded TestFlight, which is causing what is an objectively bad user experience. What should the result be? One common reaction in this kind of situation is to blame the user, telling them some version of ‘you’re holding it wrong.’

I’ll grant you that TestFlight would probably work better if I dropped off of a bunch of betas, but that’s not realistic. I don’t have every beta installed at all times, but my job is to stay on top of the apps we cover, and TestFlight is a big part of that. I read the beta release notes in TestFlight, which plays a significant role in whether I take a closer look and consider writing about an app. Also, if I drop a beta to make TestFlight work better, only to hear about a feature I want to try, now I’ve got to waste a developer’s time requesting a new invitation, assuming they even have free TestFlight slots available. So no, I’m not dumping betas to make TestFlight work better. Nor should I have to.

I generally agree with the notion that most Apple system apps can cater to 80% of the features users want, leaving 20% to be fulfilled by third-party developers. That works well with apps like Notes and Reminders, where alternatives are possible and plentiful. However, TestFlight is an extension of App Store Connect, which is the only way to get apps on the iPhone, iPad, and Apple Watch, plus a significant but non-exclusive way to get them on the Mac. There is no good alternative, so TestFlight has to do better than 80%. I may be a weird outlier, but TestFlight should work for me too.

I like the TestFlight section in the App Library, but users should be able to turn it off.

So, as the self-designated App Store CTO (Chief TestFlight Outlier), I have some suggestions I’d like Apple to consider:

• Focus on performance first, starting with the Mac. My problems with the Mac version of TestFlight might be more severe than most people’s, but I’m not alone in my frustration. Betas should load quickly, scrolling should be smooth, and installing updates should happen without a hitch.
• Automatic installation requires some special attention on all platforms. It worked for me briefly after iOS 16 was released, which gave me enough of a taste of the good life that I want more. I’m not alone in this complaint either, so let’s make 2023 the year that automatic is, you know, automatic.
• Add an email notifications section to TestFlight’s settings, so I don’t have to adjust these settings app-by-app.
• Implement sorting by a beta’s release date from oldest to newest or newest to oldest, and alphabetically by name, developer, or app category.
• Implement search with filters for whether a beta has been installed or not.
• Offer folders inside TestFlight to allow me to organize betas based on whatever other criteria I want.
• Implement a versioning system that cross-checks against the App Store to let me know when the released version of an app has caught up to its beta version.
• I like the separate TestFlight section in the App Library, but make it optional, so, for example, that beta music app someone is testing will show up with their other music apps instead of a TestFlight group if that’s what they want.
• TestFlight should work better with factory resets of devices. I recently had to restore my Mac Studio from a Time Machine backup, and the process took a long time but worked well with one exception. When I logged into my newly restored Mac, I was greeted by messages of damaged or unavailable betas. Deleting and reinstalling betas or App Store versions of apps fixed things, but betas really should either be migrated smoothly to restored devices or not installed at all. A similar thing occurs on iOS and iPadOS, including when setting up a new device.

I could probably come up with even more ideas, but that should keep a few engineers busy for a while.

Source.

My first experience with TestFlight was at WWDC in 2013. I had been sitting in the cold outside the Moscone Center, waiting for the conference to kick off with Apple’s keynote. That’s the one where OS X switched to California destination names, Apple painted itself into a thermal corner, and we were introduced to a singing koala.

As the line began to move, a group of TestFlight employees walked up with t-shirts and breakfast burritos. The timing was perfect. I was starving and gratefully wolfed down a warm burrito, pulling the t-shirt on as an extra layer of insulation against the cool SoMa morning air. It was a great way to promote the already popular service to a bunch of cold and hungry developers.

Less than a year later, Apple acquired TestFlight, rolling it into their suite of developer tools. I don’t know what’s left of the TestFlight team that joined Apple, but I still think of that burrito and t-shirt now and then and wonder if the thoughtfulness that they represented got lost along the way somewhere. I hope not. Beta testing is an important part of the development process, and app developers and the testers who rely on TestFlight deserve better.