This Week's Sponsor:

PowerPhotos

The Ultimate Toolbox for Photos on the Mac

Posts tagged with "ChatGPT"

Apple Intelligence in iOS 18.2: A Deep Dive into Working with Siri and ChatGPT, Together

The ChatGPT integration in iOS 18.2.

The ChatGPT integration in iOS 18.2.

Apple is releasing iOS and iPadOS 18.2 today, and with those software updates, the company is rolling out the second wave of Apple Intelligence features as part of their previously announced roadmap that will culminate with the arrival of deeper integration between Siri and third-party apps next year.

In today’s release, users will find native integration between Siri and ChatGPT, more options in Writing Tools, a smarter Mail app with automatic message categorization, generative image creation in Image Playground, Genmoji, Visual Intelligence, and more. It’s certainly a more ambitious rollout than the somewhat disjointed debut of Apple Intelligence with iOS 18.1, and one that will garner more attention if only by virtue of Siri’s native access to OpenAI’s ChatGPT.

And yet, despite the long list of AI features in these software updates, I find myself mostly underwhelmed – if not downright annoyed – by the majority of the Apple Intelligence changes, but not for the reasons you may expect coming from me.

Some context is necessary here. As I explained in a recent episode of AppStories, I’ve embarked on a bit of a journey lately in terms of understanding the role of AI products and features in modern software. I’ve been doing a lot of research, testing, and reading about the different flavors of AI tools that we see pop up on almost a daily basis now in a rapidly changing landscape. As I discussed on the show, I’ve landed on two takeaways, at least for now:

  • I’m completely uninterested in generative products that aim to produce images, video, or text to replace human creativity and input. I find products that create fake “art” sloppy, distasteful, and objectively harmful for humankind because they aim to replace the creative process with a thoughtless approximation of what it means to be creative and express one’s feelings, culture, and craft through genuine, meaningful creative work.
  • I’m deeply interested in the idea of assistive and agentic AI as a means to remove busywork from people’s lives and, well, assist people in the creative process. In my opinion, this is where the more intriguing parts of the modern AI industry lie:
    • agents that can perform boring tasks for humans with a higher degree of precision and faster output;
    • coding assistants to put software in the hands of more people and allow programmers to tackle higher-level tasks;
    • RAG-infused assistive tools that can help academics and researchers; and
    • protocols that can map an LLM to external data sources such as Claude’s Model Context Protocol.

I see these tools as a natural evolution of automation and, as you can guess, that has inevitably caught my interest. The implications for the Accessibility community in this field are also something we should keep in mind.

To put it more simply, I think empowering LLMs to be “creative” with the goal of displacing artists is a mistake, and also a distraction – a glossy facade largely amounting to a party trick that gets boring fast and misses the bigger picture of how these AI tools may practically help us in the workplace, healthcare, biology, and other industries.

This is how I approached my tests with Apple Intelligence in iOS and iPadOS 18.2. For the past month, I’ve extensively used Claude to assist me with the making of advanced shortcuts, used ChatGPT’s search feature as a Google replacement, indexed the archive of my iOS reviews with NotebookLM, relied on Zapier’s Copilot to more quickly spin up web automations, and used both Sonnet 3.5 and GPT-4o to rethink my Obsidian templating system and note-taking workflow. I’ve used AI tools for real, meaningful work that revolved around me – the creative person – doing the actual work and letting software assist me. And at the same time, I tried to add Apple’s new AI features to the mix.

Perhaps it’s not “fair” to compare Apple’s newfangled efforts to products by companies that have been iterating on their LLMs and related services for the past five years, but when the biggest tech company in the world makes bold claims about their entrance into the AI space, we have to take them at face value.

It’s been an interesting exercise to see how far behind Apple is compared to OpenAI and Anthropic in terms of the sheer capabilities of their respective assistants; at the same time, I believe Apple has some serious advantages in the long term as the platform owner, with untapped potential for integrating AI more deeply within the OS and apps in a way that other AI companies won’t be able to. There are parts of Apple Intelligence in 18.2 that hint at much bigger things to come in the future that I find exciting, as well as features available today that I’ve found useful and, occasionally, even surprising.

With this context in mind, in this story you won’t see any coverage of Image Playground and Image Wand, which I believe are ridiculously primitive and perfect examples of why Apple may think they’re two years behind their competitors. Image Playground in particular produces “illustrations” that you’d be kind to call abominations; they remind me of the worst Midjourney creations from 2022. Instead, I will focus on the more assistive aspects of AI and share my experience with trying to get work done using Apple Intelligence on my iPhone and iPad alongside its integration with ChatGPT, which is the marquee addition of this release.

Let’s dive in.

Read more

How ChatGPT Changed Tech Forever→

I thoroughly enjoyed this story from a couple weeks ago by David Pierce, writing for The Verge about OpenAI’s ChatGPT turning one and how it created a revolution in the tech industry that no one saw coming:

We definitely seem to like being able to more quickly write business emails, and we like being able to ask Excel to “make this into a bar graph” instead of hunting through menus. We like being able to code just by telling ChatGPT what we want our app to do. But do we want SEO-optimized, AI-generated news stories to take over publications we used to love? Do we want AI bots that act like real-life characters and become anthropomorphized companions in our lives? Should we think of AI more as a tool or a collaborator? If an AI tool can be trained to create the exact song / movie / image / story I want right now, is that art or is that dystopia? Even as we start to answer those questions, AI tech seems to always stay one step and one cultural revolution ahead.

At the same time, there have been lawsuits accusing AI companies of stealing artists’ work, to which multiple US judges have said, essentially: our existing copyright laws just don’t know what to do with AI at all. Lawmakers have wrung their hands about AI safety, and President Joe Biden signed a fairly generic executive order that instructed agencies to create safety standards and companies to do good and not evil. There’s a case to be made that the AI revolution was built on immoral and / or illegal grounds, and yet the creators of these models and companies continue to confidently go ahead with their plans, while saying it’s both impossible and anti-progress to stop them or slow them down.

This all gets really heady really fast, I know. And the truth is, nobody knows where all this will be even 12 months from now, especially not the people making the loudest predictions. All you have to do is look at recent hype cycles — the blockchain, the metaverse, and many others — for evidence that things don’t usually turn out the way we think. But there’s so much momentum behind the AI revolution, and so many companies deeply invested in its future, that it’s hard to imagine GPTs going the way of NFTs.

I recommend reading the whole piece on The Verge. I quoted these paragraphs because they get right to the heart of the conflict that I also feel whenever I think about ChatGPT and similar tools. On the one hand, they were (largely? Partially?) built with data sets stolen from artists and creators (including this very website); on the other, the practical benefits of, say, using ChatGPT to help me proof-read my articles are undeniable.

I’ve been thinking about these issues a lot, perhaps because I make a living out of, well, creating content for the Internet. Is there a way to enjoy the power of LLMs without feeling weird and conflicted about how they were made in the first place? Will it even matter years from now? I don’t know the answer, but I’m hoping Apple will have one.

Permalink

S-GPT 1.0.2 Brings Date and Time Awareness, Integration with macOS Services Menu, Passthrough Mode, Better HomePod Support, and More

S-GPT 1.0.2.

S-GPT 1.0.2.

I just published version 1.0.2 of S-GPT, the shortcut I released last week to have conversations with OpenAI’s ChatGPT and integrate it directly with native features of Apple’s OSes. You can find the updated download link at the end of this post, in the original article, and in the MacStories Shortcuts Archive; before you replace version 1.0.1 of S-GPT, save your existing OpenAI API key somewhere as you’ll have to paste it again in the shortcut later.

I’m going to include the full changelog for S-GPT 1.0.2 below, but long story short: S-GPT is now aware of the current date and time, and I’ve heard all the requests about improving interactions with the HomePod and Siri, so I made that part much better. S-GPT can now perform a variety of date/time calculations with natural language, and you can end a conversation by saying “no” or “stop”.

Read more

S-GPT 1.0.1→

I just released a small bug fix update for S-GPT, my shortcut to integrate OpenAI’s ChatGPT large language model with the Shortcuts app on all Apple platforms.

Version 1.0.1 of S-GPT is a relatively minor update that comes with an initial round of improvements based on early feedback I’ve received for the shortcut, including:

  • A proper error-checking alert that tells what went wrong with a request to the ChatGPT API;
  • A better summarization of Safari webpages passed from the share sheet that no longer “hallucinates” results but actually summarizes text extracted via Safari’s Reader technology from any web article;
  • A new behavior for text input on watchOS, which now defaults to dictation rather than keyboard input. I’ve covered this more in detail in today’s issue of MacStories Weekly for Club members.

Additionally, I also realized that the usage tips that S-GPT was displaying every time it asked you to enter some text may have been nice the first three times you used the shortcut, but became annoying very quickly. That was especially true when using S-GPT with Siri in a voice context since they would be read aloud every time. For these reasons, I removed tips and simplified the shortcut’s questions to “What do you want to ask?” and “Want to follow up?”.

In case you missed my introduction of S-GPT earlier this week, you can read the original story here and find out more about how the shortcut works and what it does. I updated the links to the S-GPT and S-GPT Encoder shortcuts in the story to the latest version; you can also find the updated shortcuts in MacStories Shortcuts Archive.

I just released a small bug fix update for S-GPT, my shortcut to integrate OpenAI’s ChatGPT large language model with the Shortcuts app on all Apple platforms.

Version 1.0.1 of S-GPT is a relatively minor update that comes with an initial round of improvements based on early feedback I’ve received for the shortcut, including:

  • A proper error-checking alert that tells what went wrong with a request to the ChatGPT API;
  • A better summarization of Safari webpages passed from the share sheet that no longer “hallucinates” results but actually summarizes text extracted via Safari’s Reader technology from any web article;
  • A new behavior for text input on watchOS, which now defaults to dictation rather than keyboard input. I’ve covered this more in detail in today’s issue of MacStories Weekly for Club members.

Additionally, I also realized that the usage tips that S-GPT was displaying every time it asked you to enter some text may have been nice the first three times you used the shortcut, but became annoying very quickly. That was especially true when using S-GPT with Siri in a voice context since they would be read aloud every time. For these reasons, I removed tips and simplified the shortcut’s questions to “What do you want to ask?” and “Want to follow up?”.

In case you missed my introduction of S-GPT earlier this week, you can read the original story here and find out more about how the shortcut works and what it does. I updated the links to the S-GPT and S-GPT Encoder shortcuts in the story to the latest version; you can also find the updated shortcuts in MacStories Shortcuts Archive.

S-GPT

S-GPT is a shortcut to have conversations with OpenAI’s ChatGPT assistant on your iPhone, iPad, and Mac. The shortcut supports both text conversations as well as voice interactions when used inside Siri. S-GPT comes with native system integrations on Apple platforms including the ability to process text from your clipboard, summarize text found in photos, export conversations to Files and Finder, and even create playlists in the Music app. The shortcut requires an OpenAI API token and a helper shortcut called S-GPT Encoder that needs to be downloaded separately.

Get the shortcut here.

S-GPT Encoder

This is a helper shortcut for S-GPT that needs to be downloaded and installed separately. Without this shortcut, S-GPT won’t work.

Get the shortcut here.

Permalink

Introducing S-GPT, A Shortcut to Connect OpenAI’s ChatGPT with Native Features of Apple’s Operating Systems

S-GPT for Shortcuts.

S-GPT for Shortcuts.

Update, April 13: I’ve updated S-GPT to version 1.0.2. You can read the full changelog here. All download links have been updated.

Update, April 13: For Club MacStories+ and Premier members, I’ve published Part 1 of an extensive ‘Making Of’ series about S-GPT. This is a technical deep dive for my Automation Academy series. You can find it here and sign up for or upgrade to a Premier account using the buttons below.

Update, April 7: For Club MacStories members, I’ve shared some optional prompts to add different personalities to S-GPT, including two inspired by Roy Kent and Steve Jobs. You can get the prompts and read more here; the main S-GPT shortcut is and will remain free-to-use for everyone, of course.

Update, April 7: I’ve updated S-GPT to version 1.0.1. You can read more details here. All download links to the shortcuts have been updated to the latest version.

It’s the inaugural week of the second annual edition of Automation April, and to celebrate the occasion, I’ve been working on something special: today, I’m introducing S-GPT, an advanced conversational shortcut for ChatGPT that bridges OpenAI’s assistant to native system features of iOS, iPadOS, macOS, and watchOS.

S-GPT (which stands for Shortcuts-GPT) is free to use for everyone, but it requires an OpenAI account with an associated pay-as-you-go billing plan since it takes advantage of OpenAI’s developer API, which has a cost. S-GPT was built with the latest ChatGPT API, and it can be used both with the existing ChatGPT 3.5 model or – if you have access to it – the ChatGPT 4 API.

While the shortcut is free for MacStories readers, I will be publishing a detailed, in-depth Automation Academy class soon for Club MacStories Plus or Premier members to explain the techniques and strategies I used to build this shortcut. I genuinely think that S-GPT is, from a technical perspective, my best and most advanced work to date; I hope my Academy class will help others learn some useful tips for Shortcuts and, in return, make even better automations for our contest.

With that said, let’s look at what S-GPT is and what you can do with it.

Read more