By Frank Yang, Aidan Mulrooney, and Rey Sparby

Introduction

You may have heard the buzz — Apple announced that starting with macOS 10.14.5:

With the public release of macOS 10.14.5, we require that all developers creating a Developer ID certificate for the first time notarize their apps, and that all new and updated kernel extensions be notarized as well.

So what does this mean for Mac Admins? And why are people talking about staplers?

We spent a lot of time investigating this, and we’ve got some preliminary answers for you.

This information is meant to be high level and is up to date as of the time of posting. We will try to keep this post updated as we learn more and if anything changes!

Spoiler: Things might break, and it depends solely on developers updating their apps before the 10.14.5 release. Read on to find out why!

What is notarization?

From Twocanoe’s very informational summary:

Prior to notarization, macOS used Gatekeeper to prevent apps downloaded from the internet from being launched. macOS kept a list of known apps that were known to have issues and prevent then from being executed. However, after the app passed Gatekeeper and was approved by the user, it was difficult to detect if an existing binary got infected and there was no good way to revoke the approval of the application (the developer’s distribution certificate could be revoked but that would revoke all the developer’s applications). To combat malware, Apple introduced notarization, hardening, and stapling.

Notarization is a new layer of verification on top of the current Gatekeeper security checks.

Prior to Mojave (10.14), Apple only required a registered Apple ID code signature to trust an app.  With notarization, Apple now also inspects submitted code for “known malware” and “common code signing problems that can prevent your software from installing properly.” An app that goes through these (currently optional) additional checks and passes is considered “notarized.”

This is likely Apple’s attempt to better defend against security issues like those found earlier this year, which revealed that improperly signed software (that can be modified to include malicious software) would still be trusted by macOS.

Give users even more confidence in your software by submitting it to Apple to be notarized. The service automatically scans your Developer ID-signed software and performs security checks. When it’s ready to export for distribution, a ticket is attached to your software to let Gatekeeper know it’s been notarized.

So far, so good. Additional security! What’s not to like?

Well, that depends on whether or not the apps running on your devices are already notarized.

What do we know so far?

Here’s the high level:

Notarization is the process of submitting the app or kernel extension (kext) to Apple for the stamp of approval.

The notarization process requires app developers to submit their apps or kernel extensions to Apple for review.  After reviewing, Apple gives a nod of approval and notes the app/kext in its servers. This is the responsibility of the app developer to do before distributing their app, not the Apple admin.

Stapling is a final step of the notarization process that allows the notarized app to run on macOS without checking into Apple servers.

Once an app is notarized, Apple provides developers with a “ticket” that can be “stapled” to the notarized object. If an app or kext is not stapled, when the app/kext loads, macOS will check-in with Apple servers to see if it’s okay to run, which requires an internet connection.

In a future version of macOS, notarization will be required by default for all software. It is currently not required for applications, and will not be for 10.14.5.

Apple would like for developers to start notarizing their apps prior to distribution, but is rolling it out slowly. At the moment, notarization is very strongly suggested, but not quite a requirement.

Starting with macOS 10.14.5, kernel extensions must be notarized, and those that are not notarized will fail on load.

We’ve seen this internally with installations of VMWare on macOS 10.14.5 betas, where the app fails with the following error:

Exception: kernel extensions that have been signed before April 7 will be grandfathered in and will continue to function (as of 10.14.5 beta 4).

What this means:

  • Apps that have not been updated since April 7, at this moment, will work in macOS 10.14.5 beta 4 as expected.
  • If an app with kernel extensions (kext) has updated since April 7 and is not notarized, it will not install successfully.

Notarized kexts have an associated secure timestamp that will be required as of macOS 10.14.5. Non-notarized kexts signed before 4/7/2019 will be grandfathered in and will continue working in macOS 10.14.5 beta 4. However, signed kexts deliberately not timestamped or signed in an air-gapped environment may not. The timestamp is the important part of the equation.

So, to recap:

  • Notarization is the process of submitting it to Apple so Apple “knows” about the app.
  • Stapling is the process that attaches notarization to an app or kext such that it can run offline or on a limited network.
  • App notarization is the responsibility of the developer, and is currently a suggestion, not yet a requirement. macOS 10.14.5 will most likely prominently highlight the notarization of an app.
  • Kernel extension notarization IS a requirement for 10.14.5. Kernel extensions that are not notarized will fail in 10.14.5 on load.
  • Exception: If the kext has a timestamp before 4/7/2019, notarization is not required for the kext to load in 10.14.5 beta 4. If it’s after that cutoff date, notarization is required.
  • NOTE: This cutoff date will change during the course of the betas, up until the final release.
  • Cutoff date for 10.14.5 beta 2: 3/11/2019.
  • Cutoff date for 10.14.5 beta 3: 4/7/2019

What does this mean for me as a MacAdmin?

The information above is useful for awareness, but unfortunately there’s not much you as an admin can do to prevent issues from happening.  

It is the responsibility of the app developers themselves to properly notarize their apps and corresponding kexts. Failure to do so will simply make various apps unsupported in 10.14.5. Technically, you could notarize and staple an app on its developer’s behalf, but it would require you to build a custom deployment process to get it out. Having just gone through this ourselves… we don’t recommend it.

Your best bet is to evaluate your fleet and triage the apps now, before they cause disruption. You should do the following:

  1. Make a list of all the apps that you are distributing, and how they update (Are they updating automatically? Manual patch management?)
  2. If an app you are managing has NOT had an update since April 7, take note of the next time it updates and make sure it is notarized.
  3. If an app has had an update after April 7, check the release notes to see if the app and its kernel extensions have been notarized.
  4. Check to see if notarizing kernel extensions will change Team IDs and kernel extension whitelisting profiles. MacAdmins #notarization may be a good resource here. This is not likely to change, but it is a possibility.

TL;DR The biggest risk for admins is outdated apps at the time of 10.14.5 release. As 10.14.5 approaches, expect developers to release new notarized versions of their apps, and be diligent about deploying them across your fleet!

UPDATE (4/23/19):

It's worth taking note of your own app deployment processes as an admin as well! We've learned that stapling may not be recursive through the contents of what is stapled. What does this mean? If you are unpackaging and repackaging .pkgs for distribution, you should ensure that the contents of the .pkg are already stapled. They should be, but because these are best practices that are NOT clearly documented, there IS a chance that they are not.

Some troubleshooting tips

While there’s not a whole lot you can do as a MacAdmin if you find an app that hasn’t been updated, at least with this information you’ll know why an app isn’t working when 10.14.5 is released.

Here’s an easy way to validate the timestamp of a given kext. This does not provide confirmation that the kext was notarized, but it will let you know if it needs it:

  • stapler validate -v on the kext. Note: Use of the command does require Xcode and its command line tools installed.
  • Examine the secureTimestamp and see if it is prior to the cutoff or not.
  • If secureTimestamp is prior to the cutoff date, that particular kext will load.
  • Any kext signed since then, with a newer timestamp, will not load unless notarized.

To verify that a kext is signed, use this command:

kextutil -nt MyKext.kext

To check if an app is notarized:

spctl -v -a <app>

To get a list of all apps that have been notarized, run this in your terminal (this requires Xcode and Command Line Tools installed):

for i in /Applications/* ; do stapler validate "${i}" | grep -B 1 worked; done

How is Fleetsmith handling this?

We will be including notarization into our agent build process in the upcoming weeks, so the Fleetsmith Agent will definitely be good to go Day 0 of macOS 10.14.5! 🎉

What about the apps in the Fleetsmith catalog?

Glad you asked! We’ll continue to monitor the releases ourselves, and will be sanity checking each release against the beta releases. If there are new kext whitelisting Team IDs or Developer IDs, we will update them as we learn of them.

As we find problem apps that have not been properly notarized, we will surface the information in the catalog app’s infobox. You can be sure we’ll be raising a stink with any developers that aren’t notarizing their apps properly, and you should do this too. It may or may not get them notarized faster, but it will make you feel better.

It's also worth noting that proper kernel extension whitelisting (which Fleetsmith will do automatically) will bypass the notarization requirement! So either way, if you're using Fleetsmith, you should have a smooth transition with the new release of macOS 10.14.5.

With that said, Fleetsmith makes it easy to push the latest versions of the apps, so as long as you are keeping your fleet up-to-date with the latest versions, you are doing all that you need to do!

Resources we found useful

We recommend reading the Twocanoes post on this! The post does a great job summarizing why notarization is happening, and includes great definitions of notarization, stapling, app hardening, and ticket revocation:

https://twocanoes.com/apple-ramps-up-fight-against-malware-with-notarization-stapling-and-hardening/

Apple Documentation

Some resources on how to notarize:

Come join the conversation in the MacAdmins Slack group, in the #notarization channel; it’s honestly where we learned most of the information in this article. While you’re at it, stop by the #fleetsmith channel and say hi!