We’ve recently been receiving reports of customers having intermittent issues with enforcing macOS Mojave 10.14.5, where the OS update would seemingly fail. These devices, after receiving a notification for a pending macOS Mojave enforcement, begin the OS update process, but after multiple reboots, would end up on the same OS version they had started with. It seemed strange to us that an OS update would abort mid-update, so we did some research as soon as we had heard about the problem. It's something we've encountered internally as well. (We use Fleetsmith to manage our devices, after all)
Looking into the reported issues and also our own internally reported cases, we noticed that all the devices that had experienced an OS upgrade failure had a T2 chip. At this point our working theory is that there are inconsistencies involved when using
startosinstall (the command we use to enforce OS upgrades), and we’ve filed an Apple Bug Report (Problem ID: 51006318) to share what we’ve found with Apple.
What you can do
If your employees are experiencing this, we recommend updating the device through System Preferences > Software Update. We’ve found that this method to be a good backup for OS updates, and a fast time to resolution. While we’ve also seen the OS upgrade succeed via Fleetsmith enforcement after a number of attempts, we’re not confident enough at the moment to say that repeated attempts will definitely work. Because of this bad interaction between our enforcement feature and macOS upgrade failures, if the macOS upgrade fails and is enforced, Fleetsmith will repeatedly attempt to upgrade (and fail). For that reason, we recommend going through System Preferences.
Additional technical context
For those who are curious about the T2 chips and the nitty gritty behind how Fleetsmith triggers OS updates, we’ve detailed the results of our investigations so far. Some OS upgrade nuances for T2 chip devices:
- The T2 chip has its own OS: bridgeOS.
- As part of an OS install, macOS encodes the minimum bridgeOS version.
- When macOS begins the OS upgrade process, the device will first check and see if it's on the right version of bridgeOS, and if it isn’t, macOS will attempt to download and update bridgeOS in preparation for the macOS update. We’ve seen this step fail in previous versions of macOS upgrades, as it was captured in the installer logs.
- When the bridgeOS download fails, the OS upgrade process aborts completely.
For additional reading, check out this presentation.
We currently don’t have enough information to clearly pin the upgrade failures to this problem, but we do suspect it to be related. Unfortunately, the logs that we’ve looked into do not clearly call out these bridgeOS failures, but we have seen them in the past. Here’s some insight as to what the Fleetsmith agent is doing when the OS update is triggered:
- Agent downloads DMG that contains the latest Install macOS app (currently version 10.14.5, build 18F132).
- Agent mounts the DMG
- Agent runs the following command:
launchctl asuser <current user's UID> \ /<path to DMG>/Contents/Resources/macosmojave-10.14.5.app/Contents/Resources/startosinstall \ --agreetolicense \ --rebootdelay=1 \ --applicationpath=/<path to DMG>/Contents/Resources/macosmojave-10.14.5.app \ --pidtosignal=<agent process ID>
We’re currently re-testing this to understand the implications of running the command using asuser or as root, but seeing how failures only occur intermittently, we don’t suspect this to be the core issue. We are actively investigating this theory, but in the meantime, if you’re encountering this, please dupe the Radar so that Apple has more datapoints to isolate the issue!