The OTA Bottleneck: Why Treble Wasn't Enough
Imagine security researchers discover a critical vulnerability in the Android media framework. They publish a proof of concept, and the clock starts ticking for every user in the world. Google engineers patch the bug within hours and merge the fix into the Android Open Source Project. You might expect that fix to land on your device a few days later, but instead, you wait three months.
Project Treble separated the Android OS from device hardware, but left the operating system itself as a giant monolithic block. If Google needed to update the media codec library, they could not just send you that one library. They had to compile an entirely new /system partition image.
Google handed that monolithic system image to your device manufacturer to ensure the changes did not break their custom interface. The manufacturer then handed the image to your cellular carrier. Carriers require extensive network certification for any full system update. A one-line code fix in a media library waited behind weeks of redundant 4G and 5G network tests.
The Treble architecture fixed driver compatibility. The architecture did absolutely nothing to speed up updating Android's core software libraries. The system partition acted as a single chunk of immutable data, and updating any part of that partition meant replacing the whole thing.
To solve this problem, Google realized they needed to shatter the core OS into individual, independent pieces.
Shattering the Monolith: Reclaiming Core Components
Updating critical system libraries required bypassing carriers without breaking the trust model. If you find a bug in a standard app on your phone, the developer pushes a fix to the Play Store. You download a small package, the app restarts, and the bug disappears. The cellular provider has zero involvement in that process.
Android platform engineers wanted that exact same delivery model for the operating system internals. If Google could update the networking stack or the media framework like a standard app, they could push security patches directly to the device. This strategy required pulling core libraries out of the monolithic /system partition.
We call these direct-to-device deployments Google Play System Updates. Instead of one massive update containing everything, Google packages components like DNS resolvers, media codecs, and timezone data individually. When a government creates a new timezone, Google pushes a tiny package just for the timezone data. The deployment completely bypasses the carrier testing pipeline.
But delivering a framework update like an app presented a massive technical hurdle. Standard APKs cannot load early enough in the boot sequence to replace core system libraries.
Project Mainline: Modularizing the Framework
Google can easily update higher-level components like the system user interface using standard APK files. Native system libraries required a completely new approach. These libraries manage the low-level functions of the device. If the media daemon crashes, the Android runtime cannot play sound or video.
Project Mainline introduced vertical modularity to Android 10. Treble was horizontal modularity, splitting the OS from the hardware HALs. Mainline split the OS from itself. Google identified specific security-critical and privacy-sensitive parts of the framework and ripped these components out of the immutable system image to turn them into interchangeable blocks.
We call these interchangeable blocks Mainline modules. Some modules manage network security, like the Conscrypt library for TLS encryption. Other modules handle media parsing or Wi-Fi connectivity. The primary goal was consistency across the ecosystem, meaning a device manufacturer can still customize their launcher, but they cannot heavily modify the Mainline Wi-Fi module.
While standard APKs were sufficient for some higher-level modules, native system libraries required a completely new file format capable of loading before the Android runtime even exists.
The APEX Format: Like APKs, But For The OS
You cannot use an APK to update the native C++ libraries that the operating system needs during boot. The PackageManager service parses APKs and extracts their contents. PackageManager operates as a Java service running inside the Android runtime. If you need to patch the libraries that start the Android runtime, you create an impossible circular dependency.
Platform engineers invented the Android Pony EXpress format to break this loop. We call them APEX files. An APEX file looks like an APK from the outside, but its internal structure solves the early-boot problem. Instead of containing compressed files that require extraction, an APEX file contains a raw filesystem image.
Let us look at the structure of an APEX file to see how it differs from a standard application package. We will map out the internal components that make early mounting possible.
The critical difference rests in payload.img. This file provides a self-contained ext4 or f2fs filesystem image. The operating system does not extract the files inside this image. The kernel mounts the image directly onto the device directory tree as a loopback device, and the apex_manifest.json tells the system exactly what this image contains and what version it represents.
Common Mistake: Many developers assume the system extracts APEX files to disk like ZIP archives. The daemon mounts the payload image as a virtual filesystem, completely bypassing the need for file extraction during boot.
With the APEX file constructed as a mountable image, engineers had to rewrite the boot sequence to load these modules before the rest of Android wakes up.
Early Init and the APEX Daemon (apexd)
To make an APEX module act like a native system library, the file must be available the exact millisecond the core system processes start looking for their dependencies. The Linux kernel boots first, followed by the Android init process. The init process takes responsibility for starting everything else. Engineers had to teach init how to handle modular filesystems before it started the main Android framework.
Android introduced a new low-level daemon called apexd. The init process launches apexd incredibly early in the boot sequence. The daemon scans the device for APEX files, verifies their cryptographic signatures, and creates a loopback mount for each one.
We can trace this boot sequence to see exactly when the APEX files become part of the active directory tree.
The apexd daemon mounts the payloads into a specific directory called /apex. If the module bears the name com.android.media, its contents appear at /apex/com.android.media. The system environment variables tell the operating system to look in this /apex directory for libraries before falling back to the old /system partition. If a corrupted APEX update causes a boot crash, apexd detects the failure on the next restart and automatically rolls back to the factory version.
This low-level mounting mechanism operates invisibly to the user, but its power becomes instantly obvious when security researchers discover a critical vulnerability in the wild.
Anatomy of an Update: Patching a Media Codec Vulnerability
Understanding the theory of modularity helps, but the true test happens on a physical device during an actual update. Let us return to our theoretical media framework vulnerability. With Project Mainline in place, the response timeline changes entirely.
Google engineers merge the fix into the Android Open Source Project. They compile a new com.android.media.swcodec APEX file and upload the package to the Play Store servers. Your device periodically checks for Google Play System Updates in the background.
The system downloads the new APEX file and stages it in the /data/apex/active directory. The currently running operating system ignores the new file because the old media codec still resides in active memory. Android simply displays a notification asking you to restart your phone to finish installing an update.
When you tap restart, the phone reboots. The init process starts apexd, which finds the newly downloaded APEX file in the staging directory. The daemon verifies the signature, mounts the new payload.img, and proceeds with the boot. The entire process takes the exact same amount of time as a normal reboot.
By making this process routine, Android fundamentally changed its architectural identity from a solid block of stone to a collection of interchangeable bricks.
The Big Picture: Android's Modular Architecture
Building an operating system out of separate modules solves the update problem, but completely shatters the traditional compilation model. Early versions of Android shipped as a single, tightly coupled codebase. Engineers compiled every component together into one massive system image. Project Treble sliced the operating system horizontally, permanently separating the framework from the vendor hardware drivers, and Project Mainline then sliced the framework vertically, isolating core components into upgradeable modules.
The modern Android platform functions as a grid of dynamic dependencies. The vendor partition handles the hardware. The APEX mounts handle the core system libraries. Standard APKs handle the user interface. When you turn on your phone, the device dynamically stitches all these independent pieces together into a unified experience.
Constructing an OS dynamically out of mounted modules and hardware abstraction layers offers incredible flexibility. This flexibility comes with a massive cost. Layering all these interfaces creates inherent performance overhead, and memory management becomes significantly more complicated when libraries no longer compile as a single unit. You might wonder how Google prevented this heavily abstracted OS from grinding to a halt under its own weight.