by Jesse Endahl
As part of our commitment to security, we’re happy to announce that we’ve been helping HashiCorp and Google Cloud test an exciting new Vault plugin called the Google Compute Engine (GCE) authentication plugin for Vault, which was released today.
Here’s why we chose Vault and Google, how we secured new GCE instances to Vault, and how we maintained our high bar for security throughout:
One of our company values is "Secure by Design," which means security is deeply considered in every aspect of our product, and in every decision we make. When it comes to building a modern product like Fleetsmith, multiple layers—each with sensitive information—have to be considered:
- Web services and edges, which live inside our Kubernetes cluster
- Virtual Machines (VMs), which host services like Consul
- Clients, like our web frontend and macOS agent
We chose to build Fleetsmith on Google Cloud Platform in part due to Google’s high standards for security, and we chose to use HashiCorp Vault as our secrets management solution due its carefully thought-out architecture and security model. Vault provides features that cover multiple important use cases including certificate generation & management, and encryption of data at rest.
Confident in those decisions, we spun up our infrastructure and Vault servers. But once we had Vault up and running, we found that solving one problem led to another—how should we authenticate brand new GCE instances to Vault? Before today, there was no easy answer.
One solution is for an Ops Engineer to manually authenticate to Vault, get a token, manually login to the new GCE instance, and place the token in a location where it can be used. But in a world where we might create multiple instances in a short period of time, this amount of manual work can quickly become a bottleneck.
Another approach is to create one or more Google Storage buckets containing a Vault token that is issued (and reissued) on a schedule. Then, new instances download their Vault token from the bucket automatically at boot. While this is an improvement over the first method, it requires additional overhead to manage access control to the buckets.
Additionally, the most straightforward implementation is to use one shared token per group of instances, rather than one unique token per instance. Without one to one mapping between instances and tokens however, we couldn’t accomplish one of our related security goals—the ability to attribute token usage to individual machines for the sake of identifying token theft/reuse.
We chose the new GCE authentication plugin, which allowed for a substantially better approach. It takes advantage of the recently introduced signed metadata feature to allow for secure bootstrapping of Vault authentication tokens on new GCE instances. This solved the chicken and egg problem of how to grant a Vault token to a brand new GCE instance in a trusted manner, right at the moment it’s created.
To do this, the process is relatively simple. From the GCE instance:
- We make a request to the metadata server.
- The metadata server responds, and we extract the signed JSON Web Token (JWT) from the response.
- We pass the signed JWT to Vault, and Vault gives us back a Vault auth token.
- We use the Vault auth token we received in step 3 to authenticate to Vault.
Here’s a diagram of how it works:
After testing, our last step was automating the feature. We strongly believe in infrastructure as code. When operating in a cloud environment, the goal is to automate as much as possible so that resources can be scaled automatically based on demand.
To help other companies get started, we released a small sample script that demonstrates how to use the feature using an automated approach. You can find it here.
Finally, we’re excited to continue improving our secrets management infrastructure by implementing the new Vault ↔ Kubernetes integration that was also announced today.
More information on today’s announcements can be found here:
- New Google Cloud Platform authentication backends
- Google Cloud Platform solution
- Google Cloud Platform blog