Post History

How to properly manage PGP signing key storage, access, and usage for dev and automation?

I work for an IoT company. We recently got secure boot to work for a customer-visible product. For secure boot to work, the images we generate must be signed. Obviously, we want customer-visible system releases to be signed. However, we need to make sure our signing keys are stored, accessed, and used *properly*.

At a high level, given our current infrastructure (Docker, Jenkins, etc), we seem to have 3 options:

1. Put the keys in the Docker images used to generate the customer releases;
2. Make it so that only Jenkins has an automated process to access keys and sign the images;
3. Build a separate internal service where images to be signed get submitted by automation and/or users.

The problem with #1 is that the signing keys would be everywhere and every dev/QA user using the Docker images would have access to the keys, by definition. **We'd like to avoid this.**

The problem with #2 is that it looks very similar to #1, but with additional steps. In addition, anyone with sys-admin access to the server(s) would also have access to those keys. While this is a smaller subset, all the knowledge of how/where to access keys would also be visible in automated scripts, which are visible to everyone. **We'd *also* like to avoid this.**

The main issue with #3 is that we're concerned that it might become a bottleneck. (The images to be signed are hundreds of MBs, so transfer speeds from homes over VPN become an issue quickly, among other things.) At least in my opinion, this one *seems* to be the most promising regarding secure storage, controlled access, and so on.

Proper key storage and controlled access/usage is also a concern and, personally, I'm leaning towards option #3 for the following reasons:

1. Keys stored in a single internal server;
2. Keys accessed *indirectly* in controlled manner;
3. Server access is limited to a very small number of people;

Regarding actual key storage, I had considered using the Gnome Keyring, but that seems to require an X session, and having graphical environments in a server is something we don't intend to do.

Given the options, infrastructure, and constraints I mentioned:

1. **What are some *good* options could we consider to manage image signing when we want secure boot enabled?**
2. **What are *good* key management approaches we could realistically consider/use?**

For #2, I'm thinking about the possibility of generating "ephemeral" keys signed by the root/master key. This seems like it could help alleviate the image transfer issue, because you'd be transferring short-lived keys and signing locally, not transferring the images - but this also seems like a potential security hole and extra work for the actual controlled access.

I'd appreciate good recommendations so that we can properly/securely manage our keys while allowing devs and automation "Easy"(tm) ways to sign generated images when needed.