How to properly manage PGP signing key storage, access, and usage for dev and automation?

I work for an IoT company. We recently got secure boot to work for a customer-visible product. For secure boot to work, the images we generate must be signed. Obviously, we want customer-visible system releases to be signed. However, we need to make sure our signing keys are stored, accessed, and used properly.

At a high level, given our current infrastructure (Docker, Jenkins, etc), we seem to have 3 options:

1. Put the keys in the Docker images used to generate the customer releases;
2. Make it so that only Jenkins has an automated process to access keys and sign the images;
3. Build a separate internal service where images to be signed get submitted by automation and/or users.

The problem with #1 is that the signing keys would be everywhere and every dev/QA user using the Docker images would have access to the keys, by definition. We'd like to avoid this.

The problem with #2 is that it looks very similar to #1, but with additional steps. In addition, anyone with sys-admin access to the server(s) would also have access to those keys. While this is a smaller subset, all the knowledge of how/where to access keys would also be visible in automated scripts, which are visible to everyone. We'd also like to avoid this.

The main issue with #3 is that we're concerned that it might become a bottleneck. (The images to be signed are hundreds of MBs, so transfer speeds from homes over VPN become an issue quickly, among other things.) At least in my opinion, this one seems to be the most promising regarding secure storage, controlled access, and so on.

Proper key storage and controlled access/usage is also a concern and, personally, I'm leaning towards option #3 for the following reasons:

1. Keys stored in a single internal server;
2. Keys accessed indirectly in controlled manner;
3. Server access is limited to a very small number of people;

Regarding actual key storage, I had considered using the Gnome Keyring, but that seems to require an X session, and having graphical environments in a server is something we don't intend to do.

Given the options, infrastructure, and constraints I mentioned:

1. What are some good options could we consider to manage image signing when we want secure boot enabled?
2. What are good key management approaches we could realistically consider/use?

For #2, I'm thinking about the possibility of generating "ephemeral" keys signed by the root/master key. This seems like it could help alleviate the image transfer issue, because you'd be transferring short-lived keys and signing locally, not transferring the images - but this also seems like a potential security hole and extra work for the actual controlled access.

I'd appreciate good recommendations so that we can properly/securely manage our keys while allowing devs and automation "Easy"(tm) ways to sign generated images when needed.

posted about 3 years ago

CC BY-SA 4.0

3y ago

ghost-in-the-zsh‭

703 reputation 8 12 88 45