Cryptographic keys and PKI: key material, rotation, and what holds up in production

Most cryptography failures I see in production have nothing to do with broken algorithms. AES-256 is not the problem. RSA is not the problem. The problem is almost always the key: where it came from, where it lives, who can use it, and what happens when it has to change.

A strong cipher protecting a key that sits in a Git history is not security. A private key that has not moved in years because nobody dares touch it is not security either. Both are deferred incidents. This is the unglamorous half of cryptography, and it is the half that decides whether the encryption actually protects anything.

A key is not the same thing as key material

The word "key" usually refers to the logical object: an identifier, some metadata, a usage policy, often several versions over time. Key material is the actual secret: the raw bytes of a symmetric key, or the private exponent and parameters of an asymmetric key pair. The distinction matters because nearly every good practice here is about protecting the material while letting the logical key stay stable.

Two families to keep straight. Symmetric keys are a single shared secret, fast, used for bulk data, with AES-256-GCM the usual choice. Asymmetric keys come as a pair, a private and a public half, used for signatures, key exchange and identity, with RSA-3072 or elliptic curve options like P-256 and Ed25519 in common use.

A key is only ever as strong as two things you cannot judge from the algorithm name: the randomness used to generate it, and the protection around the material once it exists. A 256-bit key drawn from a weak random source is not a 256-bit key. Generation needs a cryptographically secure random source seeded from real entropy, which is one of the reasons keys that matter are generated inside dedicated hardware rather than in application code.

Where keys live: HSMs, KMS, and the material that should never leave

The safest place for key material is somewhere it can be used but never extracted in the clear. That is the whole premise of a hardware security module. The private key is generated inside the device, signing and decryption happen inside the device, and the material never leaves in plaintext. HSMs are validated against FIPS 140-2, or the newer FIPS 140-3, and for regulated work you will usually see Level 3 as the bar.

Cloud key management services (AWS KMS, Azure Key Vault, Google Cloud KMS) sit on HSMs and give you an API instead of a device to rack. The pattern worth internalising is envelope encryption. You do not encrypt your data directly with the key held in the service. The service holds a key-encryption key that never leaves, and you use it to wrap a locally generated data-encryption key. The data key encrypts the data; the wrapped data key is stored next to the ciphertext.

Two consequences fall out of this, and both matter for rotation: the key-encryption key can change without re-encrypting terabytes of data, and the blast radius of any single data key is limited to the data it wrapped. You can also bring your own key material, or hold the key outside the provider entirely, for cases where you cannot accept the provider generating or holding it. Useful, but every step toward holding it yourself is a step toward owning the operational burden, including being the person who has to be available when it needs rotating.

For the most sensitive keys, no single person should be able to use or export them alone. Dual control, where two people are required, and split knowledge, where no one person holds the whole secret, are old controls from the payments world and still the right ones for root material.

PKI: the trust chain, and why the root sits offline

Public key infrastructure is how you turn "this is a public key" into "this is the public key for that identity, and someone I trust vouches for it." The structure is a chain. A root certificate authority sits at the top, one or more intermediate or issuing authorities sit below it, and the end-entity certificates at the bottom are issued to your servers, services and clients. A client trusts a leaf certificate because it chains up to a root already in its trust store.

The single most important operational fact about the root is that it stays offline. It signs a small number of intermediates, then sits powered down, often on an HSM, brought out only for rare ceremonies. The intermediates do the everyday issuing. If an intermediate is compromised you revoke it and issue a new one from the root. If the root is compromised, every certificate beneath it is worthless and you are rebuilding trust from nothing. That asymmetry is the entire reason for the hierarchy.

Revocation is the part most teams underestimate. When a certificate must be killed before it expires, you publish that through a Certificate Revocation List or the Online Certificate Status Protocol, often with OCSP stapling so the server presents its own freshness proof. Revocation has always been the weak joint, because clients do not always check, and checking adds latency and a dependency. That weakness is exactly why the industry is moving to short-lived certificates, so that a certificate expiring on its own does most of the work revocation was supposed to do.

One more distinction. Public certificate authorities issue what browsers trust. Internal PKI, through AWS Private CA, Azure, or Active Directory Certificate Services, issues certificates the public internet never sees: service-to-service mTLS, internal APIs, device identity. The rules differ, and the public lifetime changes below do not bind your internal CA. They are still a good prompt to check whether anything internal is quietly issuing five-year leaf certificates.

Rotation: the discipline most teams defer

Rotation is changing the key material on a schedule, or on demand after a suspected compromise, while keeping the system working. The reasons are easy to state and easy to ignore: limit how much data any single key protects, limit how long a stolen key stays useful, recover cleanly from compromise, and stay within the cryptoperiod guidance that frameworks such as NIST SP 800-57 lay out per key type.

This is where the envelope pattern earns its place. Rotating a key-encryption key in a KMS does not re-encrypt your data. It creates a new version, starts using it to wrap new data keys, and keeps the old versions available for decryption only. Old data still decrypts under its old version; new data uses the new one. AWS KMS and Azure Key Vault both automate this on a schedule.

aws kms enable-key-rotation \
  --key-id arn:aws:kms:eu-west-2:111122223333:key/abcd... \
  --rotation-period-in-days 90

The trap is assuming "rotation enabled" means the old versions are gone. They are not, and they should not be, or you would lose access to everything encrypted under them. Rotation limits forward exposure. It does not retroactively re-protect old ciphertext unless you explicitly re-encrypt it, which is a separate, deliberate job.

Certificate rotation is a different animal, and it is about to get much more frequent. In April 2025 the CA/Browser Forum approved a phased reduction in the maximum lifetime of public TLS certificates. The cap drops from 398 days to 200 days in March 2026, to 100 days in March 2027, and to 47 days in March 2029, with the window for reusing domain validation shrinking to around 10 days by the end. The message is not subtle. Manual certificate renewal is finished. The 2026 step alone breaks an annual cadence, and by 2029 you are renewing every six to seven weeks.

The answer is automation through ACME, the protocol behind Let's Encrypt and now most issuers, with DNS-01 or HTTP-01 validation wired into your platform. Let's Encrypt has signalled even shorter certificates, around 45 days, ahead of the mandate. Internal PKI is exempt from these specific rules, but the direction everywhere is the same: shorter lifetimes, more automation, less reliance on revocation.

The gotchas that actually cause outages are worth naming plainly:

• Deleted old key versions. Removing old key-encryption key versions because "rotation was done" orphans every data key they wrapped, and the data they protected becomes unreadable.
• Pinning plus rotation. If a mobile app or service pins a specific certificate or key, rotating it breaks every pinned client until they update. Pin to a CA or a public key you control across rotations, never to a leaf you replace every six weeks.
• Expired intermediates. A leaf can still be valid while the intermediate above it has expired, and the chain fails anyway. Monitor the whole chain, not just the leaf.
• The forgotten certificate. The most common cause of certificate outages is the one nobody knew existed. You cannot rotate what you have not inventoried.

For the last one, the cheapest insurance is knowing what you actually have. A quick way to inspect a live chain:

openssl s_client -connect example.com:443 -showcerts < /dev/null

Best practices that survive contact with production

A short, opinionated list, in rough priority order.

1. Generate key material in an HSM or KMS, never in application code, and never commit it to a repository or let it pass through CI logs.
2. Separate the key-encryption key from the data keys and use envelope encryption, so rotation and blast radius stay manageable.
3. Apply least privilege to key use, not just key storage. Who can call decrypt or sign is the control that matters. Log every use through CloudTrail or Key Vault logging and alert on the unusual.
4. Automate rotation for both KMS keys and certificates. If rotation depends on a human remembering, it will eventually not happen.
5. Plan revocation and recovery before you need them. Know how you would kill a compromised key and what breaks when you do.
6. Build crypto-agility: the ability to change algorithm or key without re-architecting. This is no longer abstract, for the reason in the next point.
7. Treat the post-quantum move as inventory, not panic. In August 2024 NIST finalised its first post-quantum standards, FIPS 203, 204 and 205. The near-term risk is "harvest now, decrypt later," where an adversary stores your encrypted traffic today to break it once large quantum computers exist. The first step is not swapping algorithms; it is knowing where your long-lived secrets and long-lived encrypted data sit, so you can prioritise what to migrate first.

The part to take away

Treat keys as a lifecycle, not a setup step. Generation, storage, access, rotation, revocation, retirement: each stage is a place where the cryptography either holds or quietly fails. The algorithms are settled and rarely the issue. The discipline around the keys is where the security actually lives, and it is the part that rewards being boring and consistent far more than being clever.

If you are deciding where to start, start with an inventory. You cannot rotate, revoke, or migrate what you have not written down.