Sealing secrets to the edge clusters was an interesting idea, but it is causing some operational problems:
- The write-only nature of the secret store makes it difficult to manage secrets. Edge secrets are generally being left deployed forever, even when they've been replaced.
- The chewing-gum-and-string deployment pathway is fine when it works but difficult to debug when it doesn't.
- The sealed secrets private key is a piece of state unique to an edge cluster. This means it is not possible to re-bootstrap an edge cluster onto new hardware and have everything come up clean.
- Attempting to re-bootstrap and fix the secrets afterwards doesn't work, because there are SealedSecret CRs deployed to the cluster repo but the sealed-secrets Helm chart hasn't deployed the CRDs yet.
They should be replaced by something else. One possibility would be Hashicorp Vault. Another would be to enhance the cluster-manager to store secret information in Secrets on the central cluster (so they are encrypted at rest) and then make the secret information available, with auth, to edge-sync so it can be synced into Secrets on the edge cluster.
Sealing secrets to the edge clusters was an interesting idea, but it is causing some operational problems:
They should be replaced by something else. One possibility would be Hashicorp Vault. Another would be to enhance the cluster-manager to store secret information in Secrets on the central cluster (so they are encrypted at rest) and then make the secret information available, with auth, to edge-sync so it can be synced into Secrets on the edge cluster.