add readme

examples/certificate/README.md

@@ -0,0 +1,68 @@
+# Certificates
+
+## Refreshing cert-manager issuer
+
+The expectations are that everything that is normally done in terraform scripts will be updated and then the terraform scripts applied.
+
+* Terraform will update the clusterissuer for the EKS cluster with an intermediate CA generated from the new Windows CA.
+* Terraform will update Certificate objects with new fields and parameters as desired, if they are configured through a terraform resource
+
+After the Terraform updates are applied, there are two things that need to happen:
+
+* Every Certificate is associated with an issuer and has an associated `secretName` specified.  When these secrets are deleted,
+cert-manager will detect that and use the (possibly updated) Certificate to request new certificates from the given issuer
+and generate a new ssl cert to be stored in the secret, automatically regenerating the secret.
+* Any pods in the system that directly mount the secrets into the pod *may* have issues with the certificate change.
+To ensure that the change to the certificate is honored, the best course of action is to delete the pod and allow it to
+restart.  I have confirmed that istio-ingresgateway handles the updates directly, and does not need to be restarted for the changed certs to take effect.
+
+
+[Here](refresh-certs.sh) is a bash script that will do this a bit slowly to minimize disruptions to the cluster.
+It processes one secret at a time, and waits after deleting the secret before continuing.  Similarly, it deletes
+one pod at a time and waits for it to become ready before going on to additional pods.  This should minimize any
+disruptions, as this will take one kafka broker at a time and reset it with new certs. 
+
+
+Note that when running the script, it's normal to see errors while things are being recreated.  Here is an example of a run:
+
+```console
+% ./refresh-certs paradymelab-cluster-issuer
+name:  admin-cert; namespace:  istio-system
+secret "admin-cert" deleted
+Error from server (NotFound): secrets "admin-cert" not found
+NAME         TYPE                DATA   AGE
+admin-cert   kubernetes.io/tls   2      9s
+name:  argocd-cert; namespace:  istio-system
+secret "argocd-cert" deleted
+Error from server (NotFound): secrets "argocd-cert" not found
+NAME          TYPE                DATA   AGE
+argocd-cert   kubernetes.io/tls   2      9s
+name:  jenkins-cert; namespace:  istio-system
+secret "jenkins-cert" deleted
+Error from server (NotFound): secrets "jenkins-cert" not found
+NAME           TYPE                DATA   AGE
+jenkins-cert   kubernetes.io/tls   2      9s
+name:  keycloak-cert; namespace:  istio-system
+secret "keycloak-cert" deleted
+Error from server (NotFound): secrets "keycloak-cert" not found
+NAME            TYPE                DATA   AGE
+keycloak-cert   kubernetes.io/tls   2      9s
+name:  sonar-cert; namespace:  istio-system
+secret "sonar-cert" deleted
+Error from server (NotFound): secrets "sonar-cert" not found
+NAME         TYPE                DATA   AGE
+sonar-cert   kubernetes.io/tls   2      9s
+name:  ztc-cert; namespace:  istio-system
+secret "ztc-cert" deleted
+Error from server (NotFound): secrets "ztc-cert" not found
+NAME       TYPE                DATA   AGE
+ztc-cert   kubernetes.io/tls   2      9s
+name:  kafka-external-cert; namespace:  kafka
+secret "kafka-external-cert" deleted
+Error from server (NotFound): secrets "kafka-external-cert" not found
+NAME                  TYPE                DATA   AGE
+kafka-external-cert   kubernetes.io/tls   3      9s
+pod "kafka-kafka-0" deleted
+pod "kafka-kafka-1" deleted
+pod "kafka-kafka-2" deleted
+```