In 1.5.5, we have reverted a change that caused a regression in the ACME Issuer.
Before 1.5.4, the Ingress created by cert-manager while solving an HTTP-01 challenge contained the
apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata:annotations:kubernetes.io/ingress.class: istio # The `class` present on the Issuer.
In 1.5.4, the Ingress does not contain the annotation anymore. Instead, cert-manager uses the
apiVersion: networking.k8s.io/v1kind: Ingressspec:ingressClassName: istio # 🔥 Breaking change!
This broke many users that either don't use an Ingress controller that supports the field (such as ingress-gce and Azure AGIC), as well as people who did not need to create an IngressClass previously (such as with Istio and Traefik).
The regression is present in cert-manager 1.5.4, 1.6.0, 1.6.1. It is only present on Kubernetes 1.19+ and only appears when using an Issuer or ClusterIssuer with an ACME HTTP-01 solver configured.
In 1.5.5, we have restored the original behavior which is to use the annotation. We will also backport this fix to 1.5.5 and 1.6.2, allowing people to upgrade safely.
Most people won't have any trouble upgrading from a version that contains the regression to 1.7.0, 1.6.2 or 1.5.5. If you are using Gloo, Contour, Skipper, or kube-ingress-aws-controller, you shouldn't have any issues. If you use the default "class" (e.g.,
istio for Istio) for Traefik, Istio, Ambassador, or ingress-nginx, then these should also continue to work without issue.
If you are using Traefik, Istio, Ambassador, or ingress-nginx and you are using a non-default value for the class (e.g.,
istio-internal), or if you experience any issues with your HTTP-01 challenges please read the notes on Ingress v1 compatibility.
- Fixed a regression where cert-manager was creating Ingresses using the field
ingressClassNameinstead of the annotation
kubernetes.io/ingress.class. More details about this regression are available in the 1.7 release notes. (#4783, @maelvls)
- cert-manager now does one call to the ACME API instead of two when an Order fails. This belongs to the effort towards mitigating the high load that cert-manager deployments have on the Let's Encrypt API (#4618, @irbekrm)
- Fixed a bug that caused cert-manager to panic when the Vault Issuer failed to reach the health endpoint. (#4476, @JoshVanL)
- Helm chart: the post-install hook
startupapicheckis now compatible with the PodSecurityPolicy resource. (#4432, @ndegory)
- Helm chart: the post-install hook
startupapichecknow deletes any post-install hook resources left after a previous failed install allowing
helm installto be re-run after a failed attempt. (#4435, @wallrj)
- Fix a bug where a Certificate may not get renewed when the issued Certificate has a one-second skew between
spec.durationis not used. This one-second skew can be observed on certificates issued with Let's Encrypt and caused a mismatch in time precision between the time stored in
status.renewalTimeand the time internally computed by cert-manager. (#4403, @irbekrm). Thanks to @mfmbarros for help with debugging the issue!
- Fix a regression introduced in v1.5.0 where the Ingress created for solving HTTP-01 challenges was created with
pathType: Exactinstead of
pathType: ImplementationSpecific. (#4385, @jakexks)
- Fixed the HTTP-01 solver creating ClusterIP instead of NodePort services by default. (#4394, @jakexks)
- Helm chart and static manifest: the pointless
statusfield is now stripped from the CRD manifests. (#4387, @irbekrm)
The CRDs for the cert-manager v1beta1 API were mistakenly changed in cert-manager v1.5.0. If you installed the CRDs for v1.5.0, you should upgrade your CRDs to v1.5.1.
The only affected API version is v1beta1, so if you're using the latest version - v1 - you won't be affected by the CRD changes. It's worth upgrading to v1 in any case, since v1alpha2, v1alpha3 and v1beta1 are all deprecated and will be removed in a future release.
The recent Kubernetes 1.22 release has removed a number of deprecated APIs. You can read the official blog post Kubernetes API and Feature Removals In 1.22 to learn more about it. If you intend to upgrade to Kubernetes 1.22, you must upgrade to cert-manager 1.5.
To keep compatibility with older Kubernetes versions (down to 1.16),
cert-manager 1.5 is now compatible with both Ingress
cert-manager will default to using
v1 Ingress, and fall back to
v1 is not available.
Please read the Ingress class compatibility notes to see if your Ingress controller has any known issues.
Additionally, the cert-manager API versions
are now deprecated, and will be removed in cert-manager 1.7. Please change all
your YAML manifests that use a deprecated API version to use
cert-manager.io/v1 instead, and re-apply them.
Features that we are currently working on are included in cert-manager releases but disabled by default, as they are likely to change in future. If any of them look interesting to you, please try them out and report bugs or quirks in a GitHub Issue.
We have seen many requests from users to support different ways of routing HTTP traffic into their clusters for solving ACME HTTP-01 challenges. As the cloud-native ecosystem has so many different ingress implementations, we searched for a solution that would avoid having to add individual support for every kind of virtual service to the cert-manager API, and settled on the sig-network Gateway API.
The Gateway API project aims to provide a universal API for modeling service networking in Kubernetes, and while it is still in its alpha stages is already gaining wide adoption. By supporting the Gateway API HTTPRoute resource, we hope that anyone using Ambassador, Contour, Gloo, HAProxy, Istio, Kong or Traefik will be able to solve HTTP-01 challenges, with more implementations coming in future.
To go along with the HTTPRoute resource support, we have also added a gateway-shim controller that will allow users to annotate their Gateway resources to get a cert-manager Certificate automatically created, similar to the current ingress-shim functionality.
CertificateSigningRequest is a built-in Kubernetes resource that was originally aimed at requesting X.509 client certificates and serving certificates for Kubernetes components such as kubelet.
We have seen a rising interest in using the CertificateSigningRequest (CSR) resource as a way to provision workload certificates in service meshes such as Istio and its Istio Custom CA Integration using Kubernetes CSR. For that purpose, the CSR resource needs to be integrated with existing signers such as HashiCorp Vault or Venafi TPP.
Back in cert-manager 1.4, the CA Issuer became the first built-in cert-manager issuer to support signing CertificateSigningRequest resources. In 1.5, we extended the support to all existing Issuers.
The support for signing CSR resources is still experimental and requires to be explicitly enabled. If you are interested, please take a look at the Kubernetes CertificateSigningRequest documentation on the cert-manager website.
To help you try the CSR support, you may want to try a new command that was added to the kubectl plugin. It allows you to create a CSR resource out of a cert-manager Certificate manifest:
kubectl cert-manager x create certificatesigningrequest example-csr certificate.yaml
Finally, we decided to remove the annotation
that was added to the CertificateSigningRequest resource after being signed by
cert-manager. This annotation was introduced in cert-manager 1.4 and will no
longer be set on CertificateSigningRequest resources. We removed this annotation
due to a technical limitation related to the fact that Kubernetes resources have
a status subresource that is separate from the main resource.
cert-manager comes with a kubectl plugin,
kubectl cert-manager, that comes in
handy for checking the status of your cert-manager Certificate resources.
In 1.5, a new experimental command for installing cert-manager has been added. Under the hood, it uses the cert-manager Helm chart. This means that all helm templating options are also supported by this install command:
kubectl cert-manager x install \--set prometheus.enabled=false \ # Example: disabling prometheus using a Helm parameter--set webhook.timeoutSeconds=4s # Example: changing the wehbook timeout using a Helm parameter
An interesting feature that comes when using with this new
install command is
that it installs the CRDs in a way that prevents
helm uninstall from deleting
the CRDs while uninstalling cert-manager.
The plugin is now capable of determining when your cert-manager deployment is ready to be used:
kubectl cert-manager check api
The plugin also learned how to discover the version of cert-manager running on your
cluster, similar to
kubectl cert-manager version
To install the plugin, check out the Kubectl plugin page on the cert-manager website.
While installing cert-manager using Helm, you might have noticed that the
--wait flag does not wait until cert-manager is fully functional.
With 1.5, the
--wait flag now works as you would expect. The Helm chart now
comes with a small startup job that waits until the cert-manager API becomes
Implemented in the cert-manager PR #4234 by Tim.
cert-manager now allows you to add custom annotations and labels to the Secret
containing the TLS key pair using the new Certificate field
This is useful when using third-party solutions such as
kubed to copy Secret resources to multiple
secretTemplate is synced to the Secret when the Certificate is
created or renewed.
Here is an example of Certificate using the
apiVersion: cert-manager.io/v1kind: Certificatespec:secretTemplate:annotations:my-secret-annotation: "foo"labels:my-secret-label: bar
Note that labels and annotations can only be added or replaced, but not removed. Removing any labels or annotations from the template or removing the template itself will have no effect.
Along with the ability to set your own annotations and labels on Secret resources created by cert-manager, you can also tell cert-manager which annotations should be copied from Certificate resources to the generated CertificateRequest resources. By default, cert-manager will skip copying the annotations with the following prefixes:
If you wish to keep the old behavior and allow all annotations to be copied, you
can pass the flag
--copied-annotations=* to the cert-manager controller.
Thanks again to all open-source contributors with commits in this release:
Also thanks to coderanger for helping people
out on the Slack
#cert-manager channel; it's a huge help and much appreciated.
- cert-manager now supports using Ed25519 private keys and signatures for Certificates. Implemented in the cert-manager PR #4079.
- cert-manager now emits an event when a CertificateSigningRequest resource has not yet been approved. Without this event, the user would never know that cert-manager is waiting for the approval of the CertificateSigningRequest resource. Implemented in the cert-manager PR #4229.
- cert-manager now only supports the version
ConversionReviewVersionresources, both available since Kubernetes 1.16. The
v1beta1version is no longer supported by cert-manager. This change was implemented in the cert-manager PRs #4254 and #4253.
- cert-manager now restarts more quickly by clearing the leader election before shutting down. Also, upon shutdown, the controller loops now cleanly stop, which allows all in-flight reconciliation functions to finish before exiting. Implemented in the cert-manager PR #4243.
- Metrics: a new metric, named
clock_time_secondswas added; this metric allows for monitoring systems that do not have a built-in time function (e.g. DataDog) to calculate the number of seconds until a certificate expires by subtracting this metric from the existing
certificate_expiration_timestampmetrics. Implemented in the cert-manager PR #4105.
- Helm chart: the Prometheus scraping service port now has a name. Implemented in the cert-manager PR #4072.
- Helm chart: you can now configure the labels for the cert-manager-webhook
service using the Helm value
webhook.serviceLabels. Implemented in the cert-manager PR #4260.
- Security: cert-manager now times out after 10 second when performing the self-check while solving HTTP-01 challenges. Fixed in the cert-manager PR #4311.
- Cloudflare: Refactored DNS01 challenge to use API for finding the nearest Zone (fixing potential DNS issues) (#4147, @thiscantbeserious)
- Fix a bug where failed CertificateRequest resources were not retried (#4130, @irbekrm)
- Fix a regression that would lead to a Certificate becoming "Failed" when the issued X.509 certificate's subject DN would be equal to the issuer's subject DN. Fixed in the cert-manager PR #4237.
- Fix a regression where the
tls.crtcertificate chain would unexpectedly not contain an intermediate CA certificate when no root CA is available in the CA chain returned by the issuer. This bug affected the Vault Issuer, Venafi Issuer and CA Issuer. This bug was fixed in the cert-manager PR #4261.
- Fix a goroutine leak that was causing the controller's memory usage to grow with time. Fixed in the cert-manager PR #4233.
- Fix a race condition introduced in cert-manager 0.15 that would crash cert-manager for clusters with a large number of certificates. Fixed in the cert-manager PR #4231.
- Fix a bug where the default renewal duration of certificate, set to 30 days,
would clash with the duration of certificates issued by the Vault Issuer. All
Certificate resources are now renewed 2/3 through the duration unless a custom
renew period is specified by setting
renewBeforeon the Certificate. Fixed in the cert-manager PR #4092.
- The cert-manager binaries, including the kubectl plugin, now exit with the correct exit code on SIGINT (Ctrl+C) and SIGTERM events. More specifically, when one of these events is caught, cert-manager will exit with the code 128 + signal number. Fixed in #4230.
- The static manifests available on the GitHub Releases page now contain a
app.kubernetes.io/version: v1.5.0. We also removed the Helm-specific labels from the static manifests. Fixed in the cert-manager PR #4190.