NEW: Get project updates onTwitterandMastodon

Route53

This guide explains how to set up an Issuer, or ClusterIssuer, to use Amazon Route53 to solve DNS01 ACME challenges. It's advised you read the DNS01 Challenge Provider page first for a more general understanding of how cert-manager handles DNS01 challenges.

ℹī¸ This guide assumes that you already have a hosted zone in Route53.

📖 Read the AWS + LoadBalancer + Let's Encrypt tutorial, which contains end-to-end instructions for those who are new to cert-manager and AWS.

Set up an IAM Policy

cert-manager needs to be able to add records to Route53 in order to solve the DNS01 challenge. To enable this, create a IAM policy with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}

ℹī¸ The route53:ListHostedZonesByName statement can be removed if you specify the (optional) hostedZoneID. You can further tighten the policy by limiting the hosted zone that cert-manager has access to (e.g. arn:aws:route53:::hostedzone/DIKER8JEXAMPLE).

📖 Read about actions supported by Amazon Route 53, in the Amazon Route 53 API Reference.

📖 Learn how eksctl can automatically create the cert-manager IAM policy, if you use EKS.

Credentials

cert-manager needs an AWS access key, to authenticate to the Route53 API. An access key is defined by AWS as follows:

access key: The combination of an access key ID (for example, AKIAIOSFODNN7EXAMPLE) and a secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY). You use access keys to sign API requests that you make to AWS.

You have two options:

  1. (Best Practice) Use an IAM Role with temporary security credentials.
  2. Use an IAM User with a long-term access key.

Using an IAM Role with temporary security credentials is considered best practice because:

  1. You do not have to store the long-term access key (e.g. in a Secret)
  2. You don't have to manage access key rotation.

Using an IAM User with long-term access key is a reasonable choice if cert-manager is deployed outside AWS but the DNS zone is on Route53.

cert-manager supports multiple ways to get the access key and these can be categorized as either "ambient" or "non-ambient".

Ambient Credentials

Ambient credentials are credentials which are made available in the cert-manager controller Pod by one of the following mechanisms:

The advantage of ambient credentials is that they are easier to set up and extensively documented by Amazon AWS. The disadvantage of ambient credentials is that they are globally available to all ClusterIssuer and all Issuer resources, which means that in a multi-tenant environment, any tenant who has permission to create Issuer or ClusterIssuer may use the ambient credentials and gain the permissions granted to that account.

📖 Read AWS SDKs and Tools standardized credential providers to learn how cert-manager supports all these ambient credential sources.

⚠ī¸ By default, cert-manager will only use ambient credentials for ClusterIssuer resources, not Issuer resources.

This is to prevent unprivileged users, who have permission to create Issuer resources, from issuing certificates using credentials that cert-manager incidentally has access to. ClusterIssuer resources are cluster scoped (not namespaced) and only platform administrators should be granted permission to create them.

⚠ī¸ It is possible (but not recommended) to enable ambient authentication mechanisms for Issuer resources, by setting the --issuer-ambient-credentials flag on the cert-manager controller to true.

Here is an example of a ClusterIssuer for using Route53 ambient credentials:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    ...
    solvers:
    - dns01:
        route53: {}

ℹī¸ Regardless of which ambient mechanism you use, the route53 section is left empty, because cert-manager can find the credentials, role, and region by looking for environment variables which will be added to the cert-manager Pod.

EKS Pod Identity

EKS Pod Identity is the simplest way to use ambient credentials, if you deploy cert-manager on EKS. It is a four step process:

  1. Setup the EKS Pod Identity agent in your cluster.

  2. Assign an IAM role to the cert-manager Kubernetes service account.

  3. Restart the cert-manager Deployment so that the EKS Pod Identity Agent can inject the necessary environment variables into the Pods.

  4. Create a ClusterIssuer resource:

    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-prod
    spec:
      acme:
        solvers:
        - dns01:
            route53: {}

EKS IAM Role for Service Accounts (IRSA)

IAM Roles for Service Accounts (IRSA) is another way to use ambient credentials, if you deploy cert-manager on EKS. It is more complicated than Pod Identity and requires coordination between the Kubernetes cluster administrator and the AWS account manager. It involves annotating the cert-manager ServiceAccount in Kubernetes, and setting up an IAM role, a trust policy and a trust relationship in AWS. A mutating webhook will automatically setup a mounted service account volume in the cert-manager Pod.

  1. Create an IAM OIDC provider for your cluster

    To use IRSA with cert-manager you must first enable the feature for your cluster. Follow the official documentation.

  2. Create a trust relationship

    In this configuration an IAM role is mapped to the cert-manager ServiceAccount allowing it to authenticate with AWS. The IAM role you map to the ServiceAccount will need permissions on any and all Route53 zones cert-manager will be using. Create a trust relationship by adding the following trust policy to the IAM role:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Principal": {
            "Federated": "arn:aws:iam::<aws-account-id>:oidc-provider/oidc.eks.<aws-region>.amazonaws.com/id/<eks-hash>"
          },
          "Condition": {
            "StringEquals": {
              "oidc.eks.<aws-region>.amazonaws.com/id/<eks-hash>:sub": "system:serviceaccount:<namespace>:<service-account-name>"
            }
          }
        }
      ]
    }

    Replace the following:

    • <aws-account-id> with the AWS account ID of the EKS cluster.

    • <aws-region> with the region where the EKS cluster is located.

    • <eks-hash> with the hash in the EKS API URL; this will be a random 32 character hex string (example: 45DABD88EEE3A227AF0FA468BE4EF0B5).

    • <namespace> with the namespace where cert-manager is running.

    • <service-account-name> with the name of the ServiceAccount object created by cert-manager.

    ℹī¸ If you're following the Cross Account example, this trust policy is attached to the cert-manager role in Account X with ARN arn:aws:iam::XXXXXXXXXXX:role/cert-manager. The permissions policy is the same as above.

  3. Annotate the cert-manager ServiceAccount

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXX:role/cert-manager

    The cert-manager Helm chart provides a variable for injecting annotations into cert-manager's ServiceAccount like so:

    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXX:role/cert-manager

    ℹī¸ If you're following the Cross Account example, modify the ClusterIssuer with the role from Account Y.

  4. (optional) Update file system permissions

    đŸ“ĸ Please help us improve this documentation

    The reason for this optional step is that on EKS Fargate and on some older versions of EKS you may observe errors such as:

    • unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    • open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied

    In this case, you can change the user and group of the cert-manager process so that it is able to read the mounted ServiceAccount token.

    Read cert-manager/website#697: IRSA Needs runAsUser: 1001 and tell us whether this step is still necessary or obsolete.

    You may also need to modify the cert-manager Deployment with a different user and group, so the ServiceAccount token can be read.

    spec:
      template:
        spec:
          securityContext:
            fsGroup: 1001
            runAsUser: 1001

    The cert-manager Helm chart provides a variable for modifying cert-manager's Deployment like so:

    securityContext:
      fsGroup: 1001
      runAsUser: 1001

  5. Restart the cert-manager Deployment

    Restart the cert-manager Deployment, so that the webhook can inject the necessary volume, volumemount, and environment variables into the Pods.

  6. Create a ClusterIssuer resource

    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-prod
    spec:
      acme:
        solvers:
        - dns01:
            route53: {}

Non-ambient Credentials

Non-ambient credentials are credentials which are explicitly configured on the Issuer or ClusterIssuer resource. For example:

  • Access key Secret reference:
    where cert-manager loads a long-term access key from a Kubernetes Secret resource.
  • ServiceAccount reference:
    where cert-manager gets a ServiceAccount token (signed JWT) from the Kubernetes API server, and uses the STS AssumeRoleWithWebIdentity endpoint to exchange it for temporary AWS credentials.

The advantage of non-ambient credentials is that cert-manager can perform Route53 operations in a multi-tenant environment. Each tenant can be granted permission to create and update Issuer resources in their namespace and they can provide their own AWS credentials in their namespace.

IAM Role with dedicated Kubernetes ServiceAccount

📖 Read the AWS + LoadBalancer + Let's Encrypt tutorial to learn how to deploy cert-manager on EKS and use this authentication mechanism.

In this configuration you can reference your own ServiceAccounts in your Issuer or ClusterIssuer and cert-manager will get a ServiceAccount token from the Kubernetes API which it will send to STS in exchange for AWS temporary credentials.

The advantages of this mechanism are:

  1. Each Issuer can reference a different ServiceAccount, which means you can lock down the permissions, such that each ServiceAccount is mapped to an IAM role that only has permission to update the zones it needs (unlike Pod Identity or IRSA).
  2. This mechanism works even when cert-manager is deployed outside AWS.

Here's how to set it up:

  1. Create a ServiceAccount

    In order to reference a ServiceAccount it must first exist. Unlike normal IRSA the eks.amazonaws.com/role-arn annotation is not required.

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: <service-account-name>

  2. Create an IAM role trust policy

    For every ServiceAccount you want to use for AWS authentication you must first set up a trust policy:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Principal": {
            "Federated": "arn:aws:iam::<aws-account-id>:oidc-provider/oidc.eks.<aws-region>.amazonaws.com/id/<eks-hash>"
          },
          "Condition": {
            "StringEquals": {
              "oidc.eks.<aws-region>.amazonaws.com/id/<eks-hash>:sub": "system:serviceaccount:<namespace>:<service-account-name>"
            }
          }
        }
      ]
    }

    Replace the following:

    • <aws-account-id> with the AWS account ID of the EKS cluster.
    • <aws-region> with the region where the EKS cluster is located.
    • <eks-hash> with the hash in the EKS API URL; this will be a random 32 character hex string (example: 45DABD88EEE3A227AF0FA468BE4EF0B5).
    • <namespace> with the namespace of the ServiceAccount object.
    • <service-account-name> with the name of the ServiceAccount object.

  3. Create an RBAC Role and RoleBinding

    In order to allow cert-manager to issue a token using your ServiceAccount you must deploy some RBAC to the cluster:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: <service-account-name>-tokenrequest
      namespace: <service-account-namespace>
    rules:
      - apiGroups: ['']
        resources: ['serviceaccounts/token']
        resourceNames: ['<service-account-name>']
        verbs: ['create']
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: cert-manager-<service-account-name>-tokenrequest
      namespace: <service-account-namespace>
    subjects:
      - kind: ServiceAccount
        name: <cert-manager-service-account-name>
        namespace: <cert-manager-namespace>
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: <service-account-name>-tokenrequest

    Replace the following:

    • <service-account-name> name of the ServiceAccount object.
    • <service-account-namespace> namespace of the ServiceAccount object.
    • <cert-manager-service-account-name> name of cert-managers ServiceAccount object, as created during cert-manager installation.
    • <cert-manager-namespace> namespace that cert-manager is deployed into.

  4. Create an Issuer or ClusterIssuer

    You should be ready at this point to configure an Issuer to use the new ServiceAccount. You can see example config for this below:

    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: example
    spec:
      acme:
        ...
        solvers:
        - dns01:
            route53:
              region: us-east-1
              role: <iam-role-arn> # This must be set so cert-manager what role to attempt to authenticate with
              auth:
                kubernetes:
                  serviceAccountRef:
                    name: <service-account-name> # The name of the service account created

IAM User with long-term access key

In this mechanism, cert-manager will load the credentials from a Secret resource. If you use an Issuer resource, the Secret must be in the same namespace as the Issuer. If you use a ClusterIssuer resource, the Secret must be in the cert-manager namespace or what ever value is supplied to the --cluster-resource-namespace option of the cert-manager component.

The advantages of this mechanism are that it is simple and it works even when cert-manager is deployed outside AWS.

Here is an example configuration for a ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    ...
    solvers:
    - dns01:
        route53:
          region: eu-central-1
          accessKeyIDSecretRef:
            name: prod-route53-credentials-secret
            key: access-key-id
          secretAccessKeySecretRef:
            name: prod-route53-credentials-secret
            key: secret-access-key
          # (optional) you can also assume a role with these credentials
          role: arn:aws:iam::YYYYYYYYYYYY:role/dns-manager

Cross Account Access

Example: Account Y manages Route53 DNS Zones. Now you want cert-manager running in Account X (or many other accounts) to be able to manage records in Route53 zones hosted in Account Y.

First, create a role with the permissions policy above (let's call the role dns-manager) in Account Y, and attach a trust relationship like the one below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::XXXXXXXXXXX:role/cert-manager"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Bear in mind, that you won't be able to define this policy until cert-manager role on account Y is created. If you are setting this up using a configuration language, you may want to define principal as:

"Principal": {
        "AWS": "XXXXXXXXXXX"
      }

And restrict it, in a future step, after all the roles are created.

This allows the role cert-manager in Account X to assume the dns-manager role in Account Y to manage the Route53 DNS zones in Account Y. For more information visit the official documentation.

Second, create the cert-manager role in Account X; this will be used as a credentials source for the cert-manager pods running in Account X. Attach to the role the following permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": "arn:aws:iam::YYYYYYYYYYYY:role/dns-manager",
      "Action": "sts:AssumeRole"
    }
  ]
}

And the following trust relationship (Add AWS Services as needed):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Region

If you omit the .spec.acme.solvers.dns01.route53.region field, cert-manager will get the region from the AWS_REGION and AWS_DEFAULT_REGION environment variables if they are set in the cert-manager controller Pod.

If you use ambient credentials, the AWS_REGION and AWS_DEFAULT_REGION environment variables have a higher priority, and the .spec.acme.solvers.dns01.route53.region field will only be used if the environment variables are not set.

The .spec.acme.solvers.dns01.route53.region field is ignored if you use EKS Pod Identities, because an AWS_REGION environment variable is added to the cert-manager controller Pod by the Amazon EKS Pod Identity Agent.

The .spec.acme.solvers.dns01.route53.region field is ignored if you use IAM Roles for Service Accounts (IRSA), because an AWS_REGION environment variable is added to the cert-manager controller Pod by the Amazon EKS Pod Identity Webhook.

ℹī¸ Route53 is a global service and does not have regional endpoints, but the region is used as a hint to help compute the correct AWS credential scope and partition when it connects to Route53.

📖 Read Amazon Route 53 endpoints and quotas and Global services to learn more.

ℹī¸ STS is a regional service and cert-manager will use regional STS endpoint URLs computed from the region field or environment variables. STS is used for IRSA credentials, dedicated ServiceAccount credentials, and cross account access.

📖 Read Manage AWS STS in an AWS Region to learn about which regions support STS.

📖 Read AWS STS Regional endpoints, to learn how to configure the use of regional STS endpoints using environment variables.