More Charts: Adding TLS to Airflow

In this post, we will be adding TLS to Airflow on Azure Kubernetes Service.

This is part three of a five-part series addressing Airflow at an enterprise scale. I will update these with links as they are published.

• Airflow: Planning a Deployment
• Airflow + Helm: Simple Airflow Deployment
• More Charts: Adding TLS to Airflow

Previously, we deployed Airflow to an Azure Kubernetes Service cluster using the official Helm chart and an Azure PostgreSQL Single Server instance for the metadata database. This post will focus on configuring TLS for Airflow using Cert-Manager, LetsEncrypt and Azure DNS Zones.

Getting Started with TLS

This isn’t unique to Airflow, as we will terminate TLS at the Ingress resource but is a useful skill nonetheless. The jetpack/cert-manager chart installs a set of Custom Resource Definitions (CRDs) to our cluster. Specifically, we will use the ClusterIssuer resource to provide a TLS certificate through LetsEncrypt. This certificate is applied to the Ingress resource through the normal method. Lastly, we need an ingress controller deployed to the cluster.

Let’s start by installing an Nginx Ingress Controller:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install --set defaultBackend.enabled=true \
    --namespace airflow \
    nginx \
    nginx-ingress/nginx-ingress

Now, we are ready to install the Cert-Manager:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace ingress-basic --version v1.5.4 --set installCRDs=true 

And now we are ready to configure our ClusterIssuer manifest for LetsEncrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
  namespace: airflow
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email:<your-email>@gmail.com
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx

• server is set to the prod instance of LetsEncrypt’s ACME servers.
• email contains contact information for certificate renewal.
• privateKeySecretRef.name is the secret name to store LetsEncrypt keys.
• http.ingress.class sets the ingress class that will service this http01 solver. More on this during the Ingress configuration.

It’s Always DNS

As always, the answer to “How should I configure the DNS?” is “It depends.” That being said, I am hosting this airflow cluster on a dedicated subdomain of a domain that I own. My configuration is as follows:

• Azure DNS Zone
  • Get the external IP from the nginx-ingress-controller (or something similarly named) service.
  • Create an A record pointing the load balancer IP, in my case, to the subdomain.
• Go to your domain provider’s management portal and register the nameservers that Azure DNS Zones lists on the overview tab.

Building the Ingress Resource

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: airflow-ingress
  namespace: airflow
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: nginx
  tls:
    - hosts:
      - airflow.************.com
      secretName: airflow-tls
  rules:
  - host: airflow.**************.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: airflow-webserver
            port: 
              number: 8080

• metadata.annotations.kubernetes.io/ingress.class sets the ingress class as nginx
• metadata.annotations.cert-manager.io/cluster-issuer: letsencrypt tells Cert-Manager that it’s ClusterIssuer named letsencrypt to service any tls blocks in the Ingress.
• spec.tls[0].hosts.secretName is the standard Ingress functionality.

The real special sauce of using Cert-Manager and LetsEncrypt is that when this Ingress resource was created with those annotations, Cert-Manager requested a cert for the host and created the secret per our configured secretName.

But why does Chrome give a “Not Secure” warning?

The SSL Report looks good.

airflow.*******.com CAA 1 issue "letsencrypt"

And that’s it, Airflow is secured with TLS.

The post More Charts: Adding TLS to Airflow first appeared on Object Partners.