Istio + cert-manager + Let’s Encrypt 解密

原文: Istio + cert-manager + Let’s Encrypt demystified

Today the concept of service mesh is on the rise and when you try Istio, an implementation of this concept, you instantly understand why. This mesh concept shines especially when you have a microservice approach. But with all your services, sometimes you want to use a solution like Let’s Encrypt to automate certificate creation. And if you are on a Kubernetes cluster, you will end up finding a Cert-Manager project that can link your cluster to Let’s Encrypt when you need a certificate.

Now you know what to do, and when you use Istio, the documentation will help you achieve what you want. But sometime, what you understand from the web, might not explain much what really behind the curtain (this example). I hope this post to be helpful for you if you want to understand what is going on when you integrate Let’s Encrypt (via cert-manager) with Istio ingress gateway.

Here, I will speak about cert-manager then Istio for you to understand the theory. On the later part, I will give an example for you to follow. It will help you discover while building your own cluster.

Note: As Istio (even in 1.x version) and cert-manager have some moving piece, this post is about Istio 1.1.x (1.1.2 at the time of writing) and cert-manager 0.7.

Photo by marcos mayer on Unsplash

Cert-Manager

Let’s start by cert-manager. Cert-manager will interact with Let’s Encrypt server and will create a ‘secret’ in Kubernetes containing the validated certificate. To do that, it uses the ACME ( A utomatic C ertificate M anagement E nvironment) protocol which has recently been standardized as RFC 8555. This protocol defines how a Certificate Authority (CA) can automate the verification step for domain ownership. The verification in cert-manager with Let’s Encrypt issuer is either done via a DNS check or an HTTP check. Let’s quickly explain those:

DNS check

  1. Cert-manager will start the registration of the domain (I will use your.domain here) on Let’s Encrypt server. It gets a token from the Let’s Encrypt response.
  2. Cert-manager will then connect to your DNS server, and add a TXT entry on _acme-challenge.your.domain entry. This entry value will be computed from the token and your Let’s Encrypt account id.
  3. Let’s Encrypt will lookup the DNS entry and upon successful check on the value, will issue the certificate
  4. Cert-Manager will query Let’s Encrypt server to get the certificate. The certificate is stored in a secret Kubernetes object

HTTP check

  1. Cert-manager will start the registration of the domain (we will use your.domain here) on Let’s Encrypt server. It gets a token from the Let’s Encrypt response.
  2. Cert-manager will create a small server to serve a single page ( /.well-known/acme-challenge/{token} ) with a content based on the token and your Let’s Encrypt account id.
  3. Cert-manager will create an ingress Kubernetes object on the your.domain host
  4. Let’s Encrypt will query the http://your.domain/.well-known/acme-challenge/{token} URL — yes it is HTTP and not HTTPS as it is not yet valid(no certificate) — and upon successful check on the value, will issue the certificate
  5. Cert-Manager will query Let’s Encrypt server to get the certificate. The certificate is stored in a secret Kubernetes object

Istio integration

To sum up:

  • Cert-Manager create a certificate for your domain. Istio will need to use it. (well, use the secret the certificate is in)
  • If you use the DNS challenge, you need to have access to your DNS server. Sometimes this challenge is not possible (your corporate network team is not really happy to let you play with domains or you use a domain not managed by you ( partnership, …) or you use an incompatible DNS server). In this case, you need to use the HTTP challenge.
  • The HTTP challenge is done via an HTTP call, which is simpler, but Cert-manager creates an ingress object on Kubernetes to route the query. Since we use Istio and as Istio use gateway / virtualservice, it is not as simple as it looks.

The DNS challenge is much easier, but as you imagine, I wrote this post because it is not fit with my use case, and I have to go with the HTTP challenge. So here, I will only speak about this last one.

Photo by Joseph Barrientos on Unsplash

Istio

Istio v0.8 introduced gateway and virtualservice object to manage fine-grained setup compare to simple ingress object. A simple way to explain this, is to think gateway as the external access of the load-balancer (listened port, domain, …, certificate if we want TLS …). The virtualservice on its end, describe the internal rules (proxying rule like path selection, service selection, policies execution(CORS, version management, …)).

Virtual service for the HTTP challenge

As I described earlier, cert-manager will create an ingress object to let Let’s Encrypt validate the certificate request. Now, What can we do to make this work with Istio?

Well, we can use a not-well-documented feature of Istio that make it convert ingress object to virtualservice one (controller source code is here). The thing is, Istio is hardcoded to look for a gateway with the name istio-autogenerated-k8s-ingress on the istio-system namespace (or the namespace Istio is installed in). If so, ingress will simply be converted to virtualservice.

This gateway can be created during the setup via the helm values here or if you want to have your own specific (options), you can create one. If you rely on the helm setup, the gateway will be created to listen to port 80 on every host (hosts: “*”). If you set the enableHttps values to true, it will add a listener on port 443 for every host, with a certificate hardcoded in a file ( directory /etc/istio/ingress-certs/, file tls.crt and tls.key) located in the istio-ingressgateway deployment (the gateway controller). It means it is not really useful.

Note that the virtual virtualservice created by converting ingress will not be created as an object in Kubernetes, It will not show if you list the virtualservice via kubectl.

With this gateway, the HTTP challenge done by the cert-manager will be successful, and the cert-manager will be able to generate the secret containing the certificate.

Gateway for https

With the previous steps, you know how to generate a secret with a domain certificate in it. Now how could you use it on a gateway? Luckily for us, In version 1.1 of Istio, we can enable the ‘Secret Discovery Service’ (SDS) service. This can be enabled with the help of a single helm value: gateways.istio-ingressgateway.sds.enabled.

With this, we can create a gateway that references a secret and not a file. The gateway need to simply reference the secret name in the spec.servers[x].tls.credentialName key. As the spec.servers[x].tls.serverCertificate and the `spec.servers[x].tls.privateKey are mandatory fields, you will need to set a meaningless value in it like “sds”.

Do not forget that the secret containing the certificate needs to be in the same namespace as the ingress gateway controller (usually, the istio-system namespace)

That’s it!

The catch

As of now, I hope you understand what is going on in your cluster. Unfortunately, there is still a small issue. If a user simply accesses via http to your services, well it will get a 404 error. It would be great if we could redirect http call to https. And it seems easy to set up as there is an option on the gateway object to do that: We can add a “httpsRedirect” option. Nice isn’t it?

Well… no… Unfortunately, setting this option will make the cert-manager stop working as Let’s Encrypt will try to access the well-known endpoint on http, be redirected to https and fail as the https is not set properly (certificate is missing).

To do this redirect, we have to create a service that redirect http to https and create a virtualservice binded to the http gateway with a single rule: everything is send to our redirect service. The catch is you cannot let the “host” to “*”. You have to set every host you use in the “host” list. Else when cert-manager will create an ingress to validate a certificate (renew for instance) you will have a temporary failure on the redirect.

Photo by Aaron Burden on Unsplash

Let’s implements this

This part is a simple step-by-step guide of what I explained above. The goal is from an empty Kubernetes cluster to be able to have some working HTTPS service with a certificate delivered by cert-manager.

We will deploy 3 services on 2 certificates:

  • The [helloworld](https://github.com/istio/istio/tree/1.1.2/samples/helloworld) and the [httpbin](https://github.com/istio/istio/tree/1.1.2/samples/httpbin) sample on their own domain but in a single certificate
  • The Istio [bookinfo](https://github.com/istio/istio/tree/1.1.2/samples/bookinfo/platform/kube) sample on its own domain, with its own certificate

We can enable in the Istio helm chart some dependencies or pre-configured objects. The goal here is to not use them to be able to show every single detail.

The required tools are:

  • A Kubernetes cluster :smiley: — But one that can allocate an external IP on the load-balancers. Also, we will deploy some applications and CPU and memory limits are set on some component. So you will need a pool of servers that give you around 6CPU and 36Gb of memory.
  • Kubectl
  • Helm

Tools setup

Let’s start with basic stuff, connect to your Kubernetes cluster with kubectl and check that it is the correct one! disaster occurs so quickly :wink:

We will use Helm, so you need to initialize it. Helm can be used as a simple template engine (client side) or as release management (a server in the cluster, the Helm CLI will interact with it). Here we will use the release management way.

Modern Kubernetes clusters have RBAC enabled. To make the Helm server part — named tiller — work, you need to create a service account that tiller will use to do its work. As we are not really ‘production’ grade, you can set an account with cluster-wide permission. Tiller will be deployed in the namespace kube-system, so you need to put the service account there:

$ cat <<EOF  kubectl apply -f -

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
---
EOF

Now you can initialize the helm server part. This will create a service and a deployment named tiller-deploy. Helm will use this to manage releases:

$ helm init --service-account tiller --history-max 200

When the tiller-deploy pod is up on the kube-system namespace, that’s it, Helm is initialized. We can now setup Istio.

Istio setup

You can basically follow the official guide, but we will use the distributed charts and, on setup, we will set some specifics values.

  1. Add the Istio helm charts repository and update your repository’s local cache. As of today, each Istio version has its own repository. On the URL, you can update the version if you want.
$ helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.1.2/charts/
$ helm repo update
  1. Install the Istio CRD from the istio-init chart.
$ helm install istio.io/istio-init --name istio-init --namespace istio-system
  1. It is time to setup Istio itself. As stated above, you need to enable SDS. Here we will simply use the Istio’s default values plus the set a boolean to enable SDS
$ helm install istio.io/istio \       --name istio \       --namespace istio-system \       --set gateways.istio-ingressgateway.sds.enabled=true

That’s it, you wait for the Istio pods to be ready and you’re good!

Cert-manager

Next step is to set up the cert-manager. It is well described here: https://docs.cert-manager.io/en/latest/getting-started/install.html#installing-with-helm Here I will copy & paste the command to run:

  • Install the CustomResourceDefinition resources separately
$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml
  • Create the namespace for cert-manager
$ kubectl create namespace cert-manager
  • Label the cert-manager namespace to disable resource validation
$ kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
  • Add the Jetstack Helm repository
$ helm repo add jetstack [https://charts.jetstack.io](https://charts.jetstack.io/)
  • Update your local Helm chart repository cache
$ helm repo update
  • Install the cert-manager Helm chart
$ helm install \
--name cert-manager \
--namespace cert-manager \
--version v0.7.0 \
jetstack/cert-manager

Domains names

As stated before, we will create 3 domains, but with 2 certificates. Here we will simply use xip.io domain. With the Let’s Encrypt rate limitation (50 certificates per week) it is virtually impossible to have a real certificate on the production. Here, I will use the staging Let’s Encrypt server, it means the certificates will not be valid. The goal here is to validate the process. If you want, you can adapt your domain name to use the production Let’s Encrypt server.

Let’s get your external IP from the Istio ingress gateway:

$ kubectl get svc -n istio-system istio-ingressgateway

The return is

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.19.240.25 a.b.c.d 80:31380/TCP,...

The external IP is a.b.c.d. From that IP, you can define 3 domains name:

If you want to use your own domains, you can create them on your DNS server. Note that those domains need to be public! (Let’s Encrypt need to resolve them)

Prerequisite for certificate generation

Earlier, I describe the way cert-manager will create certificates and what is needed on the Istio side: an issuer for cert-manager, and a gateway that will transform ingress object.

The issuer part is described here. As Istio need to have certificates in its namespace (istio-system) you can create a simple issuer in the Istio namespace and not a cluster-wide one. For the configuration, we set the challenge type to http and set Let’s Encrypt server to staging.

$ cat <<EOF  kubectl apply -f -

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: istio-system
spec:
  acme:
    email: your@email.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: example-issuer-account-key
    http01: {}
---
EOF

After a small amount of time, you should have an issuer up and running:

$ kubectl describe issuer -n istio-system...
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready

As we know this issuer will create ingress object, you need to create the Istio gateway that will do the transform. It is a simple gateway with a specific name (“istio-autogenerated-k8s-ingress”) listening everything on port 80:

$ cat <<EOF  kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: istio-system
  labels:
    app: ingressgateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      protocol: HTTP2
      name: http
    hosts:
    - "*"
---
EOF

All is ready to create some certificates!

Generating certificates

Generating certificate is now as simple as creating an object certificate in the istio-system namespace.

Here we will create 2 certificates: 1 multi-domains, and 1 single domain to show you what is possible to do.

The certificate object needs to reference the issuer (“letsencrypt-staging” created before) and the ingress controller (Istio look for “istio” ingress class so let use that)

You will also need to define a secretName to store the pending/validated certificate.

For the multi-domains certificate, we will use the secret named test-certificate:

cat <<EOF
kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: test-certificate
  namespace: istio-system
spec:
  secretName: test-certificate
  issuerRef:
    name: letsencrypt-staging
  commonName: hello.a.b.c.d.xip.io
  dnsNames:
  - hello.a.b.c.d.xip.io
  - httpbin.a.b.c.d.xip.io
  acme:
    config:
    - http01:
        ingressClass: istio
      domains:
      - hello.a.b.c.d.xip.io
      - httpbin.a.b.c.d.xip.io
---
EOF

and for the book info sample, we will use the book-certificate:

cat <<EOF
kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: book-certificate
  namespace: istio-system
spec:
  secretName: book-certificate
  issuerRef:
    name: letsencrypt-staging
  commonName: book.a.b.c.d.xip.io
  dnsNames:
  - book.a.b.c.d.xip.io
  acme:
    config:
    - http01:
        ingressClass: istio
      domains:
      - book.a.b.c.d.xip.io
---
EOF

To check when the certificate will be ready, you can describe the certificates:

$ kubectl describe certificate -n istio-system
...
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
...

Done, certificates are generated!

Creating a gateway for https

Now that your certificates are there, you can create your Istio gateway. As a gateway can only take one certificate, you will need to create 2 gateways. We will create them in the Istio namespace as the configuration state it is the best practice to keep the gateway controller and the gateway in the same namespace.

Let’s create a test-gateway with the test-certificate:

cat <<EOF
kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: istio-system
  labels:
    app: ingressgateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      protocol: HTTPS
      name: https-default
    tls:
      mode: SIMPLE
      serverCertificate: "sds"
      privateKey: "sds"
      credentialName: "test-certificate"
    hosts:
    - "*"
---
EOF

And the second gateway for the book info. Here you have to set the certificate and set a ‘hosts’ so that Istio will be able to use the specific gateway from the domain name:

$ cat <<EOF
kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: book-gateway
  namespace: istio-system
  labels:
    app: ingressgateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      protocol: HTTPS
      name: https-default
    tls:
      mode: SIMPLE
      serverCertificate: "sds"
      privateKey: "sds"
      credentialName: "book-certificate"
    hosts:
    - "book.a.b.c.d.xip.io"
---
EOF

You can use openssl client to check if everything is good:

$ openssl s_client -connect hello.a.b.c.s.xip.io:443
...
---
Certificate chain
 0 s:CN = hello.a.b.c.d.xip.io
   i:CN = Fake LE Intermediate X1
 1 s:CN = Fake LE Intermediate X1
   i:CN = Fake LE Root X1
---
...
$ openssl s_client -connect book.a.b.c.s.xip.io:443
...
---
Certificate chain
 0 s:CN = book.a.b.c.d.xip.io
   i:CN = Fake LE Intermediate X1
 1 s:CN = Fake LE Intermediate X1
   i:CN = Fake LE Root X1
---
...

Now you can deploy the apps.

Deploying sample — overview

For each application, we will have the same process:

  • Creating a namespace for the application
  • Set the Istio label on the namespace to put your application in the mesh
  • Deploy the application (deployment and service object)
  • Deploy the virtualservice to bind your service to a gateway.

Deploying helloworld sample

As the overview state, you create some objects:

$ kubectl create ns hello
namespace/hello created
$ kubectl label ns hello istio-injection=enabled
namespace/hello labeled
$ kubectl apply -n hello -f https://raw.githubusercontent.com/istio/istio/1.1.2/samples/helloworld/helloworld.yamlservice/helloworld created
deployment.extensions/helloworld-v1 created
deployment.extensions/helloworld-v2 created
$ cat <<EOF
kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld
  namespace: hello
spec:
  hosts:
  - "hello.a.b.c.d.xip.io"
  gateways:
  - istio-system/test-gateway
  http:
  - match:
    - uri:
        exact: /hello
    route:
    - destination:
        host: helloworld
        port:
          number: 5000
---
EOF

You can access the https://hello.a.b.c.d.xip.io/hello endpoint (validate the invalid certificate if you used staging Let’s Encrypt). One down, two to go!

Deploying httpbin sample

You can use the same process as helloworld:

$ kubectl create ns httpbin
namespace/httpbin created
$ kubectl label ns httpbin istio-injection=enabled
namespace/httpbin labeled
$ kubectl apply -n httpbin -f https://raw.githubusercontent.com/istio/istio/1.1.2/samples/httpbin/httpbin.yamlservice/httpbin created
deployment.extensions/httpbin created
$ cat <<EOF
kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: httpbin
spec:
  hosts:
  - "httpbin.a.b.c.d.xip.io"
  gateways:
  - istio-system/test-gateway
  http:
  - match:
    route:
    - destination:
        host: httpbin
        port:
          number: 8000
---
EOF

You can check it is working: https://httpbin.a.b.c.d.xip.io/status/418 it should show you a teapot.

Deploying the book info sample

The change from the previous deployment is the use of the other gateway to use the correct certificate.

$ kubectl create ns book
namespace/book created
$ kubectl label ns book istio-injection=enabled
namespace/book labeled
$ kubectl apply -n book -f https://raw.githubusercontent.com/istio/istio/1.1.2/samples/bookinfo/platform/kube/bookinfo.yamlservice/details created
deployment.extensions/details-v1 created
service/ratings created
deployment.extensions/ratings-v1 created
service/reviews created
deployment.extensions/reviews-v1 created
deployment.extensions/reviews-v2 created
deployment.extensions/reviews-v3 created
service/productpage created
deployment.extensions/productpage-v1 created
$ cat <<EOF
kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
  namespace: book
spec:
  hosts:
  - "book.a.b.c.d.xip.io"
  gateways:
  - istio-system/book-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
---
EOF

That’s it. You can test https://book.a.b.c.d.xip.io/productpage

HTTPS redirect

Your services are working. Yeahhhhh!!! except for the first guy that will try to access those services. He will tell you that he gets a 404 status.

Yes, he did try to access with http and not https… Why bothering put ‘https://’ as it is almost automatic everywhere (The key word is ‘almost’ this example demonstrate why;) )

As stated before, we cannot set the automatic redirect to the http gateway. You need to create a basic redirect. Here, I choose to deploy a simple nginx server to do that.

$ kubectl create ns redirect
namespace/redirect created
$ kubectl label ns redirect istio-injection=enabled
namespace/redirect labeled
$ cat <<EOF
kubectl apply -n redirect -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: 
    server {
      listen 80 default_server;
      server_name _;
      return 301  https://$host$request_uri;
    }
---
apiVersion: v1
kind: Service
metadata:
  name: redirect
  labels:
    app: redirect
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: redirect
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redirect
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redirect
  template:
    metadata:
      labels:
        app: redirect
    spec:
      containers:
      - name: redirect
        image: nginx:stable
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /etc/nginx/conf.d
          name: config
      volumes:
      - name: config
        configMap:
          name: nginx-config
---
EOF

The last step is to bind the service to the HTTP gateway. I set all the hosts I use as I want to be sure that, at certificate renew time, I will not have some break on the redirect.

$ cat <<EOF
kubectl apply -n redirect -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: redirect
spec:
  hosts:
  - "book.a.b.c.d.xip.io"
  - "test.a.b.c.d.xip.io"
  - "httpbin.a.b.c.d.xip.io"
  gateways:
  - istio-system/istio-autogenerated-k8s-ingress
  http:
  - route:
    - destination:
        host: redirect
        port:
          number: 80
---
EOF

Using an URL like http://book.a.b.c.d.xip.io/productpage now redirect to https.

Photo by Tincho Franco on Unsplash

Conclusion

I hope with this post and especially with the example, you now understand the small thing that makes all this work: The hardcoded magic gateway on Istio, The not-default option for SDS, the small tricks (well if you want to use them) for instance the one to redirect to https endpoint. I hope you learned one thing or two, and that makes your day better.

I will end this post by thanking you, reader, for reading this.