End-to-end encryption using EnRoute and Linkerd

1. Introduction

This document describes how easily EnRoute OneStep integrates with Linkerd service-mesh. LinkerD is a CNCF graduated service mesh project which uses light-weight proxies to setup the mesh inside Kubernetes.

EnRoute integration with Linkerd can be achieved in One Step. Integrating EnRoute with Linkerd involves setting one flag

At a high level End-To-End encryption includes -

  • Encryption from Client to EnRoute and
  • Encryption from EnRoute to mTLS inside the Mesh

We mesh the workload and EnRoute by injecting linkerd proxy into their pods and then setup the client certificate.

  • We start by setting up the cluster, installing linkerd, adding an example workload emojivoto and meshing this workload by injecting linkerd

  • Next we install EnRoute, expose the emojivoto application without TLS, inject linkerd proxy in the EnRoute pod to add it to the mesh

  • As the last step, we use JetStack cert manager to generate, sign and install a let’s Encrypt certificate for the emojivoto app

We trace through each of the above steps while monitoring the cluster for linkerd proxy injection.

We also verify some of the above steps on the Bouyant cloud (with it’s free tier), that shows the above steps in different UI screens

2. Pre-requisites

This setup needs an Kubernetes cluster. We have a cluster setup with two nodes -

Check Kubernetes Cluster
kubectl get nodes
NAME                   STATUS   ROLES    AGE   VERSION
pool-twnynczzy-u9k34   Ready    <none>   8d    v1.21.10
pool-twnynczzy-u9k3i   Ready    <none>   8d    v1.21.10
Install and check Linkerd installation

The installation steps for linkerd are covered in the getting started guide on the linkerd website, they aren’t covered here. Once linkerd is installed, we can verify it’s installation.

Check linkerd is installed OK

linkerd check 
Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

...

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

Status check results are √

3. Install EnRoute and emojivoto workload

Install example workload emojivoto
curl -fsL https://run.linkerd.io/emojivoto.yml | kubectl apply -f -
Linkerd core checks
namespace/emojivoto created
serviceaccount/emoji created
serviceaccount/voting created
serviceaccount/web created
service/emoji-svc created
service/voting-svc created
service/web-svc created
deployment.apps/emoji created
deployment.apps/vote-bot created
deployment.apps/voting created
deployment.apps/web created
Install EnRoute OneStep
Add helm repos
helm repo add saaras https://charts.getenroute.io
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm search repo
Use helm to install EnRoute OneStep
helm install enroute-linkerd-meshed saaras/enroute \
	--set enrouteService.rbac.create=true \
	--create-namespace \
	--namespace demo
NAME: enroute-linkerd-meshed
LAST DEPLOYED: Mon Mar 21 03:19:46 2022
NAMESPACE: demo
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
𝙴𝚗𝚁𝚘𝚞𝚝𝚎 Ingress API Gateway Community Edition Installed!
-------------------------------------------------------
Request a free evaluation license for enterprise version
by sending an email to contact@saaras.io

Slack Channel         - https://slack.saaras.io
Getting Started Guide - https://getenroute.io/docs/getting-started-enroute-ingress-controller/
EnRoute Features      - https://getenroute.io/features/

4. Configure EnRoute for Linkerd

Meshing EnRoute with linkerd involves only setting one flag linkerd_enabled on the GlobalConfig

cat <<EOF | kubectl apply -f -
apiVersion: enroute.saaras.io/v1
kind: GlobalConfig
metadata:
  labels:
    app: web
  name: enable-linkerd
  namespace: default
spec:
  name: linkerd-global-config
  type: globalconfig_globals
  config: |
        {
          "linkerd_enabled": true
        }
EOF
globalconfig.enroute.saaras.io/enable-linkerd configured

This gets EnRoute ready to run along with Linkerd in meshed mode. This is the only configuration required to run EnRoute with linkerd with end-to-end encryption globally for all services.

By default EnRoute integration adds headers to help linkerd with routing choices in addition to letting Linkerd select endpoints for service. To change this default behavior and further customize it, the flags linkerd_header_disabled and linkerd_servicemode_disabled can be set to either disable the header or have EnRoute perform endpoint selection.

---
apiVersion: enroute.saaras.io/v1
kind: GlobalConfig
metadata:
  labels:
    app: web
  name: enable-linkerd
  namespace: default
spec:
  name: linkerd-global-config
  type: globalconfig_globals
  config: |
        {
          "linkerd_enabled": true,
          "linkerd_header_disabled": false,
          "linkerd_servicemode_disabled": false
        }

5. Mesh EnRoute and Emojivoto workload - encryption inside the cluster

Inject linkerd in emojivoto workload

The example workload wasn’t meshed into the linkerd mesh. To mesh it, we need to provide an annotation to the deployment to inject linkerd

kubectl get -n emojivoto deploy -o yaml | linkerd inject - | kubectl apply -f -
deployment "emoji" injected
deployment "vote-bot" injected
deployment "voting" injected
deployment "web" injected

deployment.apps/emoji configured
deployment.apps/vote-bot configured
deployment.apps/voting configured
deployment.apps/web configured

This installs linkerd proxy along with the example application pods

Inject linkerd in the EnRoute pod

Once EnRoute is ready to work with linkerd, we inject the linkerd proxy in the EnRoute pod

kubectl get -n demo deploy -o yaml | linkerd inject - | kubectl apply -f -
deployment "enroute-linkerd-meshed" injected
deployment.apps/enroute-linkerd-meshed configured

We use the Buoyant cloud to trace the status of linkerd injection and mesh formation. The picture below shows how the EnRoute deployment was mutated to add the linkerd proxy, it is captured a couple of screens on the bouyant cloud

Deployment of EnRoute showing meshed with HTTP Metrics

The event screen also captures the linkerd proxy injection -

Events showing EnRoute LinkerD meshed

We can also use kubectl to check pods to see that linkerd got inserted in the pod

kubectl get pods -n demo -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |sort
enroute-linkerd-meshed-5d9575d575-4bskt:        cr.l5d.io/linkerd/proxy:stable-2.11.1, saarasio/enroute-gwi:latest, redis:latest, envoyproxy/envoy:v1.21.1,

6. Encryption from client to EnRoute

Install service-policy chart to expose the service externally with L7 policies
helm install emojivoto-service-policy saaras/service-policy   \
--set service.name=web-svc   \
--set service.prefix=/ \
--set service.port=80 \
--namespace emojivoto \
--set service.fqdn=enroute-linkerd-mesh.enroutedemo.com \
--set service.enableTLS=false
NAME: emojivoto-service-policy
LAST DEPLOYED: Mon Mar 21 05:04:39 2022
NAMESPACE: emojivoto
STATUS: deployed
REVISION: 1
NOTES:
------------------ Configuration Graph -----------------------------------
Filters enabled for service [web-svc]
         |
    web-svc-80-luatestfilter
    kubectl edit -n emojivoto httpfilters.enroute.saaras.io web-svc-80-luatestfilter
         |
    web-svc-80-rl2
    kubectl edit -n emojivoto routefilters.enroute.saaras.io web-svc-80-rl2 (per-route ratelimit)
    kubectl edit -n emojivoto globalconfigs.enroute.saaras.io web-svc-rl-global-config (ratelimit engine global config)

𝙴𝚗𝚁𝚘𝚞𝚝𝚎 Community Edition Service Policy
----------------------------------------
Request an evaluation license for enterprise version - contact@saaras.io

Slack Channel         - https://slack.saaras.io
Getting Started Guide - https://getenroute.io/docs/getting-started-enroute-ingress-controller/
EnRoute Features      - https://getenroute.io/features/

Send some traffic

curl -I enroute-linkerd-mesh.enroutedemo.com
HTTP/1.1 200 OK
content-type: text/html
date: Mon, 21 Mar 2022 05:09:51 GMT
content-length: 560
x-envoy-upstream-service-time: 4
vary: Accept-Encoding
lua-filter-says: Hello
server: envoy
while true; do enroute-linkerd-mesh.enroutedemo.com &> /dev/null; done

Enable end-to-end TLS by enabling TLS for downstream client connections

We will be using JetStack Cert Manager integration with EnRoute to install a certificate for example workload emojivoto. We need to setup DNS for automatic verification of certificate.

Setup DNS

We create a DNS entry for a domain to enable verification of certificate (installed using Let’s Encrypt) and install it on our example workload

Get ExternalIP

Check that the EnRoute service is installed with an External-IP

kubectl get service -n demo
NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                      AGE
enroute-linkerd-meshed   LoadBalancer   10.245.139.117   137.184.245.252   80:30695/TCP,443:31864/TCP   94m
Create DNS Entry

Create a DNS entry from domain name to the external IP. Verify the DNS is setup correctly -

ping enroute-linkerd-mesh.enroutedemo.com
PING enroute-linkerd-mesh.enroutedemo.com (137.184.245.252) 56(84) bytes of data.
64 bytes from 137.184.245.252 (137.184.245.252): icmp_seq=1 ttl=57 time=0.827 ms
^C
--- enroute-linkerd-mesh.enroutedemo.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.827/0.827/0.827/0.000 ms
Install Certificate on service
helm upgrade emojivoto-service-policy saaras/service-policy   \
--set service.name=web-svc   \
--set service.prefix=/   \
--set service.port=80   \
--namespace emojivoto   \
--set service.enableTLS=true   \
--set autoTLS.certificateCN=enroute-linkerd-mesh.enroutedemo.com    \
--set autoTLS.email=contact@example.io   \
--set autoTLS.createIssuers=true   \
--set autoTLS.enableProd=true
Release "emojivoto-service-policy" has been upgraded. Happy Helming!
NAME: emojivoto-service-policy
LAST DEPLOYED: Mon Mar 21 05:12:19 2022
NAMESPACE: emojivoto
STATUS: deployed
REVISION: 2
NOTES:
------------------ Configuration Graph -----------------------------------
Filters enabled for service [web-svc]
         |
    web-svc-80-luatestfilter
    kubectl edit -n emojivoto httpfilters.enroute.saaras.io web-svc-80-luatestfilter
         |
    web-svc-80-rl2
    kubectl edit -n emojivoto routefilters.enroute.saaras.io web-svc-80-rl2 (per-route ratelimit)
    kubectl edit -n emojivoto globalconfigs.enroute.saaras.io web-svc-rl-global-config (ratelimit engine global config)

𝙴𝚗𝚁𝚘𝚞𝚝𝚎 Community Edition Service Policy
----------------------------------------
Request an evaluation license for enterprise version - contact@saaras.io

Slack Channel         - https://slack.saaras.io
Getting Started Guide - https://getenroute.io/docs/getting-started-enroute-ingress-controller/
EnRoute Features      - https://getenroute.io/features/
Check certificates
kubectl get certificates.cert-manager.io --all-namespaces
NAMESPACE   NAME                                   READY   SECRET                                 AGE
emojivoto   enroute-linkerd-mesh.enroutedemo.com   True    enroute-linkerd-mesh.enroutedemo.com   63s
Send traffic
curl -I -v https://enroute-linkerd-mesh.enroutedemo.com


*   Trying 137.184.245.252:443...
* TCP_NODELAY set
* Connected to enroute-linkerd-mesh.enroutedemo.com (137.184.245.252) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=enroute-linkerd-mesh.enroutedemo.com
*  start date: Mar 21 04:12:46 2022 GMT
*  expire date: Jun 19 04:12:45 2022 GMT
*  subjectAltName: host "enroute-linkerd-mesh.enroutedemo.com" matched cert's "enroute-linkerd-mesh.enroutedemo.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
...
> HEAD / HTTP/2
> Host: enroute-linkerd-mesh.enroutedemo.com
> user-agent: curl/7.68.0
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
HTTP/2 200
< content-type: text/html
< date: Mon, 21 Mar 2022 05:14:01 GMT
< content-length: 560
< x-envoy-upstream-service-time: 4
< vary: Accept-Encoding
< lua-filter-says: Hello
< server: envoy
<

Let us verify End-To-End Encryption -

The browser to EnRoute connection is encrypted using the certificate we installed on emojivoto app -

SSL EmojiVoto App

We can also see the workload and EnRoute are meshed in the “Meshed” column below

emoji and EnRoute workloads meshed in the workloads screen

We can also see that the incoming traffic shows up as Application TLS

Application TLS going through EnRoute SSL

The topology screen also displays the mTLS betwen services and direction of flow of traffic

mTLS topology pic from EnRoute to Emojivoto

Conclusion

EnRoute provides an extremely easy integration with linkerd.

EnRoute integrates easily with both linkerd and Istio service mesh. The integration with istio service mesh is covered in another article.