End-to-end encryption using EnRoute and Linkerd
1. Introduction
This document describes how easily EnRoute OneStep integrates with Linkerd service-mesh. LinkerD is a CNCF graduated service mesh project which uses light-weight proxies to setup the mesh inside Kubernetes.
EnRoute integration with Linkerd can be achieved in One Step. Integrating EnRoute with Linkerd involves setting one flag
At a high level End-To-End encryption includes -
- Encryption from Client to EnRoute and
- Encryption from EnRoute to mTLS inside the Mesh
We mesh the workload and EnRoute by injecting linkerd proxy into their pods and then setup the client certificate.
We start by setting up the cluster, installing linkerd, adding an example workload emojivoto and meshing this workload by injecting linkerd
Next we install EnRoute, expose the emojivoto application without TLS, inject linkerd proxy in the EnRoute pod to add it to the mesh
As the last step, we use JetStack cert manager to generate, sign and install a let’s Encrypt certificate for the emojivoto app
We trace through each of the above steps while monitoring the cluster for linkerd proxy injection.
We also verify some of the above steps on the Bouyant cloud (with it’s free tier), that shows the above steps in different UI screens
2. Pre-requisites
This setup needs an Kubernetes cluster. We have a cluster setup with two nodes -
Check Kubernetes Cluster
kubectl get nodes
NAME STATUS ROLES AGE VERSION
pool-twnynczzy-u9k34 Ready <none> 8d v1.21.10
pool-twnynczzy-u9k3i Ready <none> 8d v1.21.10
Install and check Linkerd installation
The installation steps for linkerd are covered in the getting started guide on the linkerd website, they aren’t covered here. Once linkerd is installed, we can verify it’s installation.
Check linkerd is installed OK
linkerd check
Linkerd core checks
===================
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
...
linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match
Status check results are √
3. Install EnRoute and emojivoto workload
Install example workload emojivoto
curl -fsL https://run.linkerd.io/emojivoto.yml | kubectl apply -f -
Linkerd core checks
namespace/emojivoto created
serviceaccount/emoji created
serviceaccount/voting created
serviceaccount/web created
service/emoji-svc created
service/voting-svc created
service/web-svc created
deployment.apps/emoji created
deployment.apps/vote-bot created
deployment.apps/voting created
deployment.apps/web created
Install EnRoute OneStep
Add helm repos
helm repo add saaras https://charts.getenroute.io
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm search repo
Use helm to install EnRoute OneStep
helm install enroute-linkerd-meshed saaras/enroute \
--set enrouteService.rbac.create=true \
--create-namespace \
--namespace demo
NAME: enroute-linkerd-meshed
LAST DEPLOYED: Mon Mar 21 03:19:46 2022
NAMESPACE: demo
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
𝙴𝚗𝚁𝚘𝚞𝚝𝚎 Ingress API Gateway Community Edition Installed!
-------------------------------------------------------
Request a free evaluation license for enterprise version
by sending an email to contact@saaras.io
Slack Channel - https://slack.saaras.io
Getting Started Guide - https://getenroute.io/docs/getting-started-enroute-ingress-controller/
EnRoute Features - https://getenroute.io/features/
4. Configure EnRoute for Linkerd
Meshing EnRoute with linkerd involves only setting one flag linkerd_enabled
on the GlobalConfig
cat <<EOF | kubectl apply -f -
apiVersion: enroute.saaras.io/v1
kind: GlobalConfig
metadata:
labels:
app: web
name: enable-linkerd
namespace: default
spec:
name: linkerd-global-config
type: globalconfig_globals
config: |
{
"linkerd_enabled": true
}
EOF
globalconfig.enroute.saaras.io/enable-linkerd configured
This gets EnRoute ready to run along with Linkerd in meshed mode. This is the only configuration required to run EnRoute with linkerd with end-to-end encryption globally for all services.
By default EnRoute integration adds headers to help linkerd with routing choices in addition to letting Linkerd select endpoints for service. To change this default behavior and further customize it, the flags linkerd_header_disabled
and linkerd_servicemode_disabled
can be set to either disable the header or have EnRoute perform endpoint selection.
---
apiVersion: enroute.saaras.io/v1
kind: GlobalConfig
metadata:
labels:
app: web
name: enable-linkerd
namespace: default
spec:
name: linkerd-global-config
type: globalconfig_globals
config: |
{
"linkerd_enabled": true,
"linkerd_header_disabled": false,
"linkerd_servicemode_disabled": false
}
5. Mesh EnRoute and Emojivoto workload - encryption inside the cluster
Inject linkerd in emojivoto workload
The example workload wasn’t meshed into the linkerd mesh. To mesh it, we need to provide an annotation to the deployment to inject linkerd
kubectl get -n emojivoto deploy -o yaml | linkerd inject - | kubectl apply -f -
deployment "emoji" injected
deployment "vote-bot" injected
deployment "voting" injected
deployment "web" injected
deployment.apps/emoji configured
deployment.apps/vote-bot configured
deployment.apps/voting configured
deployment.apps/web configured
This installs linkerd proxy along with the example application pods
Inject linkerd in the EnRoute pod
Once EnRoute is ready to work with linkerd, we inject the linkerd proxy in the EnRoute pod
kubectl get -n demo deploy -o yaml | linkerd inject - | kubectl apply -f -
deployment "enroute-linkerd-meshed" injected
deployment.apps/enroute-linkerd-meshed configured
We use the Buoyant cloud to trace the status of linkerd injection and mesh formation. The picture below shows how the EnRoute deployment was mutated to add the linkerd proxy, it is captured a couple of screens on the bouyant cloud
The event screen also captures the linkerd proxy injection -
We can also use kubectl
to check pods to see that linkerd got inserted in the pod
kubectl get pods -n demo -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |sort
enroute-linkerd-meshed-5d9575d575-4bskt: cr.l5d.io/linkerd/proxy:stable-2.11.1, saarasio/enroute-gwi:latest, redis:latest, envoyproxy/envoy:v1.21.1,
6. Encryption from client to EnRoute
Install service-policy chart to expose the service externally with L7 policies
helm install emojivoto-service-policy saaras/service-policy \
--set service.name=web-svc \
--set service.prefix=/ \
--set service.port=80 \
--namespace emojivoto \
--set service.fqdn=enroute-linkerd-mesh.enroutedemo.com \
--set service.enableTLS=false
NAME: emojivoto-service-policy
LAST DEPLOYED: Mon Mar 21 05:04:39 2022
NAMESPACE: emojivoto
STATUS: deployed
REVISION: 1
NOTES:
------------------ Configuration Graph -----------------------------------
Filters enabled for service [web-svc]
|
web-svc-80-luatestfilter
kubectl edit -n emojivoto httpfilters.enroute.saaras.io web-svc-80-luatestfilter
|
web-svc-80-rl2
kubectl edit -n emojivoto routefilters.enroute.saaras.io web-svc-80-rl2 (per-route ratelimit)
kubectl edit -n emojivoto globalconfigs.enroute.saaras.io web-svc-rl-global-config (ratelimit engine global config)
𝙴𝚗𝚁𝚘𝚞𝚝𝚎 Community Edition Service Policy
----------------------------------------
Request an evaluation license for enterprise version - contact@saaras.io
Slack Channel - https://slack.saaras.io
Getting Started Guide - https://getenroute.io/docs/getting-started-enroute-ingress-controller/
EnRoute Features - https://getenroute.io/features/
Send some traffic
curl -I enroute-linkerd-mesh.enroutedemo.com
HTTP/1.1 200 OK
content-type: text/html
date: Mon, 21 Mar 2022 05:09:51 GMT
content-length: 560
x-envoy-upstream-service-time: 4
vary: Accept-Encoding
lua-filter-says: Hello
server: envoy
while true; do enroute-linkerd-mesh.enroutedemo.com &> /dev/null; done
Enable end-to-end TLS by enabling TLS for downstream client connections
We will be using JetStack Cert Manager integration with EnRoute to install a certificate for example workload emojivoto. We need to setup DNS for automatic verification of certificate.
Setup DNS
We create a DNS entry for a domain to enable verification of certificate (installed using Let’s Encrypt) and install it on our example workload
Get ExternalIP
Check that the EnRoute service is installed with an External-IP
kubectl get service -n demo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
enroute-linkerd-meshed LoadBalancer 10.245.139.117 137.184.245.252 80:30695/TCP,443:31864/TCP 94m
Create DNS Entry
Create a DNS entry from domain name to the external IP. Verify the DNS is setup correctly -
ping enroute-linkerd-mesh.enroutedemo.com
PING enroute-linkerd-mesh.enroutedemo.com (137.184.245.252) 56(84) bytes of data.
64 bytes from 137.184.245.252 (137.184.245.252): icmp_seq=1 ttl=57 time=0.827 ms
^C
--- enroute-linkerd-mesh.enroutedemo.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.827/0.827/0.827/0.000 ms
Install Certificate on service
helm upgrade emojivoto-service-policy saaras/service-policy \
--set service.name=web-svc \
--set service.prefix=/ \
--set service.port=80 \
--namespace emojivoto \
--set service.enableTLS=true \
--set autoTLS.certificateCN=enroute-linkerd-mesh.enroutedemo.com \
--set autoTLS.email=contact@example.io \
--set autoTLS.createIssuers=true \
--set autoTLS.enableProd=true
Release "emojivoto-service-policy" has been upgraded. Happy Helming!
NAME: emojivoto-service-policy
LAST DEPLOYED: Mon Mar 21 05:12:19 2022
NAMESPACE: emojivoto
STATUS: deployed
REVISION: 2
NOTES:
------------------ Configuration Graph -----------------------------------
Filters enabled for service [web-svc]
|
web-svc-80-luatestfilter
kubectl edit -n emojivoto httpfilters.enroute.saaras.io web-svc-80-luatestfilter
|
web-svc-80-rl2
kubectl edit -n emojivoto routefilters.enroute.saaras.io web-svc-80-rl2 (per-route ratelimit)
kubectl edit -n emojivoto globalconfigs.enroute.saaras.io web-svc-rl-global-config (ratelimit engine global config)
𝙴𝚗𝚁𝚘𝚞𝚝𝚎 Community Edition Service Policy
----------------------------------------
Request an evaluation license for enterprise version - contact@saaras.io
Slack Channel - https://slack.saaras.io
Getting Started Guide - https://getenroute.io/docs/getting-started-enroute-ingress-controller/
EnRoute Features - https://getenroute.io/features/
Check certificates
kubectl get certificates.cert-manager.io --all-namespaces
NAMESPACE NAME READY SECRET AGE
emojivoto enroute-linkerd-mesh.enroutedemo.com True enroute-linkerd-mesh.enroutedemo.com 63s
Send traffic
curl -I -v https://enroute-linkerd-mesh.enroutedemo.com
* Trying 137.184.245.252:443...
* TCP_NODELAY set
* Connected to enroute-linkerd-mesh.enroutedemo.com (137.184.245.252) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=enroute-linkerd-mesh.enroutedemo.com
* start date: Mar 21 04:12:46 2022 GMT
* expire date: Jun 19 04:12:45 2022 GMT
* subjectAltName: host "enroute-linkerd-mesh.enroutedemo.com" matched cert's "enroute-linkerd-mesh.enroutedemo.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
...
> HEAD / HTTP/2
> Host: enroute-linkerd-mesh.enroutedemo.com
> user-agent: curl/7.68.0
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
HTTP/2 200
< content-type: text/html
< date: Mon, 21 Mar 2022 05:14:01 GMT
< content-length: 560
< x-envoy-upstream-service-time: 4
< vary: Accept-Encoding
< lua-filter-says: Hello
< server: envoy
<
Let us verify End-To-End Encryption -
The browser to EnRoute connection is encrypted using the certificate we installed on emojivoto app -
We can also see the workload and EnRoute are meshed in the “Meshed” column below
We can also see that the incoming traffic shows up as Application TLS
The topology screen also displays the mTLS betwen services and direction of flow of traffic
Conclusion
EnRoute provides an extremely easy integration with linkerd.
EnRoute integrates easily with both linkerd and Istio service mesh. The integration with istio service mesh is covered in another article.