EnRoute Gateway With AWS
EnRoute Gateway works in any cloud environment that runs a Kubernetes Cluster. EnRoute gateway works with public cloud provider Kubernetes or any other distribution.
This guide provides details on running EnRoute gateway on AWS, most common deployment configuration and different options available in AWS environment.
In AWS environment, EnRoute service annotations can program the AWS load balancer. AWS has several different loadbalancers that differ in the functionality provided. We explore different configurations to work with AWS LoadBalancers, the details of configuration and how good architectural practices.
We also discuss how less reliance on cloud provided load balancer can help achieve a better cloud-agnostic architecture which can be ported to other cloud environments.
AWS LoadBalancer options are
- Classic Load Balancer
- Application LoadBalancer (ALB)
- Network Load Balancer (NLB)
When a Kubernetes service of type
LoadBalancer is created, it provisions a Load Balancer on AWS. By default, a classic load balancer is provisioned by the Kubernetes Controller on AWS. Provisioning of an Application Load Balancer has to be manual. Annotations can be used on the
LoadBalancer service to provision a network load balancer.
Load Balancer annotations used on the
LoadBalancer service can be used to provide configuration options to the Classic Load Balancer or the Network Load Balancer
The Classic Load Balancer was the first one created by AWS. Subsequently, a Network Load Balancer (NLB) and Application Load Balancer (ALB) were created as evolved versions of the load balancer. While Kubernetes
LoadBalancer service can be run with either NLB or ALB, the preferred mechanism is using the NLB -
- Using a NLB load balancer keeps most of the L7 functionality inside Kubernetes and hence portable across clouds
- A NLB load balancer is cheaper to use compared to the ALB
- A NLB does not inspect traffic at L7. This also makes it the most efficient Load Balancer on AWS capable of handling millions of connections
- Annotations on the
LoadBalancerservice can be use to provide configuration options to the NLB
Network LoadBalancer (NLB)
The NLB runs at TCP/SSL layer to load balance TCP/IP traffic. It does not look at any HTTP or HTTPs state to make load balancing decisions. The NLB can however terminate SSL connections.
NLB is the recommended mechanism to work in Kubernetes environments on AWS to keep costs minimal and delegate all the functionality to the L7 Proxy inside the cluster. See the section on Architectural Choices below to understand why delegating SSL termination and load balancing to a service inside Kubernetes is more desirable.
Kubernetes controller on AWS provisions a cloud Load Balancer for
LoadBalancer type of service. Configuration to the load balancer can be provided by specifying annotations on service definition.
helm chart includes support for provisioning a NLB on AWS along with switches to control annotations. We discuss the most common options that are recommended to run EnRoute with AWS NLB load balancer
|service.beta.kubernetes.io/aws-load-balancer-type: nlb||Sets the load balancer to NLB, if this annotation is not used it defaults to classic load balancer|
|service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp||Configures NLB for a simple TCP passthrough (L4) or can be configured to terminate SSL. Valid values are tcp and ssl|
|service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"||Configures NLB for High-Availability for with load balancing across zones|
|service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'||Enables the Proxy protocol, used to preserve original client IP|
EnRoute helm chart has a switch to enable AWS annotations to achieve this configuration when setting up the enroute service
helm upgrade enroute-demo saaras/enroute \ --set enrouteService.rbac.create=true \ --set awsSettings.enable=true
Example Working configuration for EnRoute service
Here is an example of how a working sample configuration looks like for an EnRoute deployment using NLB with original client IP preserved
apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true" service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' service.beta.kubernetes.io/aws-load-balancer-type: nlb name: enroute namespace: enroutedemo spec: externalTrafficPolicy: Local ports: - name: http port: 80 protocol: TCP targetPort: 8080 - name: https port: 443 protocol: TCP targetPort: 8443 selector: app: enroute sessionAffinity: None type: LoadBalancer
We discuss here some of the architectural choices that can be made for an application. Appropriate choices help application make more cloud agnostic and provide better control and visibility over the application.
For example, terminating TLS inside Kubernetes using EnRoute make TLS termination and cipher selection cloud agnostic. EnRoute also integrates with external certificate providers using Jetstack and helps fetch and install certificates using one step. This greatly improves developer experience.
The same mechanism to terminate SSL traffic using EnRoute can be used on Google Cloud where certificates can be fetched an installed in one-step using the ACME protocol.
EnRoute support for ACME protocol using Jetstack Cert manager also provides the support for Let’s Encrypt Certificate Authority using one-step. When using Let’s Encrypt, a helm switch can be used to switch between staging and production Let’s Encrypt servers
Retain real client-IP address
EnRoute is built on unmodified Envoy proxy and can leverage the PROXY protocol to retain the original client-IP. Using NLB with annotations described above help with retaining the original source-IP.
HTTP to HTTPS redirection
EnRoute supports HTTP -> HTTPs redirection automatically when a certificate is installed for an application. No additional steps are required from the user.
L7 logging is an integral aspect for both troubleshooting and compliance. EnRoute integrates with external logging systems like fluentd and datadog to pipe log traffic directly to them. Logs can then be received at a choice of destination. For AWS environments, logs can be directed to CloudWatch destination to integrate with lest of the logging.
Since EnRoute can be horizontally scaled inside Kubernetes, it is very cost effective. EnRoute pricing is predictable and simple. It is simple consumption based pricing for billions of requests that matches the cloud consumption and pricing models. Typical EnRoute pricing results in over 90% savings over alternative solutions
Cloud Agnostic Architecture
Using Envoy / EnRoute inside Kubernetes for L7 function makes the architecture, logging and visibility, debugging and L7 security cloud-agnostic. The same architecture can be recreated for edge use-cases inside a Kubernetes cluster inside any other cloud. EnRoute’s flexibility and integration with independent logging providers also helps with this approach.
Envoy’s deep visibility
Envoy proxy provides deep visibility that provides deep insights into the working of the application. This is critical for operational efficiency and reducing the mean time to resolving problems when they arise.