Setting up Ingress controller NGINX along with HAproxy for Microservice deployed inside Kubernetes cluster
Working with kubernetes and managing the external traffic is like juggling more than two balls. All cloud service provider (GCP,AWS, Openshift, Digital Ocean) comes with their own load balancer which can help us exposing internal service deployed inside kubernetes cluster to the external world.
Exposing the services deployed within kubernetes cluster over Loadbalancer with external IP is really easy but considering the production use case -
Will it be easy for you to remember the URL which has IP address in it?
In my opinion I wouldn’t like to use any web-service where I always need to remember IP address while accessing it.
Well do not worry we have HAProxy Ingress Controller to take care of external traffic coming to kubernetes cluster
HAProxy Ingress Controller - It does all the heavy lifting when it comes to managing external traffic into kubernetes cluster and it requires primarily -
- IP address
What are we going to do?
- Setup kuberentes Cluster
- Install/setup HAProxy on kubernetes node
- Update frontend, backend configuration of haproxy.cfg (/etc/haproxy/haproxy.cfg)
- Setup kubernetes Ingress controller
- Deploy spring boot microservice inside kubernetes cluster
- Create Ingress resource
- Use HAProxy to access the deployed microservice withing kubernetes cluster
1. Setup kubernetes Cluster
The first and most essential step up for you is - “You should have a kuberentes cluster setup and it should be running. “
There are two ways to setup the kubernetes cluster -
- Setting it locally on your laptop - Click here to setup your local kubernetes cluster
- Using cloud service provider such as Google Cloud Platform -Click here to user Google cloud for setting it up kubernetes cluster
If you are doing this for learning purpose than I would prefer to go for option 1 for setting it locally on your laptop.
But if you are already familiar with Google cloud platform than I would choose option 2.
Once you setup your kubernetes cluster you can run the following kubectl command to verify your cluster
1$ kubectl get all 2`` 3 4You should see default kubernetes service running as ClusterIP 5 6```bash 7NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 8service/kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 31m
2. Install/setup HAProxy on kubernetes node
Before we deep dive into the Kubernetes Ingress controller, lets complete our first and most important per-requisite of installing HAProxy Loadbalancer.
First update your package information using following command
1$ sudo apt-get update
1$ sudo yum check-update
You can use the following command to install HAProxy Loadbalancer
1$ sudo apt-get -y install haproxy
1$ sudo yum install haproxy
Verify the installation - After the successful installation you should be able to see haproxy.cfg at /etc/haproxy/haproxy.cfg
You can check the installation version also -
1$ haproxy version 2 HA-Proxy version 1.8.8-1ubuntu0.11 2020/06/22
3. Update frontend, backend configuration of haproxy.cfg (/etc/haproxy/haproxy.cfg)
After the installation now you need to update the frontend as well as backend configuration for HAProxy.
Frontend - It receives the requests from the clients.
1frontend Local_Server 2 bind *:80 3 mode http 4 default_backend k8s_server</code></pre>
(If you do not have the above frontend configuration in your haproxy.cfg file then please do add this configuration at the end of the file.)
Backend - It is responsible for fulfilling the request sent from Frontend. Look at the following Backend configuration -
1backend k8s_server 2 mode http 3 balance roundrobin 4 server web1.example.com 188.8.131.52:8080
(If you do not have this backend configuration in your haproxy.cfg then i would suggest you to add at the end of the file)
If you are wondering where to find the haproxy.cfg then use the following command
1$ sudo vi /etc/haproxy/haproxy.cfg
You final haproxy.cfg should look like this -
1global 2 log /dev/log local0 3 log /dev/log local1 notice 4 chroot /var/lib/haproxy 5 stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners 6 stats timeout 30s 7 user haproxy 8 group haproxy 9 daemon 10 11 # Default SSL material locations 12 ca-base /etc/ssl/certs 13 crt-base /etc/ssl/private 14 15 # Default ciphers to use on SSL-enabled listening sockets. 16 # For more information, see ciphers(1SSL). This list is from: 17 # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ 18 # An alternative list with additional directives can be obtained from 19 # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy 20 ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS 21 ssl-default-bind-options no-sslv3 22 23defaults 24 log global 25 mode http 26 option httplog 27 option dontlognull 28 timeout connect 5000 29 timeout client 50000 30 timeout server 50000 31 errorfile 400 /etc/haproxy/errors/400.http 32 errorfile 403 /etc/haproxy/errors/403.http 33 errorfile 408 /etc/haproxy/errors/408.http 34 errorfile 500 /etc/haproxy/errors/500.http 35 errorfile 502 /etc/haproxy/errors/502.http 36 errorfile 503 /etc/haproxy/errors/503.http 37 errorfile 504 /etc/haproxy/errors/504.http 38 39frontend Local_Server 40 bind *:80 41 mode http 42 default_backend k8s_server 43 44backend k8s_server 45 mode http 46 balance roundrobin 47 server web1.example.com 184.108.40.206:8080
Once you have updated you haproxy.cfg file then you need to verify you configuration.
You can run the following command to check the correctness of the configuration .
1$ haproxy -c -f /etc/haproxy/haproxy.cfg
If your configuration is correct then you should see the following message
1Configuration file is valid
Now you need to restart haproxy
1$ sudo service haproxy restart
4. Setup kubernetes Ingress controller
Ingress controller are not set up by default inside the kubernetes cluster. We need to set up them manually. There are many Ingress controllers available such as -AKS Application Gateway Ingress Controller , Ambassador, AppsCode Inc., AWS ALB Ingress Controller, Contour, Citrix Ingress Controller, F5 BIG-IP, Gloo, Istio, Kong, Skipper, Traefik
But in our case we are going with NGINX Ingress Controller for Kubernetes.
If you like than there is official guide for setting up NGINX Ingress controller.
But here are the steps which I followed for setting it up -
- You need to clone the git repo -
1$ git clone https://github.com/nginxinc/kubernetes-ingress.git
- Go to the directory kubernetes-ingress/deployments
- Inside the deployments directory you will find namespace and service account yaml .e.g. ns-and-sa.yaml. Using this yaml we need to create namespace and service account for the Ingress controller.
You can find ns-and-sa.yaml, inside the directory common/ns-and-sa.yaml
1$kubectl apply -f common/ns-and-sa.yaml
1namespace/nginx-ingress created 2serviceaccount/nginx-ingress created
- As a next step you need to create cluster role and cluster role binding for the service account which we have created in step no 3.
For the cluster role and cluster role binding you can find rbac.yaml inside the directory rbac/rbac.yaml
1$ kubectl apply -f rbac/rbac.yaml
1clusterrole.rbac.authorization.k8s.io/nginx-ingress created 2clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress created
- For the App protect role create the following role binding
1kubectl apply -f rbac/ap-rbac.yaml
- Now you need to create secret using TLS certificate and key for the server.
Use the default-server-secret.yaml available inside the directory common/default-server-secret.yaml
1$ kubectl apply -f common/default-server-secret.yaml
- For customizing NGINX configuration you need to create config map using nginx-config.yaml available at common/nginx-config.yaml
1$ kubectl apply -f common/nginx-config.yaml
- Lets create ingress controller pod using the deployment
1$ kubectl apply -f deployment/nginx-ingress.yaml
- Now run it as a Daemon set
1$ kubectl apply -f daemon-set/nginx-ingress.yaml
- Now we can check all the container images running inside the namespace - nginx-ingress
1$ kubectl get all -n nginx-ingress
After running the above command you should see something similar in your terminal
1NAME READY STATUS RESTARTS AGE 2pod/nginx-ingress-hqghc 1/1 Running 0 42s 3pod/nginx-ingress-jcxjv 1/1 Running 0 42s
1NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE 2daemonset.apps/nginx-ingress 2 2 2 2 2 <none> 42s
Till now we have setup NGINX controller but not the Ingress resource yet.
Before setting up Ingress resource first we need to deploy some application inside our kubernetes cluster.
“Why we need to deploy application before setting up Ingress resource?”
The answer to this question is - We need to have service deployed and running at certain port, so that we can use service name and port number inside Ingress resource.
5.Deploy spring boot microservice inside kubernetes cluster
Alright lets deploy spring boot microservice using follow command.
1$ kubectl create deployment demo --image=rahulwagh17/kubernetes:jhooq-k8s-springboot
(Note - If you want to know more about deploying Spring boot microservice i_nside kubernetes cluster than I would recommend to go through - Deploy Spring Boot microservices on kubernetes)_
Check the deployment
1$ kubectl get deployments 2NAME READY UP-TO-DATE AVAILABLE AGE 3demo 1/1 1 1 5h20m
Expose the deployment as service
1$ kubectl expose deployment demo --type=ClusterIP --name=demo-service --port=8080
Okay so now we have created the deployment and exposed the service as ClusterIP running on port 8080.
You can view the exposed service
1$ kubectl get service demo-service
1NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 2demo-service ClusterIP 10.233.62.13 <none> 8080/TCP 43h
6. Create Ingress resource
Before you create the Ingress resource you should be mindful about the two things -
- deployed service name .e.g. - demo-service
- service port .e.g. 8080
Now you should create a yaml with the name ingress-resource.yaml
1$ touch ingress-resource.yaml
You need to edit the ingress-resource.yaml and fill in with configuration needed for the service .e.g. - demo-service
Copy the following configuration and paste it into your ingress-resource.yaml
1apiVersion: extensions/v1beta1 2kind: Ingress 3metadata: 4 name: springboot-ingress 5 annotations: 6 ingress.kubernetes.io/rewrite-target: / 7spec: 8 rules: 9 - host: jhooq.demo 10 http: 11 paths: 12 - backend: 13 serviceName: demo-service 14 servicePort: 8080
Alright now after updating the ingress-resource.yaml, you need to create the ingress resource using following command
1$ kubectl create -f ingress-resource.yaml
Once your Ingress resource is created, you can check it with the following command
1$ kubectl describe ing springboot-ingress
You should be able to see something similar in your terminal
1 Name: springboot-ingress 2Namespace: default 3Address: 4Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) 5Rules: 6 Host Path Backends 7 ---- ---- -------- 8 jhooq.demo 9 demo-service:8080 (10.233.90.4:8080) 10Annotations: ingress.kubernetes.io/rewrite-target: / 11Events: 12 Type Reason Age From Message 13 ---- ------ ---- ---- ------- 14 Normal AddedOrUpdated 103s nginx-ingress-controller Configuration for default/springboot-ingress was added or updated 15 Normal AddedOrUpdated 103s nginx-ingress-controller Configuration for default/springboot-ingress was added or updated
7. Use HAProxy to access the deployed microservice withing kubernetes cluster
Once you have implemented all the 6 steps then you are pretty much done setting up everything.
One last thing which remaining is to add an host entry into your /etc/hosts file.
Since our host IP address in this example is 220.127.116.11, so make an following entry inside the /etc/hosts
1$ vi /etc/hosts
Lets test our microservice using the hostname .i.e. jhooq.demo
1$ curl jhooq.demo/hello
1Hello - Jhooq-k8s
(Note : - You should test this from outside of your kubernetes cluster )
Here is screenshot from the browser.