Configure pod to pod communication using Calico
Configuring a network in Kubernetes is a challenge. It requires you to deal with many nodes and pods in a cluster system. There are four problems you need to address while configuring the network:
- Container to container (which collectively provides a service) communication
- Pod to pod communication
- Pod to service communication
- External to service communication
Pod to pod communication
By default, docker creates a virtual bridge called docker0 on the host machine and it assigns a private network range to it. For each container that is created, a virtual Ethernet device is attached to this bridge. The virtual Ethernet device is then mapped to eth0 inside the container, with an IP from the network range. This process happens for each host that is running docker. There is no coordination between these hosts therefore the network ranges might collide.
Because of this, containers can only communicate with containers that are connected to the same virtual bridge. To communicate with other containers on other hosts, they must rely on port mapping. That means, you need to assign a port on the host machine to each container and then forward all the traffic on that port to that container.
Since the local IP address of the application is translated to the host IP address and port on the host machine, Kubernetes assumes that all nodes can communicate with each other without NAT. It also assumes that the IP address that a container sees for itself is the same IP address that the other containers see for the container. This approach also enables you to port applications easily from virtual machines to containers.
Calico is one of the many different networking options that offer these capabilities for Kubernetes.
Calico
Calico is designed to simplify, scale, and secure cloud networks. The open source framework enables Kubernetes networking and network policy for clusters across the cloud. Within the Kubernetes ecosystem, Calico is starting to emerge as one of the most popularly used network frameworks or plug-ins, with many enterprises using it at scale.
Calico uses a pure IP networking fabric to deliver high performance Kubernetes networking, and its policy engine enforces developer intent for high-level network policy management. Calico provides Layer 3 networking capabilities and associates a virtual router with each node. It enables host to host and pod to pod networking. Calico allows establishment of zone boundaries through BGP or encapsulation through IP on IP or VXLAN methods.
Integration between Kubernetes and Calico
Calico integrates with Kubernetes through a CNI plug-in built on a fully distributed, layer 3 architecture. Hence, it scales smoothly from a single laptop to large enterprise. It relies on an IP layer and it is relatively easy to debug with existing tools.
Configure the network with Calico
First, bring up a Kubernetes cluster with Calico using the following commands:
> kubeadm init --pod-network-cidr=192.168.0.0/16
> export KUBECONFIG=/etc/kubernetes/admin.conf
> kubectl apply -f calico.yaml
A master node is created with Calico as the CNI. After the master node is up and running, you can join the other nodes to the master using the join command.
Calico processes that are part of the Kubernetes master node are:
-
Calico etcd
kube-system calico-etcd-j4rwc 1/1 Running
-
Calico controller
kube-system calico-kube-controllers-679568f47c-vz69g 1/1 Running
-
Calico nodes
kube-system calico-node-ct6c9 2/2 Running
Note:
When you join a node to the Kubernetes cluster, a new Calico node is initiated on the Kubernetes node.
Configure BGP peer with Ingress NetScaler
Whenever you deploy an application after establishing the Calico network in the cluster, Kubernetes assigns an IP address from the IP address pool of Calico to the service associated with the application.
Border Gateway Protocol (BGP) uses autonomous system number (AS number) to identify the remote nodes. The AS number is a special number assigned by IANA used primarily with BGP to identify a network under a single network administration that uses unique routing policy.
Configure BGP on Kubernetes using Ingress NetScaler
Using a YAML file, you can apply BGP configuration of a remote node using the kubectl create command. In the YAML file, you need to add the peer IP address and the AS number. The peer IP address is the Ingress NetScaler IP address and the AS number is the AS number that is used in the Ingress NetScaler.
Obtain the AS Number of the cluster
Using the calicoctl command, you can obtain the AS number that is used by Calico BGP in the Kubernetes cluster as shown in the following image:
Configure global BGP peer
Using the calicoctl utility, you can peer Calico nodes with global BGP speakers. This kind of peers is called global peers.
Create a YAML definition file called bgp.yml with the following definition:
apiVersion: projectcalico.org/v3 # This is the version of Calico
kind: BGPPeer # BGPPeer specifies that its Global peering.
metadata:
name: bgppeer-global-3040 # The name of the configuration
spec:
peerIP: 10.102.33.208 # IP address of the Ingress NetScaler
asNumber: 500 # AS number configured on the Ingress NetScaler
<!--NeedCopy-->
Deploy the definition file using the following command:
> kubectl create -f bgp.yml
Add the BGP configurations on the Ingress NetScaler
Perform the following:
-
Log on to the NetScaler command-line interface.
-
Enable the BGP feature using the following command:
> en feature bgp Done -
Type
vtyshand press Enter.> vtysh ns# -
Change to config terminal using the
conf tcommand:ns#conf t Enter configuration commands, one per line. End with CNTL/Z. ns(config)# -
Add the BGP route with the AS number as 500 for demonstration purpose. You can use any number as the AS number.
ns(config)# router bgp 500 ns(config-router)# -
Add neighbors using the following command:
ns(config-router)# Neighbor 10.102.33.198 remote-as 64512 ns(config-router)# Neighbor 10.102.22.202 remote-as 64512 -
Review the running configuration using the following command:
ns(config-router)#show running-config ! log syslog log record-priority ! ns route-install bgp ! interface lo0 ip adress 127.0.0.1/8 ipv6 address fe80: :1/64 ipv6 address : :1/128 ! interface vlan0 ip address 10.102.33.208/24 ipv6 address fe80::2cf6:beff:fe94:9f63/64 ! router bgp 500 max-paths ebgp 8 max-paths ibgp 8 neighbor 10.102.33.198 remote-as 64512 neighbor 10.102.33.202 remote-as 64512 ! end ns(config-router)# In the sample, the AS number of Calico is 64512, you can change this number as per your requirement. -
Install the BGP routes to NetScaler routing table using the following command:
ns(config)# ns route-install bgp ns(config)# exit ns#exit Done -
Verify the route and add to the routing table using the following command:
Once the route is installed, the NetScaler is able to communicate with services that are present in the Kubernetes cluster:
Troubleshooting
You can verify BGP configurations on the master node in the Kubernetes cluster using the calicoctl script.
View the peer IP address and AS number configurations
You can view the peer IP address and AS number configurations using the following command:
>./calicoctl.1 get bgpPeer
NAME PEERIP NODE ASN
bgppeer-global-3040 10.102.33.208 (global) 500
View the BGP node status
You can view the status of a BGP node using the following command:
>calicoctl node status
IPV4 BGP status
+---------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+-----------+-------+----------+-------------+
| 10.102.33.208 | global | up | 16:38:14 | Established |
+---------------+-----------+-------+----------+-------------+