Kubernetes Workload Cluster with k3s

This guide will show you how to deploy a Kubernetes cluster using k3s, with ingress for running workloads; this will be provided by Traefik as the ingress controller and kube-vip to advertise the BGP address.

Equinix Metal Metadata

As we're going to use the Equinix Metal metadata a fair bit during this guide, we're going to use curl to fetch the data and store it locally, to prevent future requests from going over the network.

curl -fsSL https://metadata.platformequinix.com/metadata > /tmp/metadata.json

k3s

The first thing you need to determine is whether you need your k3s API available outside of Equinix Metal. We strongly encourage you to think this constraint through thoroughly. If you're using GitOps to deploy your workloads, we'd recommend binding k3s to the private IPv4 management interface. You can grab this IP address with the following command.

export API_IP=$(jq -r '.network.addresses | map(select(.public==false and .management==true)) | first | .address' /tmp/metadata.json)

If you decide that it should be public, you'll want to use the public IPv4 management address.

export API_IP=$(jq -r '.network.addresses | map(select(.public==true and .management==true)) | first | .address' /tmp/metadata.json)

You can then install k3s with the following snippet. This configures k3s to bind to the correct IP address, based on the above, as well as disabling the cloud controller and service load balancer. We disable the service load balancer because Equinix Metal doesn't provide that functionality and we'll need to deploy kube-vip instead.

export INSTALL_K3S_EXEC="\
    --bind-address ${API_IP} \
    --advertise-address ${API_IP} \
    --node-ip ${API_IP} \
    --tls-san ${API_IP} \
    --no-deploy servicelb \
    --disable-cloud-controller"

curl -sfL https://get.k3s.io | sh -

kube-vip

BGP Routes

Now that k3s is running, we need to configure the routing table on the machine to understand how to communicate with the BGP peers for advertising any load balancer services with kube-vip.

We can fetch the gateway address from the metadata API and use ip route to add.

GATEWAY_IP=$(jq -r ".network.addresses[] | select(.public == false) | .gateway" /tmp/metadata.json)
ip route add 169.254.255.1 via $GATEWAY_IP
ip route add 169.254.255.2 via $GATEWAY_IP

RBAC

We now need to apply the kube-vip service account and role to our k3s control plane. This will apply just enough permissions for kube-vip to understand the services running within.

kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip/main/docs/manifests/rbac.yaml

This applies the following rules to our service account:

rules:
  - apiGroups: [""]
    resources: ["services", "services/status", "nodes"]
    verbs: ["list","get","watch", "update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["list", "get", "watch", "update", "create"]

Deploy

k3s supports running workloads via the manifests directory, available at /var/lib/rancher/k3s/server/manifests. We're going to use ctr to pull the kube-vip image and generate a manifest.

crictl pull ghcr.io/kube-vip/kube-vip:v0.3.5

ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:v0.3.5 kube-vip /kube-vip manifest daemonset \
  --interface lo \
  --vip ${API_IP} \
  --metal \
  --metalKey YOUR_METAL_API_KEY \
  --metalProjectID YOUR_METAL_PROJECT_ID \
  --services \
  --inCluster \
  --taint \
  --bgp \
   | tee /var/lib/rancher/k3s/server/manifests/kube-vip.yaml

Traefik

k3s automatically ships with Traefik on our cluster as the ingress controller, so we don't need to deploy anything. However, as we're not using a Service Load Balancer, we need to configure Traefik with the EIP.

kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml patch svc traefik -n kube-system -p '{"spec": {"type": "LoadBalancer", "loadBalancerIP":"${API_IP}"}}'