Kubernetes

• Cluster descriptions
• Monitoring and logging
• Taint and label name convention
• Helm and charts
• New cluster provision checklist
• Aliases
• How to set up a new project for deployment to a cluster
• Kubeseal
• Merge multiple kube config
• Useful tools

Cluster descriptions

Check apliteni/infra/inventory repository.

Monitoring and logging

• We use Grafana and prometheus scrappers to monitor our infrastructure.

Taint and label name convention

• Role label format: node-role.kubernetes.io/maintenance=true (maintenance is the role).
• Node taint format: node-role.kubernetes.io/maintenance=true:NoSchedule (maintenance is the taint's name).

Helm and charts

• Use Helm v3.
• Store application chart in application repository ./ci/chart. It's easier to sync chart with any apllication changes.
• Do not store passwords, tokens and keys in charts.
• Use apliteni/kubernetes-helm:3 image in CI/CD pipelines.

Optional:

Write helm tests (opens new window) for shared charts.

• Write helm release tests (opens new window) for application charts.

New cluster provision checklist

• Install sealed-secrets to make kubeseal works.
• Install csi-driver for Hetzner Volumes.
• Install hcloud-controller-manager to make Hetzner Load Balancers works.
• Install cert-manager for automated SSL certificates.
• Install nginx-ingress to create Load Balancer in cluster.

Aliases

alias k=kubectl 
alias h=helm

How to set up a new project for deployment to a cluster

Check apliteni/shared-tools repo.

Kubeseal

We use kubeseal to safely store secrets.

1. Install kubeseal-cli to encode secrets:

   $ brew install kubeseal

2. Configure kubectl to the cluster context

3. Encrypt your Secret into a SealedSecret:

   kubectl create secret generic secertname --dry-run --from-literal=token=${token} -o yaml -n kube-system |
   kubeseal
   --controller-namespace kube-system
   --controller-name sealed-secrets
   --format yaml \

   sealed-secret.yaml

4. Commit sealed secret into the repo. It is safe to store.

Merge multiple kube config

1. Create file ~/.kube/load-configs

#!/usr/bin/env bash

DEFAULT_KUBECONFIG_FILE="$HOME/.kube/config"
if test -f "${DEFAULT_KUBECONFIG_FILE}"
then
  export KUBECONFIG="$DEFAULT_KUBECONFIG_FILE"
fi

# Your additional kubeconfig files should be inside ~/.kube/configs
ADD_KUBECONFIG_FILES="$HOME/.kube/configs"
mkdir -p "${ADD_KUBECONFIG_FILES}"
OIFS="$IFS"
IFS=$'\n'
for kubeconfigFile in `find "${ADD_KUBECONFIG_FILES}" -type f -name "*.yml" -o -name "*.yaml"`
do
    export KUBECONFIG="$kubeconfigFile:$KUBECONFIG"
done
IFS="$OIFS"

2. Add to .bashrc or .zshrc:

   source ~/.kube/load-configs

Useful tools

• https://github.com/ahmetb/kubectx (opens new window) helps switch contexts fast.

• Get taints kubectl get nodes -o json | jq ".items[]|{NAME:.metadata.name, TAINTS:.spec.taints}".

• Get nodes with labels kubectl get nodes --show-labels.

• Debug chart templates helm install --dry-run --debug ci/chart --values ci/chart/values.production.yaml --name RELEASE_NAME.

• If you need to delete all evicted pods kubectl delete pod --field-selector="status.phase==Failed"

• Stern (opens new window) to read logs fast.

• k9s (opens new window) it a UI for terminal to browse cluster resources rapidly.