# Kubernetes

# Cluster security checklist

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFileSystem: false
  runAsNonRoot: true

Optional:

# Monitoring and logging

  • We use Grafana and prometheus scrappers to monitor our infrastructure.

# Taint and label name convention

  • Role label format: node-role.kubernetes.io/maintenance=true (maintenance is the role).
  • Node taint format: node-role.kubernetes.io/maintenance=true:NoSchedule (maintenance is the taint's name).

# Helm and charts

  • Use Helm v3.
  • Store application chart in application repository ./ci/chart. It's easier to sync chart with any apllication changes.
  • Do not store passwords, tokens and keys in charts.
  • Use apliteni/kubernetes-helm:3 image in CI/CD pipelines.

Optional:

# New cluster provision checklist

  • Install sealed-secrets to make kubeseal works.
  • Install csi-driver for Hetzner Volumes.
  • Install hcloud-controller-manager to make Hetzner Load Balancers works.
  • Install cert-manager for automated SSL certificates.
  • Install nginx-ingress to create Load Balancer in cluster.

# Aliases

alias k=kubectl 
alias h=helm

# How to set up a new project for deployment to a cluster

Check apliteni/shared-tools repo.

# Kubeseal

We use kubeseal to safely store secrets.

  1. Install kubeseal-cli to encode secrets:

    $ brew install kubeseal

  2. Configure kubectl to the cluster context

  3. Encrypt your Secret into a SealedSecret:

    kubectl create secret generic secertname --dry-run --from-literal=token=${token} -o yaml -n kube-system |
    kubeseal
    --controller-namespace kube-system
    --controller-name sealed-secrets
    --format yaml
    > sealed-secret.yaml

  4. Commit sealed secret into the repo. It is safe to store.

# Merge multiple kube config

  1. Create file ~/.kube/load-configs
#!/usr/bin/env bash

DEFAULT_KUBECONFIG_FILE="$HOME/.kube/config"
if test -f "${DEFAULT_KUBECONFIG_FILE}"
then
  export KUBECONFIG="$DEFAULT_KUBECONFIG_FILE"
fi

# Your additional kubeconfig files should be inside ~/.kube/configs
ADD_KUBECONFIG_FILES="$HOME/.kube/configs"
mkdir -p "${ADD_KUBECONFIG_FILES}"
OIFS="$IFS"
IFS=$'\n'
for kubeconfigFile in `find "${ADD_KUBECONFIG_FILES}" -type f -name "*.yml" -o -name "*.yaml"`
do
    export KUBECONFIG="$kubeconfigFile:$KUBECONFIG"
done
IFS="$OIFS"
  1. Add to .bashrc or .zshrc:

    source ~/.kube/load-configs

# Useful tools

  • https://github.com/ahmetb/kubectx (opens new window) helps switch contexts fast.

  • Get taints kubectl get nodes -o json | jq ".items[]|{NAME:.metadata.name, TAINTS:.spec.taints}".

  • Get nodes with labels kubectl get nodes --show-labels.

  • Debug chart templates helm install --dry-run --debug ci/chart --values ci/chart/values.production.yaml --name RELEASE_NAME.

  • If you need to delete all evicted pods kubectl delete pod --field-selector="status.phase==Failed"

  • Stern (opens new window) to read logs fast.

  • k9s (opens new window) it an UI for terminal to browse cluster resources rapidly.