# Kubernetes
# Cluster descriptions
Check apliteni/infra/inventory
repository.
# Monitoring and logging
- We use Grafana and prometheus scrappers to monitor our infrastructure.
# Taint and label name convention
- Role label format:
node-role.kubernetes.io/maintenance=true
(maintenance
is the role). - Node taint format:
node-role.kubernetes.io/maintenance=true:NoSchedule
(maintenance
is the taint's name).
# Helm and charts
- Use Helm v3.
- Store application chart in application repository
./ci/chart
. It's easier to sync chart with any apllication changes. - Do not store passwords, tokens and keys in charts.
- Use
apliteni/kubernetes-helm:3
image in CI/CD pipelines.
Optional:
Write helm tests (opens new window) for shared charts.
- Write helm release tests (opens new window) for application charts.
# New cluster provision checklist
- Install
sealed-secrets
to make kubeseal works. - Install
csi-driver
for Hetzner Volumes. - Install
hcloud-controller-manager
to make Hetzner Load Balancers works. - Install
cert-manager
for automated SSL certificates. - Install
nginx-ingress
to create Load Balancer in cluster.
# Aliases
alias k=kubectl
alias h=helm
# How to set up a new project for deployment to a cluster
Check apliteni/shared-tools
repo.
# Kubeseal
We use kubeseal to safely store secrets.
Install kubeseal-cli to encode secrets:
$ brew install kubeseal
Configure kubectl to the cluster context
Encrypt your Secret into a SealedSecret:
kubectl create secret generic secertname --dry-run --from-literal=token=${token} -o yaml -n kube-system |
kubeseal
--controller-namespace kube-system
--controller-name sealed-secrets
--format yaml \sealed-secret.yaml
Commit sealed secret into the repo. It is safe to store.
# Merge multiple kube config
- Create file
~/.kube/load-configs
#!/usr/bin/env bash
DEFAULT_KUBECONFIG_FILE="$HOME/.kube/config"
if test -f "${DEFAULT_KUBECONFIG_FILE}"
then
export KUBECONFIG="$DEFAULT_KUBECONFIG_FILE"
fi
# Your additional kubeconfig files should be inside ~/.kube/configs
ADD_KUBECONFIG_FILES="$HOME/.kube/configs"
mkdir -p "${ADD_KUBECONFIG_FILES}"
OIFS="$IFS"
IFS=$'\n'
for kubeconfigFile in `find "${ADD_KUBECONFIG_FILES}" -type f -name "*.yml" -o -name "*.yaml"`
do
export KUBECONFIG="$kubeconfigFile:$KUBECONFIG"
done
IFS="$OIFS"
Add to
.bashrc
or.zshrc
:source ~/.kube/load-configs
# Useful tools
https://github.com/ahmetb/kubectx (opens new window) helps switch contexts fast.
Get taints
kubectl get nodes -o json | jq ".items[]|{NAME:.metadata.name, TAINTS:.spec.taints}"
.Get nodes with labels
kubectl get nodes --show-labels
.Debug chart templates
helm install --dry-run --debug ci/chart --values ci/chart/values.production.yaml --name RELEASE_NAME
.If you need to delete all evicted pods
kubectl delete pod --field-selector="status.phase==Failed"
Stern (opens new window) to read logs fast.
k9s (opens new window) it a UI for terminal to browse cluster resources rapidly.