CKA course attended in Anisa
to initiate a cluster check out this repo.
- kubelet:
- interacts with the container runtime to deploy containers on the node.
- reports status of the node and the containers on that node to the API server.
- register nodes on the cluster
- healthcheck (probs)
systemctl restart kubelet
config file: /var/lib/kubelet/config.yaml
- Controller manager:
- monitors the nodes and containers
- API server:
- everyone talk to API server with anything they want to do or report
- ETCD:
- Key-value DB that contains cluster/applications info.
‍‍‍/etc/kubernetes/manifests/etcd.yml
- kube-scheduler:
- schedules containers/pods on the nodes.
- kubeproxy:
- between pods/hosts network management
- creates service endpoints
nerdctl and cri are utilities to manage containerd
runc creates namespaces and cgroups
*.conf files contain the access info on different roles like admin, kubelet, etc.
We can add multiple clusters' info on this file. and using kubectx or kubeconf switch between users/clusters.
works as net interfaces for containers. calico and flannel and ... work as the overlay network assigning IPs to those containers.
kubectl get node
kubectl get pod --namespace kube-system -o wide
kubectl get node --kubeconfig
kubeadm token list
kubeadm token create --print-join-command --ttl=12h
kubectl label node worker1 kubernetes.io/role=worker1
kubectl get all -A
kubectl get events
kubectl run nginx_name --image nginx:latest
kubectl get pod -o yaml
kubectl explain deployments.kind
install:
NERDCTL_VERSION=1.0.0 # see https://github.com/containerd/nerdctl/releases for the latest release
archType="amd64"
if test "$(uname -m)" = "aarch64"; then archType="arm64"; fi
wget "https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VERSION}/nerdctl-full-${NERDCTL_VERSION}-linux-${archType}.tar.gz" -O /tmp/nerdctl.tar.gz
mkdir -p ~/.local/bin
tar -C ~/.local/bin/ -xzf /tmp/nerdctl.tar.gz --strip-components 1 bin/nerdctl
echo -e '\nexport PATH="${PATH}:~/.local/bin"' >> ~/.bashrc
export PATH="${PATH}:~/.local/bin"
nerdctl completion bash | tee /etc/bash_completion.d/kubeadm > /dev/null
nerdctl
nerdctl -n k8s.io ps
nerdctl -n k8s.io images
nerdctl -n k8s.io kill 84a0b9e88743
can have multiple containers in it(helper container):
Helper containers can be used to perform a wide range of tasks, such as:
- Preparing the environment for the main container(s) by downloading configuration files, secrets, and other data.
- Managing the lifecycle of the main container(s) by monitoring their health and restarting them if necessary.
- Collecting and sending logs, metrics, and other data to external systems.
- Running tools and utilities for debugging, profiling, or testing the main container(s).
kubectl -n calico-system edit pod calico-node-rrpxr
pod itself does not have control over its state, e.g if the node fails pod won't be relocated.
kubectl get replicaset
kubectl get rs -o wide
kubectl replace -f rs-definition.yml
kubectl scale --replicas=3 -f rs-definition.yml
kubectl apply -f rs-definition.yml # no need of a kubectl replace later, do apply again
if we already have a pod with lable type: front-end that pod will be a part of this rs.
A ReplicaSet ensures that a specified number of pod replicas are running at any given time. However, a Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features like rolling update/roll back options.
kubectl rollout status deployment/dp-name
kubectl rollout history deployment/dp-name
kubectl set image # better not use it unless testing
the new versions are deployed in a new replicaset:
kubectl rollout undo deployment/dp-name # rollbacks to the previous version # we can use --to-revision
only pods on the same ns see each other by default. we can define resource limitations and accesses using ns.
kubectl create ns dev
the kube-public namespace exposes its pods to everyone.
- change default ns:
kubectl config set-context kubernetes-admin@kubernetes --namespace dev
set requested/limits to resources/object(pods/deployments,etc) for a namespace,lable or etc...
kubectl get resourcequotas
expose pods to outside/other pods within the cluster. load balance traffic to pods.
node port: only exposes on the host and the endpoint is each host's ip.
kubectl get ep nginx-service -o wide
kubectl get svc nginx-service -o wide
iptables-save | grep 10.104.199.182 # svc ip
clusterIP: exposes pod in cluster only. access pods between two namespaces:
curl <svc-name>.<ns-name>.svc.cluster.local
LoadBalancer:
manual through pod definition:
label & selectors can be used to do this.
if any pod can tolerate blue taint will be deployed on that node. other than that no node will be deployed on that node.
types of taints:
kubectl taint nodes <node-name> app=blue:NoSchedule
toleration configuration:
remove taint:
kubectl taint node worker01 app=nginx:NoSchedule-
kubectl label nodes worker01 size=Large
Affinity meaning:
define more detailed conditions/predictions than node selector.
CPU: K8s throttles(in queues) the cpu for the pod Memory: K8s terminate the pod (OOM(Out of Memory) killed)
just like the swarm service on global mode. 1 container/pod on each node. kube-proxy is deployed in this way.
kubectl -n kube-system get ds
The initial pods like etcd come up before the apiServer, so some pods get initialized by kubelet directly and not handled by the apiServer.
kubelet is always watching the pod manifest directory to understand what pods to run.
the trail the node their on's name.
you can define your own scheduler like do not schedule a new pod when 80% of resource allocation.
there's a cadvisor inside the kubelet that metrics server connect to get the metrics.
metrics server data is in memory and does not get stored.
install: https://github.com/kubernetes-sigs/metrics-server
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
this fails as it doesn't trust our cluster CA. so :
kubectl edit deployments.apps -n kube-system metrics-server
## add the ca or disable ca verification
usage :
kubectl top node/top
kubectl logs -f pod-name # logs generated by the pod
kubectl get events # logs generated in a cluster view
used this conf in multi-container-pod.yml
values should be in base64:
echo -n "value" | base64
they share the same network namespace(bridged).
kubectl exec -it multi-alp-pod -- sh
Defaulted container "alpine-container-1" out of: alpine-container-1, alpine-container-2
/ #
access one specific container:
root@manager1:~/CKA_Anisa/pods# kubectl exec -it multi-alp-pod -c alpine-container-2 -- sh
three design patterns and use cases of this: https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns/
run before the main container comes up.
kubectl logs multi-alp-pod --follow --container init-alpine
MUST DO:
apt-mark hold kubelet kubeadm kubectl
drain nodes:
kubectl drain <node>
keep existing pods on the node and labels the node unschedulable.
kubectl cordon <node>
to reverse both cordon and drain:
kubectl uncordon <node>
K8s components versions are recommended to be managed according to the version of kube-apiserver:
update a master:
kubeadm upgrade plan
apt-mark unhold kubelet kubeadm kubectl
apt update
apt install kubeadm=1.25.11-00 kubectl=1.25.11-00
kubeadm upgrade apply v1.25.11
apt install kubelet=1.25.11-00
apt-mark hold kubelet kubeadm kubectl
update a worker:
kubectl drain <node> # run on master
apt-mark unhold kubeadm kubectl kubelet
apt update
apt install kubeadm=1.25.11-00 kubectl=1.25.11-00
kubeadm upgrade node
apt install kubelet=1.25.11-00
apt-mark hold kubelet kubeadm kubectl
kubectl uncordon <node> # run on master
push resources to a repo in git:
export everything:
kubectl get all --all-namespaces -o yaml > all-deployed-services.yml
kubectl get deployments --all-namespaces -o yaml > all-deployed-services.yml
we could also use velero.
kube-apiserver should be down (it is an static pod the yaml file should be moved).
using etcdutl:
admin privilage CN and Organization:
kube-apiserver:
kubeadm certs check-expiration
csr generation:
openssl genrsa -out anisa.key 2048
openssl req -new -key anisa.key -out anisa.csr
A user delivers a CSR to admin and admin signs that with root CA.
cd ~/CKA_Anisa/Certs
cat anisa.csr | base64 -w 0 ## pass this as request field in csr.yml
kubectl apply -f csr.yml
kubectl get csr
kubectl describe csr anisa-csr
kubectl certificate -h
kubectl certificate approve anisa-csr
kubectl get csr anisa-csr -o yaml # this gets you the certificate in base64
echo "base64 coded" | base64 -d
## OR
kubectl get csr anisa-csr -ojsonpath='{.status.certificate}' | base64 -d
cd ~/CKA_Anisa/Certs/
cat ./kubeConfig.yml
kubernetes get pod --kubeconfig ./kubeConfig.yml
curl -k https://172.16.0.10:6443/api --key anisa.key --cert anisa.crt
curl -k https://172.16.0.10:6443/apis --key anisa.key --cert anisa.crt
curl -k https://172.16.0.10:6443/api/v1/pods --key anisa.key --cert anisa.crt
switch between contexts using kubectx and kubens: https://github.com/ahmetb/kubectx
sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
sudo ln -s /opt/kubectx/kubectx /usr/local/bin/kctx
sudo ln -s /opt/kubectx/kubens /usr/local/bin/kns
core API:
using this you won't need to specify --key and certs for curl and if you curl the proxy it has the kubectl access.
node authorization:
OU: system:node
ABAC:
RBACK:
Roles:
Get a list of resources:
kubectl api-resources
RoleBinding:
kubectl apply -f developer
kubectl apply -f developer-roleBinding.yml
check authorizations:
The default registry is docker.io(dockerhub).
nerdctl login equals docker login.
The better solution is to use secrets:
if the registry's certificate is self-signed:
Capabilities and run as user:
(add/drop)
- ingress: input traffic to pods
- egress: output traffic to pods
set a policy that a pod can only receive ingress/ send egress traffic to a specific policy.
connect to ceph, glusterfs, etc
The kubernetes object for a volume is a PV. Then we use a PVC to request a PV.
many means it can be assigned to multiple nodes. once means only one node.
persistenVolumeReclaimPolicy:
binding:
cni config files: /etc/cni/net.d/
When a service is created with an advertising IP and port, kube-proxy creates firewall rules to forward the requests to that endpoint to the corresponding pod of that service. Then, through routing the host knows where to send this request based on the routes that the cni has advertised.
Interesting thing is that iptables has a feature to set probability of acceptance of a rule. this K8s loadbalances between pods:
kube-proxy's mode can be set to iptables to IPVS.
Recommended to not use iptables if you have over 1000 services.
https://coredns.io/2017/05/08/custom-dns-entries-for-kubernetes/
kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 93d
metrics-server ClusterIP 10.106.101.124 <none> 443/TCP 70d
kubectl exec -it myapp-deployment-6fdd5f58cd-9l6lx -- cat /etc/resolv.conf
search dev.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
Just like RBAC but for non-human resources either within or outside of the cluster. for example if a pod wants to access secrets defined in the cluster or list pods in another namespace, part of a CICD pipe, etc.
Tshoot:
curl -L https://github.com/projectcalico/calico/releases/latest/download/calicoctl-linux-amd64 -o /usr/bin/calicoctl
chmod +x /usr/bin/calicoctl
calicoctl node status
how to get apiVersion, kind , etc(differs for each object/manifest (pod, etc)):
core group is the default and does not need to be mentioned in yaml.
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#pod-v1-core
these are the four necessary keys for pods:
apiVersion: v1
kind: Pod # always starts with capital letter
metadata: # name of the pod, namespace, labels etc
name: myapp-pod
labels:
app: myapp
type: front-end
spec:
containers:
- name: nginx-container
image: nginx:latest # default pulls out of dockerhub