Logging/Monitoring Architecture
Promtail, Loki, Grafana
1. Basic Logging Construction
1-1) Deployment 생성
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-log
spec:
selector:
matchLabels:
type: app
template:
metadata:
labels:
type: app
spec:
nodeSelector:
kubernetes.io/hostname: k8s-node1
containers:
- name: container
image: kubetm/app
api 호출
curl <pod-ip>:8080/hostname
curl <pod-ip>:8080/version
1-2) Container Log 확인
kubectl exec로 Container 내부 로그파일 확인
kubectl exec <pod-name> -it -- /bin/sh
kubectl logs로 Stdout 로그 확인
kubectl logs <pod-name>
kubectl logs <pod-name> --tail 10 --follow
Docker Log Driver 설정 확인
cat /etc/docker/daemon.json
Docker Container 로그 파일 확인
cd /var/lib/docker/containers
docker ps
cd <container-id>
ls
1-3) Node1에서 Log Link 확인
cd /var/log/pods
cd <Namespace>_<pod-name>_<pod-id>
cd container
ls -al
cd /var/log/containers
ls -al
1-4) Event Log 확인
kubectl get events
kubectl get events | grep app-log
1-5) Termination Log 확인
Termination API Call
curl <pod-ip>:8080/termination
Container Status 확인
kubectl describe pods <pod-name>
2. Loki Stack 설치
2-1) Helm 설치
https://helm.sh/docs/intro/install/
curl -O https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz
tar -xf helm-v3.3.4-linux-amd64.tar.gz
cd linux-amd64
cp ./helm /usr/local/bin/
2-2) loki-stack 설치
패키지 다운로드 https://artifacthub.io/packages/helm/loki/loki-stack
helm repo add loki https://grafana.github.io/loki/charts
helm fetch loki/loki-stack --version 0.41.2
tar -xf loki-stack-0.41.2.tgz
cd loki-stack/
vi values.yaml
(Grafana Enable 설정)
helm 배포 / 삭제
kubectl create ns loki-stack
helm install loki-stack -f values.yaml . -n loki-stack
helm uninstall loki-stack -n loki-stack
2-3) Grafana Service
kubectl edit -n loki-stack svc loki-stack-grafana
----
type: NodePort
nodePort : 30001
----
- kubernetes dahsboard UI Port가 30000으로 변경되어 해당 실습 port는 30001로 하시면 됩니다
kubectl get secret --namespace loki-stack loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Referenece
Kubernetes
- Logging Architecture : https://kubernetes.io/docs/concepts/cluster-administration/logging/