Logging/Monitoring Architecture

Promtail, Loki, Grafana
Logging and Monitoring Architecture with Fluentd, ElasticSearch, Grafana for Kubernetes.


1. Basic Logging Construction


Node Level Logging for Kubernetes.

1-1) Deployment 생성

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-log
spec:
  selector:
    matchLabels:
      type: app
  template:
    metadata:
      labels:
        type: app
    spec:
      nodeSelector:
        kubernetes.io/hostname: k8s-node1
      containers:
      - name: container
        image: kubetm/app

api 호출

curl <pod-ip>:8080/hostname
curl <pod-ip>:8080/version

1-2) Container Log 확인

kubectl exec로 Container 내부 로그파일 확인

kubectl exec <pod-name> -it -- /bin/sh

kubectl logs로 Stdout 로그 확인

kubectl logs <pod-name>
kubectl logs <pod-name> --tail 10 --follow

Docker Log Driver 설정 확인

cat /etc/docker/daemon.json

Docker Container 로그 파일 확인

cd  /var/lib/docker/containers
docker ps
cd <container-id>
ls

cd /var/log/pods
cd <Namespace>_<pod-name>_<pod-id>
cd container
ls -al

cd /var/log/containers
ls -al

1-4) Event Log 확인

kubectl get events
kubectl get events | grep app-log

1-5) Termination Log 확인

Termination API Call

curl <pod-ip>:8080/termination

Container Status 확인

kubectl describe pods <pod-name>


2. Loki Stack 설치


Loki Stack for Kubernetes.

2-1) Helm 설치

https://helm.sh/docs/intro/install/

curl -O https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz
tar -xf helm-v3.3.4-linux-amd64.tar.gz
cd linux-amd64
cp ./helm /usr/local/bin/

2-2) loki-stack 설치

패키지 다운로드 https://artifacthub.io/packages/helm/loki/loki-stack

helm repo add loki https://grafana.github.io/loki/charts
helm fetch loki/loki-stack --version 0.41.2

tar -xf loki-stack-0.41.2.tgz
cd loki-stack/
vi values.yaml
(Grafana Enable 설정)

helm 배포 / 삭제

kubectl create ns loki-stack
helm install loki-stack -f values.yaml . -n loki-stack
helm uninstall loki-stack -n loki-stack

2-3) Grafana Service

kubectl edit -n loki-stack svc loki-stack-grafana
----
type: NodePort
nodePort : 30000
----

kubectl get secret --namespace loki-stack loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


Referenece


Kubernetes