YJWANG

[Grafana-loki] docker-compose로 구축 및 rsyslog / journal 연동하기 본문

60.Cloud

[Grafana-loki] docker-compose로 구축 및 rsyslog / journal 연동하기

왕영주 2021. 5. 26. 16:23

2021.05.25 - [60.Cloud] - [Grafana-loki] Grafana-loki로 Logging system 구축하기

이전에 발행한 포스팅에 이어 Grafana-Loki에서 각 서버의 syslog 및 journal, static log를 송신하고자합니다.
구축 완료 후의 구성도는 아래와 같습니다.

 

Monitoring Server

Grafana

  • Dashboard

Loki

  • log를 적재하고 query문을통해 read할 수 있습니다. (tcp 3100)

Promtail

  • syslog를 타 서버에서 전송할 수 있도록 reciever 역할을 하고 (tcp 1514)
  • localhost의 journal을 Loki에 전송
  • localhost의 static log를 Loki에 전송 (exmaple /var/log/kolla 등)

 

Client Server

Promtail

  • localhost의 journal을 Loki에 전송
  • localhost의 static log를 Loki에 전송 (exmaple /var/log/kolla 등)

rsyslog

  • syslog를 타 서버에 전송하는 역할을 수행

 

구성 진행

references

 

Monitoring Server

loki에서 사용할 volume을 생성합니다. loki container의 user가 uid : 10001 이므로 맞게 생성합니다.

# mkdir -p /loki
# chown 10001:10001 /loki

이후 공식 홈페이지에서 docker-compose.yaml file과 loki-config.yaml , promtail-config.yaml 파일을 다운받습니다.

# cd /loki
# mkdir /loki/config
# wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/loki/loki-local-config.yaml -O config/loki-config.yaml
# wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-docker-config.yaml -O config/promtail-config.yaml
# wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/production/docker-compose.yaml -O docker-compose.yaml

내용을 환경에 맞게 수정합니다.

Loki

table_manager를 설정하여 2주 정도의 log만 보관되도록 설정했습니다.
자세한 내용은 공식 홈페이지의 configuration을 참고하시기 바랍니다.

# cat /loki/config/loki-config.yaml 
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

ingester:
  wal:
    enabled: true
    dir: /loki/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem

chunk_store_config:
  max_look_back_period: 336h # how far data can be queried

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h # how long data remaind

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Promtail

syslog recieve와 journal 및 static log를 모두 설정했습니다.
자세한 내용은 공식 홈페이지의 configuration을 참고하시기 바랍니다.

# cat /loki/config/promtail-config.yaml 
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions.yaml

clients:
  - url: http://192.168.123.100:3100/loki/api/v1/push

scrape_configs:
- job_name: yjwang
  static_configs:
  - targets:
      - localhost
    labels:
      job: yjwangtest
      __path__: /var/log/yjwang/*/*log

- job_name: syslog
  syslog:
    listen_address: 0.0.0.0:1514
    labels:
      job: "syslog"
  relabel_configs:
    - source_labels: ["__syslog_connection_ip_address"]
      target_label: "ip_address"
    - source_labels: ["__syslog_message_severity"]
      target_label: "severity"
    - source_labels: ["__syslog_message_facility"]
      target_label: "facility"
    - source_labels: ["__syslog_message_app_name"]
      target_label: "app_name"
    - source_labels: ["__syslog_message_hostname"]
      target_label: "host"

- job_name: journal
  journal:
    max_age: 12h
    labels:
      job: systemd-journal
  relabel_configs:
    - source_labels: ["__journal__systemd_unit"]
      target_label: "systemd_unit"
    - source_labels: ["__journal__hostname"]
      target_label: "nodename"
    - source_labels: ["__journal_syslog_identifier"]
      target_label: "syslog_identifier"

이후 docker-compose 파일을 수정합니다.
Host의 /loki 경로에 모든 data가 수집되도록 mount 했습니다.

# cat docker-compose.yaml 
version: "3"

networks:
  loki:

services:
  loki:
    container_name: gafana-loki
    image: grafana/loki:latest
    volumes:
      - /loki/config:/mnt/config
      - /loki:/loki
    ports:
      - "3100:3100"
    command: -config.file=/mnt/config/loki-config.yaml
    networks:
      - loki

  promtail:
    container_name: loki-promtail
    image: grafana/promtail:latest
    ports:
      - "1514:1514"
    volumes:
      - /var/log:/var/log
      - /loki/config:/mnt/config 
      - /var/log/journal/:/var/log/journal/ 
      - /run/log/journal/:/run/log/journal/ 
      - /etc/machine-id:/etc/machine-id 

    command: -config.file=/mnt/config/promtail-config.yaml
    networks:
      - loki

  grafana:
    container_name: grafana
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

container 실행
docker-compose가 없다면 아래 게시물을 참고하여 설치하시기 바랍니다.

https://yjwang.tistory.com/153

# docker-compose up -d
Creating loki-promtail ... done
Creating grafana       ... done
Creating gafana-loki   ... done

확인

# docker-compose ps
    Name                   Command               State                    Ports                  
-------------------------------------------------------------------------------------------------
gafana-loki     /usr/bin/loki -config.file ...   Up      0.0.0.0:3100->3100/tcp,:::3100->3100/tcp
grafana         /run.sh                          Up      0.0.0.0:3000->3000/tcp,:::3000->3000/tcp
loki-promtail   /usr/bin/promtail -config. ...   Up      0.0.0.0:1514->1514/tcp,:::1514->1514/tcp

이제 server측 설정은 끝났으므로 client 쪽을 설정해보겠습니다.

 

Client

만약 CentOS 계열이라면 SELinux를 disable 및 setroubleshooting 통해 syslog의 label을 변경해주시기 바랍니다.

rsyslog.conf 파일에 아래 내용을 추가합니다. (target과 port는 환경에 맞게 수정하시기 바랍니다.)

# tail /etc/rsyslog.conf 
action(type="omfwd" protocol="tcp" target="192.168.123.100" port="1514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted")

rsyslog 재기동

# systemctl restart rsyslog

Docker 및 Docker-compose를 사전에 설치 하시기 바랍니다.

promtail에서 사용할 dir 생성

# mkdir -p /loki
# cd /loki
# mkdir /loki/config

promtail-config.yaml, docker-compose.yaml 다운로드

# wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-docker-config.yaml -O config/promtail-config.yaml
# wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/production/docker-compose.yaml -O docker-compose.yaml

promtail-config.yaml 수정

# cat config/promtail-config.yaml 
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions.yaml

clients:
  - url: http://192.168.123.100:3100/loki/api/v1/push

scrape_configs:
- job_name: yjwang
  static_configs:
  - targets:
      - localhost
    labels:
      job: yjwangtest
      __path__: /var/log/yjwang/*/*log

- job_name: journal
  journal:
    max_age: 12h
    labels:
      job: systemd-journal
  relabel_configs:
    - source_labels: ["__journal__systemd_unit"]
      target_label: "systemd_unit"
    - source_labels: ["__journal__hostname"]
      target_label: "nodename"
    - source_labels: ["__journal_syslog_identifier"]
      target_label: "syslog_identifier"

docker-compose.yaml 수정

# cat docker-compose.yaml 
version: "3"

networks:
  loki:

services:
  promtail:
    container_name: loki-promtail
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - /loki/config:/mnt/config 
      - /var/log/journal/:/var/log/journal/ 
      - /run/log/journal/:/run/log/journal/ 
      - /etc/machine-id:/etc/machine-id 

    command: -config.file=/mnt/config/promtail-config.yaml
    networks:
      - loki

container 실행

# docker-compose up -d
Creating loki-promtail ... done

# docker-compose ps
    Name                   Command               State   Ports
--------------------------------------------------------------
loki-promtail   /usr/bin/promtail -config. ...   Up

 

Grafana 에서 확인

기본 암호는 admin/admin 입니다.

datasource를 아래와 같이 추가하고

explore에서 log를 검색해보시면 됩니다.

반응형
저작자표시
'60.Cloud' Related Articles more