kafka 기본 설치와 도커 컴포즈를 활용한 클러스터 구성

by

01. 사전 개념

(01) Kafka 의 종류

  • Apache Kafka [ 자유롭게 사용 가능]
  • Confluent Kafka
    • 커뮤니티 버전에서도
    • 기본 Kafka 에서 지원하지 않는 기능이 지원됨 ( 프리미어 커넥트가 주요함 )

(02) 데이터 파이프 라인 이란 ?

사람의 개입없이 데이터를 오염, 중복 , 유실과 같은 결합 없이 수집 저장 ETL 이 가능하도록 일련의 흐름을 만들어 주는 과정

(03) 시스템 모니터링 방식

kafka 모니터링

(04) 메세징 허브

[기존] 기존의 서비스가 1개 시스템의 실패시 전체의 문제를 만들수 있음

[변경] 특정 프로세스의 실패시에도 해당 프로세스만 문제로 처리 되고, 서비스 마다 자신의 성능에 맞게 설정하여 진행이 가능하다.

02. Kafka 설치와 설정

01) Apache Kafka 설치 (단일 노드)

(01) 다운로드 사이트 접속

다운로드 링크 : Apache Kafka

(02) 압축 해제

tar -xvf kafka_2.13-2.8.0.tgz

[결과]

jay@JayLees-MacBook-Pro  ~/kafka-test  ls
kafka_2.13-2.8.0.tgz
 jay@JayLees-MacBook-Pro  ~/kafka-test  tar -xvf kafka_2.13-2.8.0.tgz
x kafka_2.13-2.8.0/
x kafka_2.13-2.8.0/LICENSE
x kafka_2.13-2.8.0/NOTICE
x kafka_2.13-2.8.0/bin/
x kafka_2.13-2.8.0/bin/zookeeper-shell.sh
x kafka_2.13-2.8.0/bin/kafka-log-dirs.sh
x kafka_2.13-2.8.0/bin/zookeeper-server-stop.sh
x kafka_2.13-2.8.0/bin/kafka-configs.sh
x kafka_2.13-2.8.0/bin/kafka-server-stop.sh
x kafka_2.13-2.8.0/bin/windows/
x kafka_2.13-2.8.0/bin/windows/kafka-delegation-tokens.bat
x kafka_2.13-2.8.0/bin/windows/kafka-producer-perf-test.bat
x kafka_2.13-2.8.0/bin/windows/kafka-run-class.bat
x kafka_2.13-2.8.0/bin/windows/kafka-server-stop.bat
x kafka_2.13-2.8.0/bin/windows/kafka-streams-application-reset.bat
x kafka_2.13-2.8.0/bin/windows/kafka-dump-log.bat
x kafka_2.13-2.8.0/bin/windows/kafka-server-start.bat

(03) 자주 접근을 위해 심볼릭 링크 추가 

ln -s kafka_2.13-2.8.0 kafka
 jay@JayLees-MacBook-Pro  ~/kafka-test  ln -s kafka_2.13-2.8.0 kafka
 jay@JayLees-MacBook-Pro  ~/kafka-test  cd kafka
 jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  ls
LICENSE   NOTICE    bin       config    libs      licenses  site-docs

(04) Zookeeper 설정

[Zookeeper 설정 파일 살펴 보기]

  • 2181 포트로 서비스 진행함

[zookeeper 실행] 

bin/zookeeper-server-start.sh config/zookeeper.properties

[실행 결과]

 jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  bin/zookeeper-server-start.sh config/zookeeper.properties
[2023-08-01 10:26:07,293] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,294] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,300] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,300] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,301] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2023-08-01 10:26:07,301] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2023-08-01 10:26:07,301] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
...
[2023-08-01 10:26:07,334] INFO binding to port 0.0.0.0/0.0.0.0:2181 
....
[2023-08-01 10:26:07,355] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.Cont

[1차 확인] 2181 포트 바인딩

[2차 확인] Telnet 접속 테스트 

 jay@JayLees-MacBook-Pro  ~  telnet 0 2181
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.

[Zookeeper 쉘 접근]

 bin/zookeeper-shell.sh localhost:2181

[결과] 

 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/kafka   bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
ls /
[zookeeper]

-> 정상 실행 확인 루트내 zookeeper 확인

(05) Kafka 브로커 실행

[설정 파일 수정] 

vim config/server.properties

로컬 서버 동작을 위해 127.0.0.1 로 리스너 활성화 

...
31 #listeners=PLAINTEXT://:9092 
---
31 listeners=PLAINTEXT://127.0.0.1:9092

[브로커 실행]

bin/kafka-server-start.sh config/server.properties
 jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  bin/kafka-server-start.sh config/server.properties
[2023-08-01 10:37:25,242] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-08-01 10:37:25,406] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-08-01 10:37:25,451] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2023-08-01 10:37:25,453] INFO starting (kafka.server.KafkaServer)
[2023-08-01 10:37:25,453] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2023-08-01 10:37:25,462] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
...
[2023-08-01 10:37:26,086] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT://127.0.0.1:9092, czxid (broker epoch): 27 (kafka.zk.KafkaZkClient)
[2023-08-01 10:37:26,117] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
...
[2023-08-01 10:37:26,193] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2023-08-01 10:37:26,251] INFO [broker-0-to-controller-send-thread]: Recorded new controller, from now on will use broker 127.0.0.1:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)

(06) “TEST” 토픽 생성 테스트

[토픽 생성]

bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
Created topic test.
 jay@JayLees-MacBook-Pro  ~/kafka-test/kafka 

[토픽 리스트 확인]

bin/kafka-topics.sh --list --bootstrap-server localhost:9092
 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  bin/kafka-topics.sh --list --bootstrap-server localhost:9092
test

토픽 생성시 주의할 점

  1. 글자수 제한이 있다
  2. “.” 과 같은 특수 문자는 사용하지 말자

(07) 데이터 넣어보기 (프로듀서 해보기)

[명령]

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test

[메세지 입력하기]

test msg 01
test msg 02

[결과]

 jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
>test msg 01
>test msg 02

(08) 데이터 확인하기 (컨슈머 해보기)

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
  • from-begining 을 넣는 이유
    기본적으로 컨슈머 콘솔이 붙을 때에는 붙는 시점 이후의 데이터를 읽게 된다. 붙이지 않으면 과거에 데이터들을 무시하는 만큼 처음 시점 데이터가 필요한 경우 넣는다.

[결과]

 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
test msg 01
test msg 02

(09) 주키퍼 재확인 하기

bin/zookeeper-shell.sh localhost:2181
 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/kafka  bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]
ls /brokers
[ids, seqid, topics]
ls /brokers/ids
[0] # 브로커 등록시 아이디 정보로 들어 갔다. 
 ls /brokers/topics
[__consumer_offsets, test] # consumer_offset 의 대한 토픽은 기본적으로 지정 된다.

쉘에 다양한 카프카 관련 요소들이 생성된 것을 알수 있다.

02) Confluence Kafka 설치 (클러스터 방식)

  • Docker Compose 를 활용하여 진행할 계획

(01) docker-compose 실행

[m1 / intel 모두 호환 가능 ] (7 버전 대부터 ARM 이 지원되기 시작하였고, 7.2 부터는 1개 버전에 통합되어 지원이 가능하다. ) 

git clone https://github.com/conduktor/kafka-stack-docker-compose.git
cd [git 폴더]

[ compose 살펴 보기 ]

version: '2.1'

services:
  zoo1:
    image: confluentinc/cp-zookeeper:7.3.2
    hostname: zoo1
    container_name: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888

  zoo2:
    image: confluentinc/cp-zookeeper:7.3.2
    hostname: zoo2
    container_name: zoo2
    ports:
      - "2182:2182"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2182
      ZOOKEEPER_SERVER_ID: 2
      ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888

  zoo3:
    image: confluentinc/cp-zookeeper:7.3.2
    hostname: zoo3
    container_name: zoo3
    ports:
      - "2183:2183"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2183
      ZOOKEEPER_SERVER_ID: 3
      ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888



  kafka1:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1
      - zoo2
      - zoo3

  kafka2:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka2
    container_name: kafka2
    ports:
      - "9093:9093"
      - "29093:29093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:19093,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093,DOCKER://host.docker.internal:29093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
      KAFKA_BROKER_ID: 2
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1
      - zoo2
      - zoo3

  kafka3:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka3
    container_name: kafka3
    ports:
      - "9094:9094"
      - "29094:29094"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:19094,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094,DOCKER://host.docker.internal:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
      KAFKA_BROKER_ID: 3
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1
      - zoo2
      - zoo3

(02) docker-compose 실행

docker compose -f zk-multiple-kafka-multiple.yml up

docker compose 결과 확인

docker ps 
jay@JayLees-MacBook-Pro  ~/kafka-test/resources   main  docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS          PORTS                                                                                                        NAMES
cc0c291695a7   confluentinc/cp-kafka:6.2.0       "/etc/confluent/dock…"   55 seconds ago   Up 52 seconds   9092/tcp, 0.0.0.0:39092->39092/tcp                                                                           resources-kafka-3-1
b52a917cdea5   confluentinc/cp-kafka:6.2.0       "/etc/confluent/dock…"   55 seconds ago   Up 52 seconds   9092/tcp, 0.0.0.0:29092->29092/tcp                                                                           resources-kafka-2-1
3f4e7494df25   confluentinc/cp-kafka:6.2.0       "/etc/confluent/dock…"   55 seconds ago   Up 52 seconds   9092/tcp, 0.0.0.0:19092->19092/tcp                                                                           resources-kafka-1-1
2909c071e03e   confluentinc/cp-zookeeper:6.2.0   "/etc/confluent/dock…"   56 seconds ago   Up 53 seconds   2181/tcp, 2888/tcp, 0.0.0.0:32181->32181/tcp, 0.0.0.0:42888->42888/tcp, 3888/tcp, 0.0.0.0:43888->43888/tcp   resources-zookeeper-3-1
1ca731a489ce   confluentinc/cp-zookeeper:6.2.0   "/etc/confluent/dock…"   56 seconds ago   Up 53 seconds   2181/tcp, 2888/tcp, 0.0.0.0:12181->12181/tcp, 0.0.0.0:22888->22888/tcp, 3888/tcp, 0.0.0.0:23888->23888/tcp   resources-zookeeper-1-1
f40153edf6f2   confluentinc/cp-zookeeper:6.2.0   "/etc/confluent/dock…"   56 seconds ago   Up 53 seconds   2181/tcp, 2888/tcp, 0.0.0.0:22181->22181/tcp, 0.0.0.0:32888->32888/tcp, 3888/tcp, 0.0.0.0:33888->33888/tcp   resources-zookeeper-2-1
  • Zookeeper
    포트 : 2191 ~ 2193 (3개)
  • Kafka
    접근 포트 : 9092 ~ 9094 / 29092 ~ 29094

(03) 정상 동작 여부 확인 하기

컨플루언트 다운로드 : Previous Versions – Confluent

[01] Tarball 로 다운로드

[02] 압축 푼 후 -> bin 폴더 접근

[토픽 생성 테스트]

./kafka-topics --bootstrap-server localhost:9092 --create --topic test --partitions 2 --replication-factor 3
  • Partition 과 factor 로 분산 저장 및 HA 유지

[결과]

 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/confluent-kafka/confluent-7.4.0/bin  ./kafka-topics --bootstrap-server localhost:9092 --create --topic test --partitions 2 --replication-factor 3
Created topic test.

[ 프로덕션]

./kafka-console-producer --bootstrap-server localhost:9092 --topic test
 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/confluent-kafka/confluent-7.4.0/bin  ./kafka-console-producer --bootstrap-server localhost:9092 --topic test
>hello
>test123
>good
>

[토픽 / 파티션 분배 현황 확인]

./kafka-topics --describe --bootstrap-server localhost:9093 --topic test
 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/confluent-kafka/confluent-7.4.0/bin  ./kafka-topics --describe --bootstrap-server localhost:9093 --topic test
Topic: test	TopicId: G5YT3ayRRRSer0X5CfB-mQ	PartitionCount: 2	ReplicationFactor: 3	Configs:
	Topic: test	Partition: 0	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: test	Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

리더 정보와 파티션들의 분배 현황 및 ISR 정보를 얻을 수 있다.

파티션은 늘리기만 가능하나, Key 로 맵핑하고 있다면, 늘리지 않는 것이 좋다.

(04) HA TEST (Kafka 1개 제거 )

kafka 컨테이너 ID 확인 

docker ps 
>  docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS          PORTS                                                        NAMES
f0425f3ec9cd   confluentinc/cp-kafka:7.3.2       "/etc/confluent/dock…"   17 minutes ago   Up 17 minutes   0.0.0.0:9094->9094/tcp, 9092/tcp, 0.0.0.0:29094->29094/tcp   kafka3
cd158bad385a   confluentinc/cp-kafka:7.3.2       "/etc/confluent/dock…"   17 minutes ago   Up 17 minutes   0.0.0.0:9093->9093/tcp, 9092/tcp, 0.0.0.0:29093->29093/tcp   kafka2
988faffd3e9c   confluentinc/cp-kafka:7.3.2       "/etc/confluent/dock…"   17 minutes ago   Up 17 minutes   0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp             kafka1
2b9c500d121d   confluentinc/cp-zookeeper:7.3.2   "/etc/confluent/dock…"   17 minutes ago   Up 17 minutes   2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2182->2182/tcp         zoo2
529a197f2cae   confluentinc/cp-zookeeper:7.3.2   "/etc/confluent/dock…"   17 minutes ago   Up 17 minutes   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp                   zoo1
ad37e8d72e92   confluentinc/cp-zookeeper:7.3.2   "/etc/confluent/dock…"   17 minutes ago   Up 17 minutes   2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2183->2183/tcp         zoo3

2번 kafka KILL 

 docker kill -s=SIGKILL cd158bad385a

TOPIC 현황 확인

 ✘ jay@JayLees-MacBook-Pro  ~/kafka-test/confluent-kafka/confluent-7.4.0/bin  ./kafka-topics --describe --bootstrap-server localhost:9092 --topic test
Topic: test	TopicId: G5YT3ayRRRSer0X5CfB-mQ	PartitionCount: 2	ReplicationFactor: 3	Configs:
	Topic: test	Partition: 0	Leader: 3	Replicas: 2,3,1	Isr: 3,1
	Topic: test	Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 3,1

-> 모두 리더가 3번으로 변경 되었다.

(참고) 2번에 접근하면 어떻게 될까 ?

jay@JayLees-MacBook-Pro  ~/kafka-test/confluent-kafka/confluent-7.4.0/bin  ./kafka-topics --describe --bootstrap-server localhost:9093 --topic test
[2023-08-01 15:59:57,955] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-08-01 15:59:58,059] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

연결에 실패 한다.

03. Kafka 설정 살펴 보기

GitHUB LINK : kafka/config at trunk · apache/kafka · GitHub

01) Server.properties

############################# Server Basics #############################

# 브로커의 대한 ID 정보 
broker.id=0

############################# Socket Server Settings #############################

# 2개로 분리한 이유는, 내부와 외부 트래픽을 나눠서 진행하여, 효율화와 보안을 강화할 수 있다. )
# 브로커에서 참조하는 주소 
#listeners=PLAINTEXT://:9092

# 프로듀서 / 컨슈머가 참고하는 주소 
#advertised.listeners=PLAINTEXT://your.host.name:9092

# 서버가 요청이나 응답을 받는데 동작하는 서비스 
num.network.threads=3

# 서버가 클라이언트 요청을 처리할 때 디스크 IO 를 처리하는 쓰레드 
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# 브로커의 데이터 저장 폴더를 지정 하는 것 
log.dirs=/tmp/kafka-logs

# 기본 파티션 개수 
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################
# 로그의 상황에 따른 삭제 정책

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
#log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

[ 프로덕션에서 주로 설정하는 항목]

  • auto.create.topics.enable : 토픽의 자동 생성 여부
  • compression type : 메세지 압축 방식 (동작은 프로듀서가 압축해서 전달하면, 브로커는 그대로 저장 하고, 추후 컨슈머가 읽어와 변환함 )
  • delete.topic.enable : 토픽 삭제의 대한 활성화
  • message.max.bytes : 메세지 페이로드의 가장 큰 사이즈 (브로커에서 전송 받을 데이터의 최대 사이즈 )
  • replica.log.time.max.ms : 팔로워가 리더의 요청 메세지를 얼마만큼 기다려 줄 것인지 [기본값 : 30초]

04. 기타

01) AKHQ (GUI 도구)

GUI 로 카프카의 대한 자원 관리를 효율적으로 수행하게 해줌

(01) 적용 방법

기존 docker-compose 파일 내 제일 하단에 아래 내용 추가 

  akhq:
    image: tchiotludo/akhq:latest
    hostname: akhq
    depends_on:
      - kafka1
      - kafka2
      - kafka3
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            kafka:
              properties:
                bootstrap.servers: kafka1:29092,kafka2:29093,kafka3:29094
    ports:
      - 8080:8080

(02) 화면

노드의 대한 설정이나 토픽의 대한 설정을 GUI 를 통해 진행할 수 있음

02) 관련 커맨드 모음 

# kafka topic 생성
./kafka-topics --bootstrap-server localhost:19092 --create --topic fastcampus --partitions 20 --replication-factor 3

# kafka에 생성된 토픽 리스트 확인
./kafka-topics --bootstrap-server localhost:19092 --list

# 특정 토픽의 파티션 수, 리플리카 수 등의 상세정보 확인
./kafka-topics --describe --bootstrap-server localhost:19092 --topic fastcampus

# kafka 콘솔 컨슈머 실행
./kafka-console-consumer --bootstrap-server localhost:19092 --topic fastcampus --from-beginning

# kafka 콘솔 프로듀서 실행
./kafka-console-producer --bootstrap-server localhost:19092 --topic fastcampus

1,325 thoughts on “kafka 기본 설치와 도커 컴포즈를 활용한 클러스터 구성”

  116. An impressive share! I’ve just forwarded this onto a colleague who had been doing a little research on this. And he actually ordered me dinner simply because I discovered it for him… lol. So allow me to reword this…. Thanks for the meal!! But yeah, thanks for spending some time to discuss this subject here on your blog.

    응답

  121. An intriguing discussion is worth comment. I think that you ought to write more on this topic, it may not be a taboo subject but typically people do not speak about these subjects. To the next! Many thanks.

    응답

  144. I blog often and I seriously thank you for your content. Your article has really peaked my interest. I am going to book mark your website and keep checking for new information about once per week. I subscribed to your RSS feed too.

    응답

  186. I would like to thank you for the efforts you’ve put in writing this blog. I am hoping to see the same high-grade blog posts from you in the future as well. In truth, your creative writing abilities has encouraged me to get my own, personal website now 😉

    응답

  196. Good day! I could have sworn I’ve been to this website before but after going through many of the posts I realized it’s new to me. Nonetheless, I’m definitely pleased I found it and I’ll be book-marking it and checking back regularly.

    응답

  231. I’m extremely pleased to uncover this great site. I need to to thank you for your time just for this fantastic read!! I definitely enjoyed every little bit of it and I have you book marked to look at new information on your blog.

    응답

  232. A fascinating discussion is worth comment. I do believe that you ought to write more on this subject matter, it may not be a taboo subject but generally people do not discuss these topics. To the next! Many thanks.

    응답

  238. This is the perfect webpage for everyone who hopes to find out about this topic. You know so much its almost tough to argue with you (not that I actually will need to…HaHa). You definitely put a fresh spin on a subject that has been written about for a long time. Excellent stuff, just wonderful.

    응답

  248. Having read this I believed it was extremely enlightening. I appreciate you spending some time and energy to put this short article together. I once again find myself personally spending a significant amount of time both reading and commenting. But so what, it was still worth it.

    응답

  262. An outstanding share! I have just forwarded this onto a colleague who has been doing a little research on this. And he actually bought me breakfast simply because I discovered it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending the time to discuss this matter here on your internet site.

    응답

  268. Right here is the right site for anybody who wishes to understand this topic. You know so much its almost hard to argue with you (not that I actually would want to…HaHa). You definitely put a new spin on a subject which has been written about for a long time. Great stuff, just great.

    응답

  296. I absolutely love your blog.. Very nice colors & theme. Did you make this amazing site yourself? Please reply back as I’m attempting to create my very own site and would like to know where you got this from or exactly what the theme is named. Thanks.

    응답

  310. I absolutely love your website.. Great colors & theme. Did you develop this website yourself? Please reply back as I’m wanting to create my own website and want to find out where you got this from or what the theme is named. Many thanks!

    응답

  319. I blog quite often and I seriously thank you for your information. Your article has truly peaked my interest. I’m going to take a note of your website and keep checking for new details about once per week. I opted in for your RSS feed too.

    응답

  333. Greetings, I believe your blog could be having web browser compatibility issues. Whenever I look at your blog in Safari, it looks fine however, if opening in IE, it’s got some overlapping issues. I simply wanted to provide you with a quick heads up! Other than that, fantastic site.

    응답

  356. The next time I read a blog, I hope that it won’t disappoint me just as much as this particular one. After all, Yes, it was my choice to read through, but I actually thought you would have something interesting to talk about. All I hear is a bunch of crying about something you could possibly fix if you were not too busy seeking attention.

    응답

  361. Hello there! This post could not be written any better! Looking at this post reminds me of my previous roommate! He continually kept preaching about this. I will forward this post to him. Pretty sure he’s going to have a great read. Many thanks for sharing!

    응답

  413. Hello there! I could have sworn I’ve been to this site before but after browsing through many of the articles I realized it’s new to me. Regardless, I’m definitely pleased I stumbled upon it and I’ll be bookmarking it and checking back frequently.

    응답

  416. I’m impressed, I have to admit. Seldom do I come across a blog that’s both educative and engaging, and without a doubt, you have hit the nail on the head. The issue is something which not enough folks are speaking intelligently about. I am very happy I stumbled across this during my search for something regarding this.

    응답

  434. Hi there! This blog post couldn’t be written any better! Going through this post reminds me of my previous roommate! He continually kept talking about this. I am going to send this information to him. Pretty sure he will have a great read. Thanks for sharing!

    응답

  440. Hi! I’m Charles. If you’re stuck in a financial Groundhog Day, repeating the very same struggles, let’s break the cycle. The 1K a Day System is your way out, leading you to new early mornings of success and capacity. Get up to something wonderful!

    응답

  444. Hey! I’m Charles, and if you’re tired of the 9-to-5 grind and believe workplace coffee tastes like anguish, I’ve got excellent news for you. Welcome to the 1K a Day System, where we switch coffee for cash flow and desks for monetary self-reliance. Are you prepared to sell your tie for a ticket to freedom? Let’s turbocharge your revenues and have some enjoyable along the method!

    응답

  449. After I originally commented I seem to have clicked the -Notify me when new comments are added- checkbox and from now on every time a comment is added I recieve four emails with the same comment. Is there an easy method you are able to remove me from that service? Thanks a lot.

    응답

  450. Hi there! Just felt like dropping in to express my appreciation for your awesome blog. Your insights on affiliate marketing are genuinely commendable. Earning an income from home has never been more accessible with affiliate promotion. It’s all about leveraging your internet presence and promoting items or services that resonate with your audience. Your blog is a treasured resource for anyone exploring affiliate marketing. Keep up the fantastic work!

    응답

  453. In today’s hectic world, remaining ahead implies watching out for new offers that can boost our lives. Whether it’s the latest tech device or an unique deal on health products, brand-new deals bring excitement and possibility. At [Your Business Name], we’re dedicated to bringing you the most innovative and luring offers available. From limited-time discount rates to unique bundles, we’re devoted to supplying worth and convenience to our clients. Stay tuned to our website and social networks channels to be the very first to understand about our latest deals. With [Your Business Call], you’ll never ever miss out on an opportunity to experience something brand-new and remarkable.

    응답

  462. In today’s busy world, remaining ahead implies watching out for new offers that can improve our lives. Whether it’s the current tech gizmo or a special offer on wellness items, brand-new deals bring excitement and possibility. At [Your Business Call], we’re dedicated to bringing you the most ingenious and enticing offers readily available. From limited-time discount rates to unique bundles, we’re devoted to offering worth and convenience to our clients. Stay tuned to our website and social media channels to be the first to know about our most recent offers. With [Your Company Call], you’ll never lose out on a chance to experience something brand-new and amazing.

    응답

  469. After looking over a few of the articles on your website, I honestly like your way of writing a blog. I book-marked it to my bookmark webpage list and will be checking back soon. Please visit my website too and let me know what you think.

    응답

  497. I really love your site.. Pleasant colors & theme. Did you create this website yourself? Please reply back as I’m attempting to create my own personal site and would love to know where you got this from or exactly what the theme is named. Thanks.

    응답

  498. Hi there, I believe your web site could possibly be having browser compatibility issues. When I look at your website in Safari, it looks fine however, when opening in Internet Explorer, it has some overlapping issues. I just wanted to provide you with a quick heads up! Besides that, fantastic blog!

    응답

  514. Hi there! I could have sworn I’ve been to this web site before but after browsing through many of the articles I realized it’s new to me. Anyways, I’m certainly pleased I found it and I’ll be book-marking it and checking back regularly!

    응답

  527. I blog frequently and I seriously appreciate your information. This great article has really peaked my interest. I am going to book mark your website and keep checking for new information about once a week. I opted in for your RSS feed as well.

    응답

  530. I blog quite often and I really thank you for your information. Your article has really peaked my interest. I’m going to take a note of your site and keep checking for new information about once per week. I subscribed to your Feed as well.

    응답

  531. I blog frequently and I seriously appreciate your content. Your article has really peaked my interest. I’m going to take a note of your blog and keep checking for new details about once a week. I subscribed to your Feed as well.

    응답

  558. I’m amazed, I have to admit. Rarely do I encounter a blog that’s both equally educative and interesting, and let me tell you, you have hit the nail on the head. The issue is something too few people are speaking intelligently about. I’m very happy I came across this during my search for something relating to this.

    응답

  562. I’m amazed, I must say. Seldom do I encounter a blog that’s equally educative and entertaining, and let me tell you, you’ve hit the nail on the head. The problem is something too few folks are speaking intelligently about. Now i’m very happy that I came across this during my hunt for something relating to this.

    응답

  614. I’d like to thank you for the efforts you have put in penning this website. I really hope to see the same high-grade content by you later on as well. In truth, your creative writing abilities has motivated me to get my own website now 😉

    응답

  659. An outstanding share! I’ve just forwarded this onto a co-worker who was conducting a little research on this. And he in fact ordered me lunch simply because I discovered it for him… lol. So let me reword this…. Thank YOU for the meal!! But yeah, thanx for spending time to discuss this topic here on your blog.

    응답

  666. I’m impressed, I must say. Rarely do I come across a blog that’s both equally educative and entertaining, and let me tell you, you have hit the nail on the head. The problem is something which not enough folks are speaking intelligently about. I’m very happy I came across this during my search for something concerning this.

    응답

  671. I’m impressed, I must say. Rarely do I come across a blog that’s both educative and amusing, and without a doubt, you have hit the nail on the head. The issue is something not enough men and women are speaking intelligently about. I’m very happy that I found this in my hunt for something concerning this.

    응답

  722. Hello! I could have sworn I’ve visited this website before but after looking at many of the articles I realized it’s new to me. Anyways, I’m definitely delighted I came across it and I’ll be book-marking it and checking back frequently!

    응답

  773. I seriously love your site.. Great colors & theme. Did you create this site yourself? Please reply back as I’m planning to create my own personal website and would love to find out where you got this from or what the theme is called. Thanks!

    응답

  793. Can I just say what a comfort to find an individual who truly understands what they’re talking about on the internet. You actually understand how to bring an issue to light and make it important. More people have to look at this and understand this side of the story. I can’t believe you aren’t more popular given that you certainly possess the gift.

    응답

  813. Hello, I believe your web site might be having web browser compatibility issues. Whenever I take a look at your blog in Safari, it looks fine however when opening in I.E., it has some overlapping issues. I merely wanted to provide you with a quick heads up! Other than that, fantastic blog.

    응답

  870. I loved as much as youu will receive carried oout right
    here.Thhe sketch iis tasteful, yiur aauthored material stylish.
    nonetheless, youu command get ggot an impatience ovesr that yyou wisdh bee deluvering tthe following.
    unwell unquestionably come more foormerly again ass exactly thee same narly vedy often inside ccase yyou shield this increase.

    응답

  874. Платформа 1вин предлагает широкий выбор спортивных событий, киберспорта и азартных игр. Пользователи получают высокие коэффициенты, быстрые выплаты и круглосуточную поддержку. Программа лояльности и бонусы делают игру выгоднее.

    응답

  925. Merci pour cet article super intéressant sur [thème de l’article] ! Je voulais juste ajouter un point qui pourrait intéresser certains d’entre vous. Si vous êtes curieux ou cherchez des informations supplémentaires sur les produits liés à l’amélioration de l’expérience de discuter et de futur rencontre et du bien-être personnel, j’ai récemment découvert un site très complet, [Chemsexworld.com]( Chemsexworld.com .?

    응답

  926. Ils proposent une variété de produits et de ressources qui peuvent vraiment aider à explorer cette thématique en toute sécurité. Ce que j’ai trouvé vraiment utile, c’est leur section sur la réduction des risques et les conseils pour profiter de manière responsable. Ça pourrait être un bon complément à cet article !

    응답

  929. Merci pour cet article super intéressant sur [thème de l’article] ! Je voulais juste ajouter un point qui pourrait intéresser certains d’entre vous. Si vous êtes curieux ou cherchez des informations supplémentaires sur les produits liés à l’amélioration de l’expérience de discuter et de futur rencontre et du bien-être personnel, j’ai récemment découvert un site très complet, [Chemsexworld.com]( Chemsexworld.com .?

    응답

  934. Merci pour cet article super intéressant sur [thème de l’article] ! Je voulais juste ajouter un point qui pourrait intéresser certains d’entre vous. Si vous êtes curieux ou cherchez des informations supplémentaires sur les produits liés à l’amélioration de l’expérience de discuter et de futur rencontre et du bien-être personnel, j’ai récemment découvert un site très complet, [Chemsexworld.com]( Chemsexworld.com .?

    응답

  941. Профессиональный сервисный центр по ремонту компьютерных видеокарт по Москве.
    Мы предлагаем: сколько стоит ремонт видеокарты
    Наши мастера оперативно устранят неисправности вашего устройства в сервисе или с выездом на дом!

    응답

  950. Ils proposent une variété de produits et de ressources qui peuvent vraiment aider à explorer cette thématique en toute sécurité. Ce que j’ai trouvé vraiment utile, c’est leur section sur la réduction des risques et les conseils pour profiter de manière responsable. Ça pourrait être un bon complément à cet article !

    응답

  959. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답

  961. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답

  970. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답

  973. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답

  976. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답

  981. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답

  984. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답

  986. I blog quite often and I seriously appreciate your information. This great article has truly peaked my interest. I will take a note of your blog and keep checking for new information about once a week. I opted in for your Feed too.

    응답


  1003. Временная регистрация в Москве: Быстро и Легально!
    Ищете, где оформить временную регистрацию в Москве?
    Мы гарантируем быстрое и легальное оформление без очередей и лишних документов.
    Ваше спокойствие – наша забота!
    Минимум усилий • Максимум удобства • Полная легальность
    Свяжитесь с нами прямо сейчас!
    Временная регистрация в Москве

    응답


  1015. Временная регистрация в Москве: Быстро и Легально!
    Ищете, где оформить временную регистрацию в Москве? Мы гарантируем быстрое и легальное оформление без очередей и лишних документов. Ваше спокойствие – наша забота!
    Минимум усилий • Максимум удобства • Полная легальность
    Свяжитесь с нами прямо сейчас!
    .

    응답


  1026. Временная регистрация в Москве: Быстро и Легально!
    Ищете, где оформить временную регистрацию в Москве? Мы гарантируем быстрое и легальное оформление без очередей и лишних документов. Ваше спокойствие – наша забота!
    Минимум усилий • Максимум удобства • Полная легальность
    Свяжитесь с нами прямо сейчас!
    .

    응답

  1058. Hey there! Thiss is kind oof ooff tolic bbut I nsed some help frfom an established blog.
    Is it diifficult tto sset up your own blog? I’m not verry techincal butt
    I can figure things out pretfty quick. I’m thinking about making
    my own but I’m not sue wherfe to begin. Do you have aany
    points orr suggestions? Thanks

    응답

  1078. Having read this I thought it was extremely enlightening. I appreciate you spending some time and energy to put this article together. I once again find myself personally spending a significant amount of time both reading and posting comments. But so what, it was still worthwhile.

    응답

  1123. As a person that’s constantly been cautious concerning my blood sugar, finding Sugar Protector has been an alleviation. I feel a lot a lot more in control, and my current exams have actually shown positive enhancements.
    Understanding I have a trusted supplement to support my regular provides me assurance.

    I’m so thankful for Sugar Defender’s influence on my health!

    응답

  1148. 37 S, E N 4 Aminophenylsulfonyl 2, 5 dimethyl 4 S N, 3, 3 trimethyl 2 S 3 methyl 2 methylamino 3 phenylbutanamido butanamido hex 2 enamide Compound 886 donde comprar priligy mexico They are only monks in the realm of the soul, and even if can you take meclizine with high blood pressure they are called Tianjiao, they do not have the strength to compete with the fire of the phoenix

    응답

  1173. I’d like to thank you for the efforts you have put in writing this site. I am hoping to view the same high-grade content from you in the future as well. In fact, your creative writing abilities has motivated me to get my very own blog now 😉

    응답

  1183. Howdy! Do you know if they make any plugins to assist with
    Search Engine Optimization? I’m trying to get my website to rank for some targeted keywords but I’m not seeing very
    good gains. If you know of any please share. Many thanks!
    I saw similar blog here: Eco blankets

    응답

  1197. I truly love your site.. Very nice colors & theme. Did you create this amazing site yourself? Please reply back as I’m attempting to create my very own blog and would love to know where you got this from or exactly what the theme is named. Kudos.

    응답

  1263. sugar defender reviews For many years, I’ve battled unpredictable blood sugar level swings that left
    me feeling drained pipes and inactive. However considering that
    integrating Sugar my energy levels are now secure and regular, and
    I no longer hit a wall in the mid-days. I appreciate that it’s a gentle, all-natural technique that does not come with any
    type of unpleasant adverse effects. It’s genuinely changed my every day life.

    응답

  1295. Good post. I learn something totally new and challenging on sites I stumbleupon every day. It’s always useful to read articles from other writers and practice a little something from other websites.

    응답

  1311. I must thank you for the efforts you’ve put in penning this blog. I really hope to see the same high-grade content from you later on as well. In fact, your creative writing abilities has encouraged me to get my own, personal blog now 😉

    응답

  1320. After going over a handful of the articles on your site, I honestly like your technique of blogging. I bookmarked it to my bookmark site list and will be checking back soon. Take a look at my web site too and let me know your opinion.

    응답

Leave a Comment