Table of Contents
01. 사전 개념
(01) Kafka 의 종류
- Apache Kafka [ 자유롭게 사용 가능]
- Confluent Kafka
- 커뮤니티 버전에서도
- 기본 Kafka 에서 지원하지 않는 기능이 지원됨 ( 프리미어 커넥트가 주요함 )
(02) 데이터 파이프 라인 이란 ?
사람의 개입없이 데이터를 오염, 중복 , 유실과 같은 결합 없이 수집 저장 ETL 이 가능하도록 일련의 흐름을 만들어 주는 과정
(03) 시스템 모니터링 방식
![kafka 모니터링](https://jayslog.com/wp-content/uploads/2023/08/image-1024x508.png)
(04) 메세징 허브
![](https://jayslog.com/wp-content/uploads/2023/08/image-1-1024x516.png)
[기존] 기존의 서비스가 1개 시스템의 실패시 전체의 문제를 만들수 있음
![](https://jayslog.com/wp-content/uploads/2023/08/image-2-1024x504.png)
[변경] 특정 프로세스의 실패시에도 해당 프로세스만 문제로 처리 되고, 서비스 마다 자신의 성능에 맞게 설정하여 진행이 가능하다.
02. Kafka 설치와 설정
01) Apache Kafka 설치 (단일 노드)
(01) 다운로드 사이트 접속
다운로드 링크 : Apache Kafka
![](https://jayslog.com/wp-content/uploads/2023/08/image-4.png)
(02) 압축 해제
tar -xvf kafka_2.13-2.8.0.tgz
[결과]
jay@JayLees-MacBook-Pro ~/kafka-test ls
kafka_2.13-2.8.0.tgz
jay@JayLees-MacBook-Pro ~/kafka-test tar -xvf kafka_2.13-2.8.0.tgz
x kafka_2.13-2.8.0/
x kafka_2.13-2.8.0/LICENSE
x kafka_2.13-2.8.0/NOTICE
x kafka_2.13-2.8.0/bin/
x kafka_2.13-2.8.0/bin/zookeeper-shell.sh
x kafka_2.13-2.8.0/bin/kafka-log-dirs.sh
x kafka_2.13-2.8.0/bin/zookeeper-server-stop.sh
x kafka_2.13-2.8.0/bin/kafka-configs.sh
x kafka_2.13-2.8.0/bin/kafka-server-stop.sh
x kafka_2.13-2.8.0/bin/windows/
x kafka_2.13-2.8.0/bin/windows/kafka-delegation-tokens.bat
x kafka_2.13-2.8.0/bin/windows/kafka-producer-perf-test.bat
x kafka_2.13-2.8.0/bin/windows/kafka-run-class.bat
x kafka_2.13-2.8.0/bin/windows/kafka-server-stop.bat
x kafka_2.13-2.8.0/bin/windows/kafka-streams-application-reset.bat
x kafka_2.13-2.8.0/bin/windows/kafka-dump-log.bat
x kafka_2.13-2.8.0/bin/windows/kafka-server-start.bat
(03) 자주 접근을 위해 심볼릭 링크 추가
ln -s kafka_2.13-2.8.0 kafka
jay@JayLees-MacBook-Pro ~/kafka-test ln -s kafka_2.13-2.8.0 kafka
jay@JayLees-MacBook-Pro ~/kafka-test cd kafka
jay@JayLees-MacBook-Pro ~/kafka-test/kafka ls
LICENSE NOTICE bin config libs licenses site-docs
(04) Zookeeper 설정
[Zookeeper 설정 파일 살펴 보기]
![](https://jayslog.com/wp-content/uploads/2023/08/image-5-1024x87.png)
![](https://jayslog.com/wp-content/uploads/2023/08/image-6.png)
- 2181 포트로 서비스 진행함
[zookeeper 실행]
bin/zookeeper-server-start.sh config/zookeeper.properties
[실행 결과]
jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/zookeeper-server-start.sh config/zookeeper.properties
[2023-08-01 10:26:07,293] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,294] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,300] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,300] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2023-08-01 10:26:07,301] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2023-08-01 10:26:07,301] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2023-08-01 10:26:07,301] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
...
[2023-08-01 10:26:07,334] INFO binding to port 0.0.0.0/0.0.0.0:2181
....
[2023-08-01 10:26:07,355] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.Cont
[1차 확인] 2181 포트 바인딩
[2차 확인] Telnet 접속 테스트
jay@JayLees-MacBook-Pro ~ telnet 0 2181
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
[Zookeeper 쉘 접근]
bin/zookeeper-shell.sh localhost:2181
[결과]
✘ jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /
[zookeeper]
-> 정상 실행 확인 루트내 zookeeper 확인
(05) Kafka 브로커 실행
[설정 파일 수정]
vim config/server.properties
로컬 서버 동작을 위해 127.0.0.1 로 리스너 활성화
...
31 #listeners=PLAINTEXT://:9092
---
31 listeners=PLAINTEXT://127.0.0.1:9092
![](https://jayslog.com/wp-content/uploads/2023/08/image-7.png)
[브로커 실행]
bin/kafka-server-start.sh config/server.properties
jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/kafka-server-start.sh config/server.properties
[2023-08-01 10:37:25,242] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-08-01 10:37:25,406] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-08-01 10:37:25,451] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2023-08-01 10:37:25,453] INFO starting (kafka.server.KafkaServer)
[2023-08-01 10:37:25,453] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2023-08-01 10:37:25,462] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
...
[2023-08-01 10:37:26,086] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT://127.0.0.1:9092, czxid (broker epoch): 27 (kafka.zk.KafkaZkClient)
[2023-08-01 10:37:26,117] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
...
[2023-08-01 10:37:26,193] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2023-08-01 10:37:26,251] INFO [broker-0-to-controller-send-thread]: Recorded new controller, from now on will use broker 127.0.0.1:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
(06) “TEST” 토픽 생성 테스트
[토픽 생성]
bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
✘ jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
Created topic test.
jay@JayLees-MacBook-Pro ~/kafka-test/kafka
[토픽 리스트 확인]
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
✘ jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/kafka-topics.sh --list --bootstrap-server localhost:9092
test
토픽 생성시 주의할 점
- 글자수 제한이 있다
- “.” 과 같은 특수 문자는 사용하지 말자
(07) 데이터 넣어보기 (프로듀서 해보기)
[명령]
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
[메세지 입력하기]
test msg 01
test msg 02
[결과]
jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
>test msg 01
>test msg 02
(08) 데이터 확인하기 (컨슈머 해보기)
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
- from-begining 을 넣는 이유
기본적으로 컨슈머 콘솔이 붙을 때에는 붙는 시점 이후의 데이터를 읽게 된다. 붙이지 않으면 과거에 데이터들을 무시하는 만큼 처음 시점 데이터가 필요한 경우 넣는다.
[결과]
✘ jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
test msg 01
test msg 02
(09) 주키퍼 재확인 하기
bin/zookeeper-shell.sh localhost:2181
✘ jay@JayLees-MacBook-Pro ~/kafka-test/kafka bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]
ls /brokers
[ids, seqid, topics]
ls /brokers/ids
[0] # 브로커 등록시 아이디 정보로 들어 갔다.
ls /brokers/topics
[__consumer_offsets, test] # consumer_offset 의 대한 토픽은 기본적으로 지정 된다.
쉘에 다양한 카프카 관련 요소들이 생성된 것을 알수 있다.
02) Confluence Kafka 설치 (클러스터 방식)
- Docker Compose 를 활용하여 진행할 계획
(01) docker-compose 실행
[m1 / intel 모두 호환 가능 ] (7 버전 대부터 ARM 이 지원되기 시작하였고, 7.2 부터는 1개 버전에 통합되어 지원이 가능하다. )
git clone https://github.com/conduktor/kafka-stack-docker-compose.git
cd [git 폴더]
[ compose 살펴 보기 ]
version: '2.1'
services:
zoo1:
image: confluentinc/cp-zookeeper:7.3.2
hostname: zoo1
container_name: zoo1
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
zoo2:
image: confluentinc/cp-zookeeper:7.3.2
hostname: zoo2
container_name: zoo2
ports:
- "2182:2182"
environment:
ZOOKEEPER_CLIENT_PORT: 2182
ZOOKEEPER_SERVER_ID: 2
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
zoo3:
image: confluentinc/cp-zookeeper:7.3.2
hostname: zoo3
container_name: zoo3
ports:
- "2183:2183"
environment:
ZOOKEEPER_CLIENT_PORT: 2183
ZOOKEEPER_SERVER_ID: 3
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
kafka1:
image: confluentinc/cp-kafka:7.3.2
hostname: kafka1
container_name: kafka1
ports:
- "9092:9092"
- "29092:29092"
environment:
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
depends_on:
- zoo1
- zoo2
- zoo3
kafka2:
image: confluentinc/cp-kafka:7.3.2
hostname: kafka2
container_name: kafka2
ports:
- "9093:9093"
- "29093:29093"
environment:
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:19093,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093,DOCKER://host.docker.internal:29093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
KAFKA_BROKER_ID: 2
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
depends_on:
- zoo1
- zoo2
- zoo3
kafka3:
image: confluentinc/cp-kafka:7.3.2
hostname: kafka3
container_name: kafka3
ports:
- "9094:9094"
- "29094:29094"
environment:
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:19094,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094,DOCKER://host.docker.internal:29094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
KAFKA_BROKER_ID: 3
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
depends_on:
- zoo1
- zoo2
- zoo3
(02) docker-compose 실행
docker compose -f zk-multiple-kafka-multiple.yml up
docker compose 결과 확인
docker ps
jay@JayLees-MacBook-Pro ~/kafka-test/resources main docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cc0c291695a7 confluentinc/cp-kafka:6.2.0 "/etc/confluent/dock…" 55 seconds ago Up 52 seconds 9092/tcp, 0.0.0.0:39092->39092/tcp resources-kafka-3-1
b52a917cdea5 confluentinc/cp-kafka:6.2.0 "/etc/confluent/dock…" 55 seconds ago Up 52 seconds 9092/tcp, 0.0.0.0:29092->29092/tcp resources-kafka-2-1
3f4e7494df25 confluentinc/cp-kafka:6.2.0 "/etc/confluent/dock…" 55 seconds ago Up 52 seconds 9092/tcp, 0.0.0.0:19092->19092/tcp resources-kafka-1-1
2909c071e03e confluentinc/cp-zookeeper:6.2.0 "/etc/confluent/dock…" 56 seconds ago Up 53 seconds 2181/tcp, 2888/tcp, 0.0.0.0:32181->32181/tcp, 0.0.0.0:42888->42888/tcp, 3888/tcp, 0.0.0.0:43888->43888/tcp resources-zookeeper-3-1
1ca731a489ce confluentinc/cp-zookeeper:6.2.0 "/etc/confluent/dock…" 56 seconds ago Up 53 seconds 2181/tcp, 2888/tcp, 0.0.0.0:12181->12181/tcp, 0.0.0.0:22888->22888/tcp, 3888/tcp, 0.0.0.0:23888->23888/tcp resources-zookeeper-1-1
f40153edf6f2 confluentinc/cp-zookeeper:6.2.0 "/etc/confluent/dock…" 56 seconds ago Up 53 seconds 2181/tcp, 2888/tcp, 0.0.0.0:22181->22181/tcp, 0.0.0.0:32888->32888/tcp, 3888/tcp, 0.0.0.0:33888->33888/tcp resources-zookeeper-2-1
- Zookeeper
포트 : 2191 ~ 2193 (3개) - Kafka
접근 포트 : 9092 ~ 9094 / 29092 ~ 29094
(03) 정상 동작 여부 확인 하기
컨플루언트 다운로드 : Previous Versions – Confluent
![](https://jayslog.com/wp-content/uploads/2023/08/image-8.png)
[01] Tarball 로 다운로드
[02] 압축 푼 후 -> bin 폴더 접근
[토픽 생성 테스트]
./kafka-topics --bootstrap-server localhost:9092 --create --topic test --partitions 2 --replication-factor 3
- Partition 과 factor 로 분산 저장 및 HA 유지
[결과]
✘ jay@JayLees-MacBook-Pro ~/kafka-test/confluent-kafka/confluent-7.4.0/bin ./kafka-topics --bootstrap-server localhost:9092 --create --topic test --partitions 2 --replication-factor 3
Created topic test.
[ 프로덕션]
./kafka-console-producer --bootstrap-server localhost:9092 --topic test
✘ jay@JayLees-MacBook-Pro ~/kafka-test/confluent-kafka/confluent-7.4.0/bin ./kafka-console-producer --bootstrap-server localhost:9092 --topic test
>hello
>test123
>good
>
[토픽 / 파티션 분배 현황 확인]
./kafka-topics --describe --bootstrap-server localhost:9093 --topic test
✘ jay@JayLees-MacBook-Pro ~/kafka-test/confluent-kafka/confluent-7.4.0/bin ./kafka-topics --describe --bootstrap-server localhost:9093 --topic test
Topic: test TopicId: G5YT3ayRRRSer0X5CfB-mQ PartitionCount: 2 ReplicationFactor: 3 Configs:
Topic: test Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: test Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
리더 정보와 파티션들의 분배 현황 및 ISR 정보를 얻을 수 있다.
파티션은 늘리기만 가능하나, Key 로 맵핑하고 있다면, 늘리지 않는 것이 좋다.
(04) HA TEST (Kafka 1개 제거 )
kafka 컨테이너 ID 확인
docker ps
> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f0425f3ec9cd confluentinc/cp-kafka:7.3.2 "/etc/confluent/dock…" 17 minutes ago Up 17 minutes 0.0.0.0:9094->9094/tcp, 9092/tcp, 0.0.0.0:29094->29094/tcp kafka3
cd158bad385a confluentinc/cp-kafka:7.3.2 "/etc/confluent/dock…" 17 minutes ago Up 17 minutes 0.0.0.0:9093->9093/tcp, 9092/tcp, 0.0.0.0:29093->29093/tcp kafka2
988faffd3e9c confluentinc/cp-kafka:7.3.2 "/etc/confluent/dock…" 17 minutes ago Up 17 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp kafka1
2b9c500d121d confluentinc/cp-zookeeper:7.3.2 "/etc/confluent/dock…" 17 minutes ago Up 17 minutes 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2182->2182/tcp zoo2
529a197f2cae confluentinc/cp-zookeeper:7.3.2 "/etc/confluent/dock…" 17 minutes ago Up 17 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zoo1
ad37e8d72e92 confluentinc/cp-zookeeper:7.3.2 "/etc/confluent/dock…" 17 minutes ago Up 17 minutes 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2183->2183/tcp zoo3
2번 kafka KILL
docker kill -s=SIGKILL cd158bad385a
TOPIC 현황 확인
✘ jay@JayLees-MacBook-Pro ~/kafka-test/confluent-kafka/confluent-7.4.0/bin ./kafka-topics --describe --bootstrap-server localhost:9092 --topic test
Topic: test TopicId: G5YT3ayRRRSer0X5CfB-mQ PartitionCount: 2 ReplicationFactor: 3 Configs:
Topic: test Partition: 0 Leader: 3 Replicas: 2,3,1 Isr: 3,1
Topic: test Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1
-> 모두 리더가 3번으로 변경 되었다.
(참고) 2번에 접근하면 어떻게 될까 ?
jay@JayLees-MacBook-Pro ~/kafka-test/confluent-kafka/confluent-7.4.0/bin ./kafka-topics --describe --bootstrap-server localhost:9093 --topic test
[2023-08-01 15:59:57,955] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2023-08-01 15:59:58,059] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
연결에 실패 한다.
03. Kafka 설정 살펴 보기
GitHUB LINK : kafka/config at trunk · apache/kafka · GitHub
01) Server.properties
############################# Server Basics #############################
# 브로커의 대한 ID 정보
broker.id=0
############################# Socket Server Settings #############################
# 2개로 분리한 이유는, 내부와 외부 트래픽을 나눠서 진행하여, 효율화와 보안을 강화할 수 있다. )
# 브로커에서 참조하는 주소
#listeners=PLAINTEXT://:9092
# 프로듀서 / 컨슈머가 참고하는 주소
#advertised.listeners=PLAINTEXT://your.host.name:9092
# 서버가 요청이나 응답을 받는데 동작하는 서비스
num.network.threads=3
# 서버가 클라이언트 요청을 처리할 때 디스크 IO 를 처리하는 쓰레드
num.io.threads=8
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# 브로커의 데이터 저장 폴더를 지정 하는 것
log.dirs=/tmp/kafka-logs
# 기본 파티션 개수
num.partitions=1
# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# 로그의 상황에 따른 삭제 정책
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168
# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
#log.segment.bytes=1073741824
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
############################# Group Coordinator Settings #############################
# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0
[ 프로덕션에서 주로 설정하는 항목]
- auto.create.topics.enable : 토픽의 자동 생성 여부
- compression type : 메세지 압축 방식 (동작은 프로듀서가 압축해서 전달하면, 브로커는 그대로 저장 하고, 추후 컨슈머가 읽어와 변환함 )
- delete.topic.enable : 토픽 삭제의 대한 활성화
- message.max.bytes : 메세지 페이로드의 가장 큰 사이즈 (브로커에서 전송 받을 데이터의 최대 사이즈 )
- replica.log.time.max.ms : 팔로워가 리더의 요청 메세지를 얼마만큼 기다려 줄 것인지 [기본값 : 30초]
04. 기타
01) AKHQ (GUI 도구)
GUI 로 카프카의 대한 자원 관리를 효율적으로 수행하게 해줌
(01) 적용 방법
기존 docker-compose 파일 내 제일 하단에 아래 내용 추가
akhq:
image: tchiotludo/akhq:latest
hostname: akhq
depends_on:
- kafka1
- kafka2
- kafka3
environment:
AKHQ_CONFIGURATION: |
akhq:
connections:
kafka:
properties:
bootstrap.servers: kafka1:29092,kafka2:29093,kafka3:29094
ports:
- 8080:8080
(02) 화면
노드의 대한 설정이나 토픽의 대한 설정을 GUI 를 통해 진행할 수 있음
![](https://jayslog.com/wp-content/uploads/2023/08/image-9-1024x224.png)
![](https://jayslog.com/wp-content/uploads/2023/08/image-10-1024x527.png)
02) 관련 커맨드 모음
# kafka topic 생성
./kafka-topics --bootstrap-server localhost:19092 --create --topic fastcampus --partitions 20 --replication-factor 3
# kafka에 생성된 토픽 리스트 확인
./kafka-topics --bootstrap-server localhost:19092 --list
# 특정 토픽의 파티션 수, 리플리카 수 등의 상세정보 확인
./kafka-topics --describe --bootstrap-server localhost:19092 --topic fastcampus
# kafka 콘솔 컨슈머 실행
./kafka-console-consumer --bootstrap-server localhost:19092 --topic fastcampus --from-beginning
# kafka 콘솔 프로듀서 실행
./kafka-console-producer --bootstrap-server localhost:19092 --topic fastcampus
yeteneksiz
child porn
pin up yukle: pin up apk yukle – pin up giris
pin-up cazino https://azerbaijancuisine.com/# pin up azerbaijan
pin up az?rbaycan
https://northern-doctors.org/# buying prescription drugs in mexico
mexican online pharmacies prescription drugs: Mexico pharmacy that ship to usa – pharmacies in mexico that ship to usa
reputable mexican pharmacies online: п»їbest mexican online pharmacies – reputable mexican pharmacies online
https://northern-doctors.org/# pharmacies in mexico that ship to usa
mexican border pharmacies shipping to usa: mexico drug stores pharmacies – mexican online pharmacies prescription drugs
mexican rx online mexican pharmacy mexican drugstore online
http://northern-doctors.org/# mexican rx online
mexican pharmacy: northern doctors – purple pharmacy mexico price list
https://northern-doctors.org/# pharmacies in mexico that ship to usa
purple pharmacy mexico price list: mexican northern doctors – mexican pharmaceuticals online
mexico drug stores pharmacies: mexican northern doctors – mexican rx online
https://northern-doctors.org/# mexican drugstore online
best online pharmacies in mexico: mexican northern doctors – mexican mail order pharmacies
best online pharmacies in mexico mexican pharmacy mexican border pharmacies shipping to usa
http://northern-doctors.org/# mexican mail order pharmacies
buying from online mexican pharmacy: п»їbest mexican online pharmacies – pharmacies in mexico that ship to usa
medicine in mexico pharmacies: northern doctors – mexico drug stores pharmacies
https://northern-doctors.org/# mexican rx online
best online pharmacies in mexico: mexican pharmacy – п»їbest mexican online pharmacies
mexico drug stores pharmacies: Mexico pharmacy that ship to usa – pharmacies in mexico that ship to usa
https://northern-doctors.org/# mexican pharmaceuticals online
purple pharmacy mexico price list Mexico pharmacy that ship to usa buying from online mexican pharmacy
reputable mexican pharmacies online: п»їbest mexican online pharmacies – best online pharmacies in mexico
https://northern-doctors.org/# mexican pharmaceuticals online
mexican border pharmacies shipping to usa: mexican pharmacy online – mexican rx online
https://northern-doctors.org/# mexico pharmacy
mexico pharmacies prescription drugs: mexican pharmacy northern doctors – п»їbest mexican online pharmacies
mexican rx online mexican pharmacy online mexican pharmaceuticals online
buying prescription drugs in mexico online: mexican pharmacy northern doctors – buying prescription drugs in mexico
https://northern-doctors.org/# mexico drug stores pharmacies
mexican drugstore online: mexican pharmacy northern doctors – mexican online pharmacies prescription drugs
https://northern-doctors.org/# pharmacies in mexico that ship to usa
п»їbest mexican online pharmacies: mexican northern doctors – mexico drug stores pharmacies
mexican pharmacy: mexican pharmacy online – mexican border pharmacies shipping to usa
https://northern-doctors.org/# medicine in mexico pharmacies
reputable mexican pharmacies online: northern doctors – buying from online mexican pharmacy
buying from online mexican pharmacy: mexican pharmacy northern doctors – mexican pharmacy
reputable mexican pharmacies online mexican pharmacy northern doctors mexico drug stores pharmacies
https://northern-doctors.org/# pharmacies in mexico that ship to usa
mexican border pharmacies shipping to usa: Mexico pharmacy that ship to usa – buying from online mexican pharmacy
https://northern-doctors.org/# buying prescription drugs in mexico online
medicine in mexico pharmacies: п»їbest mexican online pharmacies – mexican mail order pharmacies
best online pharmacies in mexico: mexican northern doctors – buying prescription drugs in mexico
https://northern-doctors.org/# mexican drugstore online
mexican rx online: mexican northern doctors – mexican mail order pharmacies
best online pharmacies in mexico: northern doctors pharmacy – mexican rx online
https://northern-doctors.org/# mexico pharmacies prescription drugs
mexico drug stores pharmacies mexican pharmacy online mexico drug stores pharmacies
pharmacies in mexico that ship to usa: northern doctors pharmacy – mexican mail order pharmacies
http://northern-doctors.org/# medicine in mexico pharmacies
purple pharmacy mexico price list: mexican rx online – best online pharmacies in mexico
buying from online mexican pharmacy: mexican pharmacy – mexico pharmacy
https://northern-doctors.org/# mexican pharmaceuticals online
п»їbest mexican online pharmacies: northern doctors – best online pharmacies in mexico
http://northern-doctors.org/# mexico pharmacies prescription drugs
mexican border pharmacies shipping to usa: northern doctors – reputable mexican pharmacies online
buying from online mexican pharmacy mexican northern doctors buying prescription drugs in mexico online
mexico drug stores pharmacies: mexican pharmacy northern doctors – mexican border pharmacies shipping to usa
mexican border pharmacies shipping to usa cmq pharma mexican drugstore online
http://cmqpharma.com/# п»їbest mexican online pharmacies
mexican rx online
buying prescription drugs in mexico online
http://cmqpharma.com/# medication from mexico pharmacy
best online pharmacies in mexico
medication from mexico pharmacy mexican online pharmacy mexican rx online
mexico pharmacy cmq pharma mexican pharmaceuticals online
п»їbest mexican online pharmacies buying from online mexican pharmacy mexican pharmaceuticals online
pharmacies in mexico that ship to usa mexico drug stores pharmacies pharmacies in mexico that ship to usa
mexican pharmacy: cmq pharma – п»їbest mexican online pharmacies
buying prescription drugs in mexico online cmqpharma.com mexican pharmacy
buying prescription drugs in mexico cmq mexican pharmacy online purple pharmacy mexico price list
mexican mail order pharmacies cmqpharma.com mexico pharmacy
mexican pharmacy cmqpharma.com mexico pharmacy
http://cmqpharma.com/# reputable mexican pharmacies online
mexican drugstore online
mexico pharmacies prescription drugs cmq pharma mexican pharmaceuticals online
Very nice info and right to the point. I don’t know if this is actually the best place to ask but do you people have any ideea where to employ some professional writers?
Thanks in advance 🙂 Najlepsze escape roomy
I like this web blog it’s a master piece! Glad I discovered this on google..
buying prescription drugs in mexico online: mexican pharmacy – best online pharmacies in mexico
п»їbest mexican online pharmacies
https://cmqpharma.com/# buying from online mexican pharmacy
buying from online mexican pharmacy
top 10 online pharmacy in india world pharmacy india india pharmacy
best canadian pharmacy online: vipps canadian pharmacy – pharmacy in canada
http://indiapharmast.com/# reputable indian online pharmacy
online shopping pharmacy india: top 10 pharmacies in india – online pharmacy india
pharmacy website india india online pharmacy top online pharmacy india
best online canadian pharmacy: canadian mail order pharmacy – canada pharmacy reviews
mexican border pharmacies shipping to usa: mexico drug stores pharmacies – mexican online pharmacies prescription drugs
indian pharmacy paypal: Online medicine order – reputable indian online pharmacy
top 10 pharmacies in india top online pharmacy india india pharmacy mail order
http://foruspharma.com/# mexican online pharmacies prescription drugs
mexican pharmacy: mexican mail order pharmacies – mexican drugstore online
mexican pharmaceuticals online: buying from online mexican pharmacy – mexican online pharmacies prescription drugs
buy prescription drugs from india: indian pharmacy – п»їlegitimate online pharmacies india
indian pharmacy india online pharmacy Online medicine order
https://indiapharmast.com/# top 10 online pharmacy in india
п»їlegitimate online pharmacies india: online shopping pharmacy india – indian pharmacy online
ed drugs online from canada: best rated canadian pharmacy – canadianpharmacyworld
Excellent post. I am experiencing some of these issues as well..
reputable canadian pharmacy: canadian mail order pharmacy – canadian pharmacy no scripts
https://paxloviddelivery.pro/# paxlovid india
doxycycline 75 mg tablet: doxycycline pills for sale – buy doxycycline online 270 tabs
Pretty! This was an incredibly wonderful article. Thank you for providing these details.
https://clomiddelivery.pro/# can i order generic clomid without insurance
paxlovid india: Paxlovid over the counter – paxlovid india
I really like reading through a post that will make men and women think. Also, thank you for allowing me to comment.
http://amoxildelivery.pro/# amoxicillin 500mg prescription
http://ciprodelivery.pro/# cipro online no prescription in the usa
doxy 100: doxycycline brand name in india – where can i buy doxycycline over the counter
http://ciprodelivery.pro/# ciprofloxacin 500 mg tablet price
doxycycline buy online usa: where can i get doxycycline online – doxycycline cap 40mg
http://ciprodelivery.pro/# antibiotics cipro
https://amoxildelivery.pro/# over the counter amoxicillin
amoxicillin price without insurance: buy amoxicillin online with paypal – amoxicillin 200 mg tablet
http://paxloviddelivery.pro/# Paxlovid buy online
An impressive share! I’ve just forwarded this onto a colleague who had been doing a little research on this. And he actually ordered me dinner simply because I discovered it for him… lol. So allow me to reword this…. Thanks for the meal!! But yeah, thanks for spending some time to discuss this subject here on your blog.
https://doxycyclinedelivery.pro/# doxycycline for sale over the counter
where can i buy generic clomid without prescription: get clomid now – how can i get cheap clomid without prescription
doxy 200: price of doxycycline – doxycycline 150 mg capsules
can we buy amoxcillin 500mg on ebay without prescription: buy cheap amoxicillin – buy amoxicillin online with paypal
An intriguing discussion is worth comment. I think that you ought to write more on this topic, it may not be a taboo subject but typically people do not speak about these subjects. To the next! Many thanks.
mexican pharmacy medicine in mexico pharmacies medication from mexico pharmacy
http://mexicandeliverypharma.com/# buying from online mexican pharmacy
mexican drugstore online: mexican online pharmacies prescription drugs – mexico drug stores pharmacies
mexico pharmacies prescription drugs: mexico drug stores pharmacies – reputable mexican pharmacies online
buying prescription drugs in mexico online medicine in mexico pharmacies п»їbest mexican online pharmacies
reputable mexican pharmacies online: mexico drug stores pharmacies – buying prescription drugs in mexico online
reputable mexican pharmacies online: buying from online mexican pharmacy – mexican drugstore online
https://mexicandeliverypharma.com/# mexican drugstore online
mexico pharmacy mexican pharmaceuticals online best online pharmacies in mexico
child porn
mexico pharmacies prescription drugs: medication from mexico pharmacy – mexico drug stores pharmacies
mexican rx online: medicine in mexico pharmacies – reputable mexican pharmacies online
https://mexicandeliverypharma.com/# buying from online mexican pharmacy
mexican pharmacy medicine in mexico pharmacies purple pharmacy mexico price list
п»їbest mexican online pharmacies: mexican rx online – reputable mexican pharmacies online
mexican border pharmacies shipping to usa: mexican drugstore online – mexico drug stores pharmacies
reputable mexican pharmacies online: mexico drug stores pharmacies – mexico drug stores pharmacies
mexico pharmacy mexico pharmacy best online pharmacies in mexico
mexican mail order pharmacies: mexican rx online – buying prescription drugs in mexico online
medicine in mexico pharmacies: reputable mexican pharmacies online – mexican drugstore online
medicine in mexico pharmacies: mexican rx online – mexican pharmaceuticals online
https://mexicandeliverypharma.online/# mexican border pharmacies shipping to usa
I blog often and I seriously thank you for your content. Your article has really peaked my interest. I am going to book mark your website and keep checking for new information about once per week. I subscribed to your RSS feed too.
mexican border pharmacies shipping to usa mexico drug stores pharmacies pharmacies in mexico that ship to usa
Good information. Lucky me I recently found your site by accident (stumbleupon). I have saved it for later!
mexican drugstore online: mexican online pharmacies prescription drugs – medication from mexico pharmacy
Hi, I believe your website might be having internet browser compatibility problems. Whenever I look at your web site in Safari, it looks fine but when opening in I.E., it’s got some overlapping issues. I just wanted to give you a quick heads up! Other than that, wonderful website.
mexican rx online: buying from online mexican pharmacy – buying from online mexican pharmacy
mexican pharmaceuticals online: buying prescription drugs in mexico online – mexican online pharmacies prescription drugs
This is a topic that is near to my heart… Thank you! Where can I find the contact details for questions?
You ought to take part in a contest for one of the finest sites on the internet. I will highly recommend this website!
best online pharmacies in mexico mexican pharmacy buying prescription drugs in mexico
mexican border pharmacies shipping to usa: pharmacies in mexico that ship to usa – mexican mail order pharmacies
pharmacies in mexico that ship to usa: mexico pharmacies prescription drugs – buying from online mexican pharmacy
mexican drugstore online: mexico drug stores pharmacies – buying prescription drugs in mexico
п»їbest mexican online pharmacies mexico pharmacies prescription drugs mexican rx online
mexican mail order pharmacies: п»їbest mexican online pharmacies – reputable mexican pharmacies online
mexican mail order pharmacies: mexican pharmaceuticals online – purple pharmacy mexico price list
mexico drug stores pharmacies: buying from online mexican pharmacy – medication from mexico pharmacy
mexican online pharmacies prescription drugs mexico pharmacies prescription drugs mexican mail order pharmacies
mexican online pharmacies prescription drugs: buying prescription drugs in mexico online – reputable mexican pharmacies online
mexico drug stores pharmacies: reputable mexican pharmacies online – mexico drug stores pharmacies
reputable mexican pharmacies online: pharmacies in mexico that ship to usa – mexican mail order pharmacies
best online pharmacies in mexico mexico drug stores pharmacies mexican border pharmacies shipping to usa
mexican pharmaceuticals online: medicine in mexico pharmacies – mexican pharmaceuticals online
buying prescription drugs in mexico online: buying from online mexican pharmacy – mexico pharmacies prescription drugs
pharmacies in mexico that ship to usa: mexican pharmaceuticals online – mexican mail order pharmacies
mexican rx online purple pharmacy mexico price list mexican rx online
mexico drug stores pharmacies: pharmacies in mexico that ship to usa – mexico pharmacies prescription drugs
mexico drug stores pharmacies: reputable mexican pharmacies online – buying prescription drugs in mexico online
mexico drug stores pharmacies: purple pharmacy mexico price list – mexican border pharmacies shipping to usa
mexican pharmacy mexican rx online mexico pharmacy
buying prescription drugs in mexico: mexico pharmacies prescription drugs – mexican pharmaceuticals online
purple pharmacy mexico price list: mexico drug stores pharmacies – buying prescription drugs in mexico online
mexican drugstore online mexican mail order pharmacies mexico drug stores pharmacies
best online pharmacies in mexico: mexico pharmacies prescription drugs – mexican pharmaceuticals online
medication from mexico pharmacy: buying prescription drugs in mexico – best online pharmacies in mexico
mexican border pharmacies shipping to usa: mexico pharmacies prescription drugs – medicine in mexico pharmacies
mexican online pharmacies prescription drugs mexican drugstore online mexican mail order pharmacies
buying from online mexican pharmacy: mexico drug stores pharmacies – mexican pharmaceuticals online
mexico pharmacies prescription drugs: medicine in mexico pharmacies – buying prescription drugs in mexico
medication from mexico pharmacy: mexican pharmaceuticals online – medication from mexico pharmacy
Your style is really unique in comparison to other people I have read stuff from. I appreciate you for posting when you have the opportunity, Guess I’ll just book mark this site.
purple pharmacy mexico price list п»їbest mexican online pharmacies buying prescription drugs in mexico online
I would like to thank you for the efforts you’ve put in writing this blog. I am hoping to see the same high-grade blog posts from you in the future as well. In truth, your creative writing abilities has encouraged me to get my own, personal website now 😉
mexico drug stores pharmacies: mexican drugstore online – purple pharmacy mexico price list
buying prescription drugs in mexico online: purple pharmacy mexico price list – best online pharmacies in mexico
Your style is so unique in comparison to other people I have read stuff from. Thank you for posting when you’ve got the opportunity, Guess I’ll just book mark this site.
mexican border pharmacies shipping to usa: buying prescription drugs in mexico online – reputable mexican pharmacies online
Your style is really unique compared to other folks I have read stuff from. Many thanks for posting when you’ve got the opportunity, Guess I’ll just book mark this blog.
mexican rx online pharmacies in mexico that ship to usa purple pharmacy mexico price list
Hello there! I just want to give you a big thumbs up for the excellent info you’ve got right here on this post. I am returning to your web site for more soon.
mexican online pharmacies prescription drugs: purple pharmacy mexico price list – mexico pharmacies prescription drugs
medicine in mexico pharmacies: mexican mail order pharmacies – medication from mexico pharmacy
Good day! I could have sworn I’ve been to this website before but after going through many of the posts I realized it’s new to me. Nonetheless, I’m definitely pleased I found it and I’ll be book-marking it and checking back regularly.
buying prescription drugs in mexico: mexico drug stores pharmacies – mexican mail order pharmacies
medication from mexico pharmacy best online pharmacies in mexico mexican pharmacy
Hi! I just would like to offer you a huge thumbs up for the excellent info you have here on this post. I will be coming back to your site for more soon.
There’s definately a lot to learn about this subject. I love all the points you’ve made.
mexican border pharmacies shipping to usa: buying prescription drugs in mexico online – mexican rx online
buying prescription drugs in mexico: mexico drug stores pharmacies – mexico drug stores pharmacies
This web site truly has all the information I wanted concerning this subject and didn’t know who to ask.
medication from mexico pharmacy: mexican rx online – buying prescription drugs in mexico
There’s definately a lot to know about this issue. I love all of the points you have made.
pharmacies in mexico that ship to usa mexican online pharmacies prescription drugs mexican mail order pharmacies
mexican online pharmacies prescription drugs: purple pharmacy mexico price list – mexican drugstore online
buying prescription drugs in mexico online: п»їbest mexican online pharmacies – pharmacies in mexico that ship to usa
mexican pharmaceuticals online: pharmacies in mexico that ship to usa – mexico pharmacies prescription drugs
mexican border pharmacies shipping to usa buying prescription drugs in mexico mexico drug stores pharmacies
Saved as a favorite, I like your website!
zithromax 500 buy generic zithromax online zithromax 250
order cheap propecia without insurance: propecia without dr prescription – get cheap propecia pill
https://zithromaxbestprice.pro/# generic zithromax medicine
http://zithromaxbestprice.pro/# buy zithromax no prescription
I couldn’t resist commenting. Well written.
Abortion pills online Abortion pills online Misoprostol 200 mg buy online
how to get prednisone tablets: prednisone 50 mg price – prednisone uk
https://cytotecbestprice.pro/# Misoprostol 200 mg buy online
I love reading a post that can make men and women think. Also, thanks for allowing for me to comment.
http://propeciabestprice.pro/# cost generic propecia without prescription
order cheap propecia without prescription cheap propecia tablets buying generic propecia without dr prescription
purchase cytotec: cytotec online – buy cytotec over the counter
https://prednisonebestprice.pro/# prednisone without rx
https://cytotecbestprice.pro/# Cytotec 200mcg price
prednisone 10 tablet: prednisone 300mg – prednisone 2 5 mg
nolvadex vs clomid tamoxifen headache tamoxifen vs clomid
nolvadex 10mg: tamoxifen endometrium – tamoxifen cancer
I was able to find good advice from your content.
buy cytotec pills: buy cytotec online – buy cytotec pills
I’m extremely pleased to uncover this great site. I need to to thank you for your time just for this fantastic read!! I definitely enjoyed every little bit of it and I have you book marked to look at new information on your blog.
A fascinating discussion is worth comment. I do believe that you ought to write more on this subject matter, it may not be a taboo subject but generally people do not discuss these topics. To the next! Many thanks.
http://prednisonebestprice.pro/# can you buy prednisone
Great info. Lucky me I came across your blog by chance (stumbleupon). I have saved it for later.
tamoxifen premenopausal: tamoxifen hip pain – nolvadex pct
I was able to find good advice from your blog posts.
It’s hard to come by knowledgeable people on this subject, but you seem like you know what you’re talking about! Thanks
This is the perfect webpage for everyone who hopes to find out about this topic. You know so much its almost tough to argue with you (not that I actually will need to…HaHa). You definitely put a fresh spin on a subject that has been written about for a long time. Excellent stuff, just wonderful.
zithromax cost canada: zithromax capsules price – zithromax over the counter
I like it whenever people get together and share views. Great site, keep it up!
tamoxifen 20 mg tablet: tamoxifen hair loss – tamoxifen pill
https://prednisonebestprice.pro/# prednisone 5 mg tablet
There is certainly a great deal to know about this topic. I really like all the points you’ve made.
cheap propecia no prescription: cheap propecia online – cost of cheap propecia without a prescription
Greetings! Very helpful advice in this particular post! It is the little changes which will make the most significant changes. Thanks for sharing!
2.5 mg prednisone daily: prednisone generic brand name – where can i get prednisone
Pretty! This was a really wonderful post. Thanks for supplying these details.
Having read this I believed it was extremely enlightening. I appreciate you spending some time and energy to put this short article together. I once again find myself personally spending a significant amount of time both reading and commenting. But so what, it was still worth it.
Farmacie on line spedizione gratuita: Farmacia online piu conveniente – farmacie online sicure
Everything is very open with a very clear explanation of the issues. It was definitely informative. Your site is extremely helpful. Thanks for sharing!
https://viagragenerico.site/# viagra ordine telefonico
Greetings! Very helpful advice within this post! It’s the little changes that will make the most significant changes. Many thanks for sharing!
Farmacia online piГ№ conveniente: kamagra oral jelly consegna 24 ore – п»їFarmacia online migliore
Farmacia online piГ№ conveniente: farmacia online senza ricetta – farmaci senza ricetta elenco
Farmacie online sicure: Avanafil a cosa serve – farmacie online affidabili
Everyone loves it when folks get together and share views. Great site, stick with it!
farmacia online: farmacia online migliore – comprare farmaci online all’estero
http://farmait.store/# farmaci senza ricetta elenco
acquisto farmaci con ricetta: farmacia online migliore – Farmacia online miglior prezzo
Your style is unique in comparison to other folks I have read stuff from. Thanks for posting when you’ve got the opportunity, Guess I’ll just bookmark this page.
farmacia online: Farmacia online migliore – top farmacia online
An outstanding share! I have just forwarded this onto a colleague who has been doing a little research on this. And he actually bought me breakfast simply because I discovered it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanx for spending the time to discuss this matter here on your internet site.
I was able to find good advice from your blog posts.
farmacie online affidabili: Farmacie on line spedizione gratuita – top farmacia online
Farmacie on line spedizione gratuita: farmacia online migliore – п»їFarmacia online migliore
https://farmait.store/# Farmacia online piГ№ conveniente
farmacia online senza ricetta: Cialis generico farmacia – acquistare farmaci senza ricetta
Right here is the right site for anybody who wishes to understand this topic. You know so much its almost hard to argue with you (not that I actually would want to…HaHa). You definitely put a new spin on a subject which has been written about for a long time. Great stuff, just great.
http://tadalafil.auction/# 36 hour cialis
canadian pharmacy no prescription generic cialis: Buy Cialis online – cialis super active cheap
Very good article. I’m going through many of these issues as well..
free viagra: Viagra online price – generic viagra without a doctor prescription
Good web site you have got here.. It’s difficult to find excellent writing like yours these days. I honestly appreciate people like you! Take care!!
http://tadalafil.auction/# cialis women video
There’s certainly a great deal to find out about this topic. I really like all of the points you made.
cialis generic cheap prices: cialis legal online – generic cialis canada
This excellent website truly has all the info I wanted concerning this subject and didn’t know who to ask.
http://sildenafil.llc/# free viagra
fastest delivery service for cialis: cheapest tadalafil – can i buy cialis without prescription pay pal
http://mexicopharmacy.win/# medicine in mexico pharmacies
india pharmacy mail order: Indian pharmacy online – buy medicines online in india
https://mexicopharmacy.win/# medicine in mexico pharmacies
cost of ed meds
indian pharmacies safe: Best Indian pharmacy – reputable indian pharmacies
http://mexicopharmacy.win/# mexican mail order pharmacies
discount ed pills
http://mexicopharmacy.win/# buying from online mexican pharmacy
Great information. Lucky me I found your blog by accident (stumbleupon). I’ve book-marked it for later.
https://edpillpharmacy.store/# ed drugs online
cheap ed pills online
cheapest ed meds: ED meds online with insurance – cheap ed meds
http://edpillpharmacy.store/# pills for ed online
bookmarked!!, I like your website!
ed medicines online: Best ED pills non prescription – how to get ed meds online
Everyone loves it when individuals come together and share opinions. Great blog, stick with it!
mexico drug stores pharmacies: Medicines Mexico – mexico drug stores pharmacies
Great information. Lucky me I came across your blog by chance (stumbleupon). I’ve book-marked it for later!
https://mexicopharmacy.win/# buying prescription drugs in mexico online
I absolutely love your blog.. Very nice colors & theme. Did you make this amazing site yourself? Please reply back as I’m attempting to create my very own site and would like to know where you got this from or exactly what the theme is named. Thanks.
mexico pharmacies prescription drugs: Best pharmacy in Mexico – buying prescription drugs in mexico online
cheap ed medication: ED meds online with insurance – ed drugs online
http://mexicopharmacy.win/# buying from online mexican pharmacy
I love it when people come together and share ideas. Great website, stick with it.
india pharmacy: Online pharmacy USA – indian pharmacy paypal
Very nice article. I definitely love this site. Stick with it!
http://edpillpharmacy.store/# buy erectile dysfunction pills
cheap ed drugs: Best ED pills non prescription – online erectile dysfunction prescription
I could not resist commenting. Very well written!
where can i buy ed pills: ED meds online with insurance – where can i buy erectile dysfunction pills
Nice post. I learn something new and challenging on sites I stumbleupon everyday. It will always be helpful to read content from other authors and use something from their sites.
https://edpillpharmacy.store/# erectile dysfunction online
buying prescription drugs in mexico online: Medicines Mexico – buying prescription drugs in mexico
I absolutely love your website.. Great colors & theme. Did you develop this website yourself? Please reply back as I’m wanting to create my own website and want to find out where you got this from or what the theme is named. Many thanks!
You should take part in a contest for one of the highest quality websites on the internet. I am going to highly recommend this website!
http://edpillpharmacy.store/# cheapest erectile dysfunction pills
Online medicine home delivery: Online pharmacy – buy medicines online in india
Pretty! This was a really wonderful article. Thank you for providing this information.
ed meds online: Cheap ED pills online – ed medicines
http://mexicopharmacy.win/# purple pharmacy mexico price list
indian pharmacies safe: Cheapest online pharmacy – indianpharmacy com
Next time I read a blog, Hopefully it doesn’t fail me as much as this one. After all, Yes, it was my choice to read through, but I genuinely thought you’d have something helpful to say. All I hear is a bunch of crying about something that you can fix if you weren’t too busy searching for attention.
I blog quite often and I seriously thank you for your information. Your article has truly peaked my interest. I’m going to take a note of your website and keep checking for new details about once per week. I opted in for your RSS feed too.
mexican pharmaceuticals online: mexico drug stores pharmacies – mexican online pharmacies prescription drugs
I’m amazed, I must say. Rarely do I come across a blog that’s both educative and interesting, and without a doubt, you’ve hit the nail on the head. The problem is an issue that not enough folks are speaking intelligently about. I am very happy that I came across this during my hunt for something relating to this.
After looking at a number of the blog posts on your blog, I honestly like your technique of writing a blog. I book-marked it to my bookmark webpage list and will be checking back in the near future. Please check out my web site too and tell me what you think.
ed prescriptions online: ED meds online with insurance – ed online prescription
I absolutely love your blog.. Pleasant colors & theme. Did you build this site yourself? Please reply back as I’m wanting to create my own personal website and would love to learn where you got this from or what the theme is named. Cheers!
online shopping pharmacy india: Online pharmacy – best online pharmacy india
Howdy, I believe your web site may be having browser compatibility problems. When I look at your site in Safari, it looks fine however when opening in Internet Explorer, it has some overlapping issues. I merely wanted to provide you with a quick heads up! Besides that, excellent blog!
I’m very happy to uncover this site. I need to to thank you for ones time just for this wonderful read!! I definitely really liked every little bit of it and I have you saved as a favorite to see new things on your website.
best online pharmacy india: indian pharmacy – indian pharmacy online
Very good information. Lucky me I found your site by chance (stumbleupon). I have bookmarked it for later!
buying from online mexican pharmacy: Medicines Mexico – mexico pharmacies prescription drugs
This blog was… how do you say it? Relevant!! Finally I have found something which helped me. Many thanks!
Hi there! This blog post couldn’t be written much better! Going through this article reminds me of my previous roommate! He constantly kept preaching about this. I’ll send this article to him. Fairly certain he will have a very good read. Many thanks for sharing!
Greetings, I believe your blog could be having web browser compatibility issues. Whenever I look at your blog in Safari, it looks fine however, if opening in IE, it’s got some overlapping issues. I simply wanted to provide you with a quick heads up! Other than that, fantastic site.
cytotec online https://cytotec.pro/# buy misoprostol over the counter
lasix 100 mg
https://lisinopril.guru/# lisinopril 40 mg on line
aromatase inhibitor tamoxifen: Purchase Nolvadex Online – п»їdcis tamoxifen
Good post. I learn something new and challenging on sites I stumbleupon everyday. It will always be helpful to read articles from other authors and use a little something from other sites.
tamoxifen and depression: buy tamoxifen citrate – what happens when you stop taking tamoxifen
buy misoprostol over the counter https://cytotec.pro/# п»їcytotec pills online
lasix pills
https://lisinopril.guru/# lisinopril 5 mg india price
buy cytotec online fast delivery: cheapest cytotec – buy misoprostol over the counter
cytotec abortion pill cheapest cytotec buy cytotec over the counter
buy cytotec pills online cheap https://tamoxifen.bid/# nolvadex 10mg
furosemide 100mg
http://lisinopril.guru/# zestoretic 20 mg
Spot on with this write-up, I honestly believe this website needs a lot more attention. I’ll probably be returning to read more, thanks for the advice!
purchase cytotec https://tamoxifen.bid/# does tamoxifen make you tired
lasix side effects
cytotec abortion pill: buy cytotec online – order cytotec online
https://cytotec.pro/# buy cytotec in usa
buy cytotec pills online cheap https://cytotec.pro/# cytotec buy online usa
buy furosemide online
cheap lipitor buy atorvastatin online buy lipitor from canada
Excellent write-up. I definitely appreciate this site. Keep it up!
buy lisinopril online no prescription: buying lisinopril in mexico – buy lisinopril mexico
https://furosemide.win/# furosemide 100mg
tamoxifen alternatives: buy tamoxifen online – does tamoxifen make you tired
buy cytotec online https://lipitor.guru/# buy lipitor 10 mg
lasix dosage
The next time I read a blog, I hope that it won’t disappoint me just as much as this particular one. After all, Yes, it was my choice to read through, but I actually thought you would have something interesting to talk about. All I hear is a bunch of crying about something you could possibly fix if you were not too busy seeking attention.
furosemide: furosemide online – lasix 40 mg
https://lisinopril.guru/# purchase lisinopril
Oh my goodness! Impressive article dude! Thanks, However I am experiencing troubles with your RSS. I don’t understand the reason why I can’t subscribe to it. Is there anyone else getting similar RSS problems? Anyone who knows the solution can you kindly respond? Thanx!
lasix pills: cheap lasix – lasix furosemide
Hello there! This post could not be written any better! Looking at this post reminds me of my previous roommate! He continually kept preaching about this. I will forward this post to him. Pretty sure he’s going to have a great read. Many thanks for sharing!
tamoxifen for breast cancer prevention: buy tamoxifen citrate – nolvadex price
aromatase inhibitors tamoxifen: common side effects of tamoxifen – tamoxifen mechanism of action
Good article. I’m facing many of these issues as well..
order cytotec online https://cytotec.pro/# buy cytotec online fast delivery
lasix generic
lisinopril generic price: buy lisinopril – zestril 5 mg price
Excellent article. I am going through some of these issues as well..
buy cytotec in usa: cheapest cytotec – buy cytotec over the counter
buy cytotec over the counter https://furosemide.win/# lasix uses
lasix medication
Excellent write-up. I definitely appreciate this site. Keep writing!
pfizer lipitor: buy atorvastatin online – lipitor 40 mg cost
brand name lipitor price: Lipitor 10 mg price – lipitor buy
buy cytotec over the counter https://cytotec.pro/# buy cytotec pills online cheap
lasix furosemide 40 mg
You ought to take part in a contest for one of the finest blogs on the internet. I most certainly will recommend this web site!
This website was… how do you say it? Relevant!! Finally I have found something that helped me. Appreciate it!
buy prescription drugs from india buy prescription drugs from india п»їlegitimate online pharmacies india
http://easyrxindia.com/# india pharmacy
legitimate canadian pharmacy: canadian valley pharmacy – recommended canadian pharmacies
http://mexstarpharma.com/# purple pharmacy mexico price list
mexico pharmacies prescription drugs buying prescription drugs in mexico online best online pharmacies in mexico
This page certainly has all of the information and facts I needed concerning this subject and didn’t know who to ask.
https://easyrxcanada.com/# canadianpharmacyworld
https://easyrxcanada.online/# northwest pharmacy canada
india pharmacy mail order: world pharmacy india – top online pharmacy india
mexican rx online mexican pharmaceuticals online pharmacies in mexico that ship to usa
mexico pharmacy: mexican online pharmacies prescription drugs – п»їbest mexican online pharmacies
buying prescription drugs in mexico online: medicine in mexico pharmacies – buying prescription drugs in mexico online
https://easyrxcanada.online/# is canadian pharmacy legit
https://easyrxindia.com/# Online medicine home delivery
canadian pharmacy service: canadian pharmacy 24 – buying from canadian pharmacies
indian pharmacy paypal: top 10 pharmacies in india – online pharmacy india
https://easyrxcanada.online/# best canadian pharmacy online
http://mexstarpharma.com/# mexican rx online
mexico pharmacies prescription drugs: pharmacies in mexico that ship to usa – buying prescription drugs in mexico online
I need to to thank you for this wonderful read!! I definitely loved every bit of it. I have you book marked to look at new things you post…
slot siteleri 2024: slot siteleri bonus veren – en cok kazandiran slot siteleri
deneme bonusu: deneme bonusu – deneme bonusu veren siteler
https://slotsiteleri.bid/# bonus veren slot siteleri
I’m impressed, I must say. Seldom do I encounter a blog that’s both educative and interesting, and without a doubt, you’ve hit the nail on the head. The problem is an issue that too few men and women are speaking intelligently about. I am very happy I came across this during my search for something relating to this.
slot casino siteleri: canl? slot siteleri – slot oyun siteleri
sweet bonanza kazanma saatleri: sweet bonanza slot demo – pragmatic play sweet bonanza
http://sweetbonanza.network/# sweet bonanza giris
You ought to be a part of a contest for one of the greatest websites on the internet. I am going to highly recommend this web site!
Hi! I just would like to offer you a huge thumbs up for your great info you’ve got right here on this post. I will be returning to your site for more soon.
Very good article! We are linking to this great article on our website. Keep up the good writing.
slot siteleri: en yeni slot siteleri – yasal slot siteleri
Hey there! I simply wish to offer you a huge thumbs up for your great info you’ve got here on this post. I am returning to your website for more soon.
https://slotsiteleri.bid/# slot bahis siteleri
en yeni slot siteleri: slot bahis siteleri – slot oyunlar? siteleri
Her path is to create sacred space for all to experience victory through energy work and the transformative technology of Kundalini Yoga dapoxetina generico She was breastfed for few weeks, supplemented with Nan- milk formula about 20- 30m1 3 hourly for the first five months of life
slot siteleri bonus veren: deneme bonusu veren siteler – en cok kazandiran slot siteleri
http://denemebonusuverensiteler.win/# bonus veren siteler
Hello there! I could have sworn I’ve been to this site before but after browsing through many of the articles I realized it’s new to me. Regardless, I’m definitely pleased I stumbled upon it and I’ll be bookmarking it and checking back frequently.
slot siteleri 2024: oyun siteleri slot – slot bahis siteleri
https://denemebonusuverensiteler.win/# bonus veren siteler
I’m impressed, I have to admit. Seldom do I come across a blog that’s both educative and engaging, and without a doubt, you have hit the nail on the head. The issue is something which not enough folks are speaking intelligently about. I am very happy I stumbled across this during my search for something regarding this.
slot bahis siteleri: slot siteleri 2024 – deneme bonusu veren siteler
I like reading through a post that will make men and women think. Also, thanks for allowing me to comment.
пин ап вход: pin up казино – пинап казино
http://1xbet.contact/# 1xbet официальный сайт мобильная версия
1xbet официальный сайт 1xbet официальный сайт мобильная версия 1xbet скачать
пин ап зеркало: пин ап вход – пин ап казино
https://1win.directory/# 1win официальный сайт
vavada зеркало: вавада казино – vavada online casino
пин ап: pin up casino – пин ап вход
1xbet зеркало рабочее на сегодня: зеркало 1хбет – 1хбет зеркало
https://vavada.auction/# vavada online casino
vavada зеркало: vavada casino – вавада казино
Great post. I will be going through a few of these issues as well..
https://1xbet.contact/# зеркало 1хбет
1xbet скачать: 1xbet официальный сайт мобильная версия – 1xbet зеркало
1xbet скачать: 1xbet скачать – 1хбет
https://1win.directory/# 1вин сайт