zookeeper安装与配置

下载并解压

https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/

编辑配置

修改zoo.cfg

vi /usr/local/zookeeper/conf/zoo.cfg 

修改数据目录和数据日志目录

dataDir=/usr/local/zookeeper/data
dataLogDir=/usr/local/zookeeper/dataLog

配置zookeeper环境变量 修改 /etc/profile文件

export ZOOKEEPER_HOME=/usr/local/zookeeper #zookeeper安装路径  
export PATH=$ZOOKEEPER_HOME/bin:$PATH

启动 Zookeer服务器

[root@kafka1 bin]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@kafka1 bin]# 

查看zookeeper状态

[root@kafka1 bin]# jps
22341 QuorumPeerMain
22389 Jps

继续查看zookeeper服务器的状态

[root@kafka1 bin]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: standalone

连接到zookeeper客户端

[root@kafka1 bin]# zkCli.sh

连接服务器的2181端口

停止服务器

[root@kafka1 bin]# zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

zookeeper 集群

配置第一个节点

搭建zookeeper 集群

修改 /etc/hosts 添加如下配置

192.168.11.12 node01

现在配置第一个kafka节点

cd /usr/local/zookeeper/conf
vi zoo.cfg

修改如下内容

# the port at which the clients will connect
clientPort=2181
server.1=kafka1:2888:3888
server.2=kafka1:4888:5888
server.3=kafka1:6888:7888

执行以下操作

[root@kafka1 bin]# zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... no zookeeper to stop (could not find file /usr/local/zookeeper/data/zookeeper_server.pid)
[root@kafka1 bin]# cd ..
[root@kafka1 zookeeper]# rm -rf /usr/local/zookeeper/data/*
[root@kafka1 zookeeper]# rm -rf /usr/local/zookeeper/dataLog/*

开始生成第二个和第三个节点

[root@kafka1 zookeeper]# cp -rf /usr/local/zookeeper/ /usr/local/zookeeper2
[root@kafka1 zookeeper]# cp -rf /usr/local/zookeeper/ /usr/local/zookeeper3
[root@kafka1 zookeeper]# mv /usr/local/zookeeper /usr/local/zookeeper1

1）修改第二台配置文件 修改第二台的zoo.cfg配置文件

vi /usr/local/zookeeper2/conf/zoo.cfg

# the port at which the clients will connect
clientPort=3181
server.1=kafka1:2888:3888
server.2=kafka1:4888:5888
server.3=kafka1:6888:7888

2）修改第三台配置文件 修改第三台的zoo.cfg配置文件 修改第二台的zoo.cfg配置文件

vi vi /usr/local/zookeeper3/conf/zoo.cfg

# the port at which the clients will connect
clientPort=4181
server.1=kafka1:2888:3888
server.2=kafka1:4888:5888
server.3=kafka1:6888:7888

3. 建数据目录

[root@mqserver /]# mkdir -p /usr/local/zookpeer/data2
[root@mqserver /]# mkdir -p /usr/local/zookpeer/data3
[root@mqserver /]# mkdir -p /usr/local/zookpeer/dataLog2
[root@mqserver /]# mkdir -p /usr/local/zookpeer/dataLog3

4. 写入myid

[root@kafka1 data]# echo 1 > /usr/local/zookeeper1/data/myid
[root@kafka1 data]# echo 2 > /usr/local/zookeeper2/data/myid
[root@kafka1 data]# echo 3 > /usr/local/zookeeper3/data/myid

5. 启动三个集群

[root@kafka1 zookeeper3]# /usr/local/zookeeper1/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper1/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@kafka1 zookeeper3]# /usr/local/zookeeper2/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper2/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@kafka1 zookeeper3]# /usr/local/zookeeper3/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

查看三个集群状态

[root@kafka1 zookeeper3]# /usr/local/zookeeper1/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper1/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
[root@kafka1 zookeeper3]# /usr/local/zookeeper2/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper2/bin/../conf/zoo.cfg
Client port found: 3181. Client address: localhost.
Mode: follower
[root@kafka1 zookeeper3]# /usr/local/zookeeper3/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper3/bin/../conf/zoo.cfg
Client port found: 4181. Client address: localhost.
Mode: leader
[root@kafka1 zookeeper3]# 

可以看到第三个节点是控制节点

配置Kafka

1. 保证2181、3181、4181在监听

[root@kafka1 bin]# lsof -i :2181
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    6597 root   32u  IPv6  58451      0t0  TCP *:eforward (LISTEN)
[root@kafka1 bin]# lsof -i :3181
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    6730 root   32u  IPv6  58314      0t0  TCP *:bmcpatrolagent (LISTEN)
[root@kafka1 bin]# lsof -i :4181
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    6800 root   32u  IPv6  58659      0t0  TCP *:macbak (LISTEN)

2. 把kafka 复制2个节点

[root@kafka1 local]# cp -r kafka/ kafka2
[root@kafka1 local]# cp -r kafka/ kafka3

3. 建日志目录

[root@kafka1 local]# mkdir -p /usr/kafkalog/log1
[root@kafka1 local]# mkdir -p /usr/kafkalog/log2
[root@kafka1 local]# mkdir -p /usr/kafkalog/log3

4. 启动Kafka单机模式

启动Kafka单机模式

Kafka如果直接启动，它的启动信息会打印在控制台。如果关闭窗口，Kafka随之关闭。

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties

在Linux环境下使用daemon守护进程运行Kafka。

$ /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties

查看Kafka是否启动

$ lsof -i:9092

5. 启动kafka集群模式

修改节点1的 kafka 的配置文件 server.properties

broker.id=1
zookeeper.connect=kafka1:2181,kafka1:3181,kafka1:4181
log.dirs=/usr/kafkalog/log1
listeners=PLAINTEXT://kafka1:9093

修改节点2的 kafka 的配置文件 server.properties

broker.id=2
zookeeper.connect=kafka1:2181,kafka1:3181,kafka1:4181
log.dirs=/usr/kafkalog/log2
listeners=PLAINTEXT://kafka1:9094

修改节点3的 kafka 的配置文件 server.properties

broker.id=3
zookeeper.connect=kafka1:2181,kafka1:3181,kafka1:4181
log.dirs=/usr/kafkalog/log3
listeners=PLAINTEXT://kafka1:9095

6. 启动集群

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties 
/usr/local/kafka2/bin/kafka-server-start.sh /usr/local/kafka2/config/server.properties 
/usr/local/kafka3/bin/kafka-server-start.sh /usr/local/kafka3/config/server.properties 

后台运行 kafka集群

/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties 
/usr/local/kafka2/bin/kafka-server-start.sh -daemon /usr/local/kafka2/config/server.properties 
/usr/local/kafka3/bin/kafka-server-start.sh -daemon /usr/local/kafka3/config/server.properties 

查看Kafka集群是否启动

$ lsof -i:9093
$ lsof -i:9094
$ lsof -i:9095

7. 停止集群

/usr/local/kafka/bin/kafka-server-stop.sh  
/usr/local/kafka2/bin/kafka-server-stop.sh  
/usr/local/kafka3/bin/kafka-server-stop.sh  

操作 kafka

1. 创建主题

/usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server kafka1:2181,kafka1:3181,kafka1:4181 --replication-factor 1 --partitions 1 --topic test

或者

/usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server  192.168.11.12:9093  --replication-factor 1 --partitions 1 --topic test

或者

/usr/local/kafka/bin/kafka-topics.sh --create --bootstrap-server 192.168.11.12:2181,192.168.11.12:3181,192.168.11.12:4181 --replication-factor 1 --partitions 1 --topic test2

2. 查看主题

查看topic

/usr/local/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka1:9093

/usr/local/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka1:9093

3. 发送消息 发送消息到test主题

/usr/local/kafka/bin/kafka-console-producer.sh --broker-list kafka1:9093 --topic test

4. 查看消息 查看test消息主题 ，--from-beginning从头查看

/usr/local/kafka/bin/kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka1:9093

5. 删除主题

/usr/local/kafka/bin/kafka-topics.sh --delete --bootstrap-server kafka1:9093 --topic test