源码编译安装ansible 一、安装一些基础软件包yum -y install xz wget gcc make gdbm-devel openssl-devel sqlite-devel zlib-devel bzip2-devel python-jinja2 PyYAML python-paramiko python-babel python-crypto --downloadonly --downloaddir=/usr/local/src/ yum -y localinstall /usr/local/src/*.rpm二、安装ansible包wget https://releases.ansible.com/ansible/ansible-2.9.27.tar.gz 三、解压ansible包并进去解压完成的目录tar xf ansible-2.9.27.tar.gz && cd ansible-2.9.27四、执行python脚本python setup.py build python setup.py install五、拷贝ansible目录mkdir /etc/ansible cp -r examples/* /etc/ansible六、查看ansible版本ansible --version

部署redis 一、离线安装方式1.下载离线软件包下载地址：[https://github.com/microsoftarchive/redis] redis-6.2.5-1.el7.remi.x86_64.rpm2.安装redisrpm -ivh redis-6.2.5-1.el7.remi.x86_64.rpm3.修改配置文件sed -i '/ *#/d; /^$/d' #把所有注释项全部为空 vim /etc/redis/redis.conf ... daemonize yes port 7000 bind 10.0.0.105 masterauth "your_password" requirepass "your_password"4.启动redis/usr/bin/redis-server /etc/redis/redis.conf5.停止redis/usr/bin/redis-cli -h your_url -p 7000 -a your_password shutdown二、源码编译安装redis1.先去Redis官网下载tar包 这儿是下载6.2.13版本[https://download.redis.io/releases/redis-7.2.4.tar.gz] [https://redis.io/download] #官网地址2.下载完成后拉倒自己服务器里面解压安装依赖包： yum -y install gcc gcc-c++ automake autoconf libtool make libc tcl [root@minlo01 ~]# tar xf redis-6.2.13.tar.gz -C /usr/3.解压完成后进入redis-6.2.13目录 然后编译安装[root@minlo01 ~]# cd /usr/redis-6.2.13 [root@minlo01 redis-6.2.13]# make && make install 4.配置文件中设置启动服务后台运行,找到daemon 改为yes[root@minlo01 ~]# vim /usr/redis-6.2.13/redis.conf ... daemonize yes ...5.配置文件中设置密码 进入配置文件redis.conf中找到# requirepass foobared[root@minlo01 ~]# vim /usr/redis-6.2.13/redis.conf ... masterauth 123 requirepass 123 ... 6.然后启动Redis服务 server是要注意需要指定配置文件，不然不读[root@minlo01 redis-6.2.13]# redis-server redis.conf7.然后登录时需要 -a 指定你修改的密码[root@minlo01 redis-6.2.13]# redis-cli -a 1238.后面一下操作测试下redis ，一些redis基本命令，出现Warning不用管这是提示密码在命令行显示不安全[root@minlo01 redis-6.2.13]# redis-cli -a 123 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> selcet 1 # select 数据库编号 ok 127.0.0.1:6379> set test 123 #设置键值对 ok 127.0.0.1:6379> get test #取出键值对 "123" 127.0.0.1:6379> keys test #查看一个键是否存在 1) "test" 127.0.0.1:6379> del test1 #删除键值对 (integer) 127.0.0.1:6379> flushall #清空所有数据库中的数据三、部署redis集群1.准备工作目录和软件包1.创建工作目录 mkdir /data/{redis-cluster-7000,logs} -p 2.下载软件包 wget https://download.redis.io/releases/redis-7.2.4.tar.gz 3.解压软件包 tar xf redis-7.2.4.tar.gz -C /data/ 4.安装redis cd /data/redis-7.2.4 make && make install2.创建目录cd /data/redis-7.2.4 mkdir {bin,conf,logs} mv src/redis-benchmark src/redis-check-aof src/redis-cli src/redis-server /data/redis-7.2.4/bin/ mv redis.conf conf/3.修改配置文件sed -i '/ *#/d; /^$/d' redis.conf #把所有注释项全部为空 vim redis.conf ... daemonize yes port 7000 bind 10.0.0.141 dir "/data/redis-7.2.4/" cluster-enabled yes cluster-config-file "/data/redis-7.2.4/nodes-7000.conf" cluster-node-timeout 15000 appendonly no # save "" dbfilename "dump-7000.rdb" rdbcompression yes rdbchecksum yes tcp-backlog 511 tcp-keepalive 60 repl-disable-tcp-nodelay yes timeout 0 loglevel notice logfile "/data/redis-7.2.4/logs/redis.log" requirepass "myredispassword" masterauth "myredispassword" maxclients 100000 # Generated by CONFIG REWRITE pidfile "/var/run/redis.pid" maxmemory-policy allkeys-lru maxmemory 1gb # user default on #074e7123c76bae5dedbc082ebe80bacb4de38ca0bcc540190428285675c1c269 ~* &* +@all4.启动redis1.启动redis redis-server /data/redis-7.2.4/conf/redis.conf 2.验证redis ps -ef|grep redis root 7175 1 0 14:38 ? 00:00:00 redis-server 10.0.0.141:7000 [cluster] root 7183 1465 0 14:38 pts/0 00:00:00 grep --color=auto redis 验证端口： ss -ntl|grep 7000 LISTEN 0 128 10.0.0.141:7000 *:* LISTEN 0 128 10.0.0.141:17000 3.停止redis (1) redis-cli -h 10.0.0.141 -p 7000 -a myredispassword shutdown exit (2) redis-cli shutdown (3) ps -ef|grep -v grep | grep redis| awk '{print $2}'| xargs kill -95.验证登录1.登录客户端 redis-cli -h 10.0.0.115 -p 7000 -a myredispassword redis-cli -h 10.0.0.116 -p 7000 -a myredispassword ... 2. redis-cli -h 10.0.0.115 -p 7000 -c 10.0.0.115:7000> auth myredispassword OK6.创建集群1.直接在命令行执行： redis-cli --cluster create \ 10.0.0.141:7000 \ 10.0.0.142:7000 \ 10.0.0.143:7000 \ 10.0.0.114:7000 \ 10.0.0.115:7000 \ 10.0.0.116:7000 \ --cluster-replicas 1 \ -a 'myredispassword' #日志输出即创建完成 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 2. 验证集群： [root@mysql-114 conf]# redis-cli -h 10.0.0.115 -p 7000 -c 10.0.0.115:7000> auth myredispassword OK 10.0.0.115:7000> cluster nodes 06791aed02dc766071118a51c4f7e32bdac52b34 10.0.0.142:7000@17000 master - 0 1749556205000 2 connected 5461-10922 47a1414fa189d91298bd63df05629099e8e8903d 10.0.0.141:7000@17000 master - 0 1749556205582 1 connected 0-5460 f6658c99625913405dc7c7cdf4988ba269365f87 10.0.0.115:7000@17000 myself,slave 47a1414fa189d91298bd63df05629099e8e8903d 0 1749556202000 1 connected d86b349fcc2699b790da4bd4ff316f7a043030b8 10.0.0.143:7000@17000 master - 0 1749556203000 3 connected 10923-16383 edcc38c882f8a86b704d14c7219d2d03da660038 10.0.0.114:7000@17000 slave d86b349fcc2699b790da4bd4ff316f7a043030b8 0 1749556204000 3 connected d32b92d35c9acf66f01f47d8fa0c5a2d52c21f6b 10.0.0.116:7000@17000 slave 06791aed02dc766071118a51c4f7e32bdac52b34 0 1749556202498 2 connected 10.0.0.115:7000> quit # 3个master和3个slave 代表集群验证正常

基于Docker部署Prometheus 1.部署Docker（使用docker仓库进行安装）1.设置仓库 （安装所需的软件包） yum install -y yum-utils device-mapper-persistent-data lvm2 2.添加仓库 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo 3.列出并排序您存储库中可用的版本。此示例按版本号（从高到低）对结果进行排序 yum list docker-ce --showduplicates | sort -r 4.安装Docker yum install docker-ce-26.1.4-1.el7 docker-ce-cli-26.1.4-1.el7 containerd.io 5.设置开机自启动 systemctl enable docker 6.启动Docker systemctl start docker 7.配置Docker镜像加速器 国内镜像加速器地址 网易：https://hub-mirror.c.163.com/ 阿里云：https://<你的ID>.mirror.aliyuncs.com 七牛云加速器：https://reg-mirror.qiniu.com [root@prometheus docker-compose]# cat /etc/docker/daemon.json { "registry-mirrors": [ "https://docker.1ms.run", "https://docker.xuanyuan.me" ], "data-root": "/data/docker" } systemctl daemon-reload systemctl restart docker 8.测试Docker [root@prometheus docker-compose]# docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world [root@prometheus docker-compose]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE hello-world latest 74cc54e27dc4 8 weeks ago 10.1kB2.安装docker-compose1.sudo curl -L "https://github.com/docker/compose/releases/download/v2.20.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose 2.sudo chmod +x /usr/local/bin/docker-compose 3.docker-compose --version3.准备镜像1.准备工作目录 [root@prometheus_servers_131 ~]# mkdir /data/docker-compose/ [root@prometheus_servers_131 ~]# cd /data/docker-compose/ [root@prometheus_servers_131 docker-compose]# mkdir {alertmanager,prometheus,data/prometheus/wal} -p 2.修改配置文件 [root@prometheus_servers_131 docker-compose]# cat alertmanager/alertmanager.yml global: resolve_timeout: 30s smtp_smarthost: 'smtp.163.com:25' smtp_from: 'your_email' smtp_auth_username: 'your_email' smtp_auth_password: 'VZffFwPPQAWzAPhw' # 邮箱授权码，需要在设置，开启SMTP服务，然后复制授权码到这里 smtp_hello: '163.com' smtp_require_tls: false templates: - '/etc/alertmanager/email.tmpl' route: group_by: ['env','instance','type','group','job','alertname'] group_wait: 30s group_interval: 5m repeat_interval: 40s receiver: 'email' receivers: - name: 'email' email_configs: - to: 'your_email' html: '{{ template "email.to.html" . }}' send_resolved: true inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'dev', 'instance'] [root@prometheus_servers_131 docker-compose]# cat alertmanager/email.tmpl {{ define "email.to.html" }} {{- if gt (len .Alerts.Firing) 0 -}}{{ range .Alerts }} <h2>@告警通知</h2> 告警程序: prometheus_alert <br> 告警级别: {{ .Labels.severity }} <br> 告警类型: {{ .Labels.alertname }} <br> 故障主机: {{ .Labels.instance }} <br> 告警主题: {{ .Annotations.summary }} <br> 告警详情: {{ .Annotations.description }} <br> 触发时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }} <br> {{ end }}{{ end -}} {{- if gt (len .Alerts.Resolved) 0 -}}{{ range .Alerts }} <h2>@告警恢复</h2> 告警程序: prometheus_alert <br> 告警级别: {{ .Labels.severity }} <br> 故障主机: {{ .Labels.instance }}<br> 故障主题: {{ .Annotations.summary }}<br> 告警详情: {{ .Annotations.description }}<br> 告警时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}<br> 恢复时间: {{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}<br> {{ end }}{{ end -}} {{- end }} [root@prometheus_servers_131 docker-compose]# cat prometheus/prometheus.yml global: scrape_interval: 1s evaluation_interval: 1s rule_files: - /etc/prometheus/rules.yml alerting: alertmanagers: - static_configs: - targets: ['10.0.0.131:9093'] scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['10.0.0.131:9090'] - job_name: 'node-export' static_configs: - targets: ['10.0.0.131:9100'] [root@prometheus_servers_131 docker-compose]# cat prometheus/rules.yml groups: - name: server-alarm rules: - alert: "内存告警" expr: (1 - (node_memory_MemAvailable_bytes / (node_memory_MemTotal_bytes))) * 100 > 68 for: 30s labels: severity: warning annotations: summary: "{{$labels.instance}}: 检测到 高内存 使用率！" description: "{{$labels.instance}}: 内存使用率在 80% 以上 (当前使用值为:{{ printf \"%.0f\" $value }}% )" - alert: "CPU告警" expr: (1 - avg(irate(node_cpu_seconds_total{mode="idle"}[2m])) by(instance)) * 100 > 80 for: 30s labels: severity: warning annotations: summary: "{{$labels.instance}}: 检测到 高CPU 使用率！" description: "{{$labels.instance}}: CPU使用率在 80% 以上 (当前使用值为:{{ printf \"%.0f\" $value }}% )" - alert: "磁盘告警" expr: 100 - (node_filesystem_free_bytes{fstype=~"tmpfs|ext4|xfs",mountpoint=~"/"} / node_filesystem_size_bytes{fstype=~"tmpfs|ext4|xfs",mountpoint=~"/"} * 100) > 85 for: 30s labels: severity: warning annotations: summary: "{{$labels.instance}}: 检测到 高磁盘 使用率！" description: "{{$labels.instance}}: / 磁盘使用率在 85% 以上 (当前使用值为:{{ printf \"%.0f\" $value }}% )" - alert: FlinkJobDown #expr: flink_jobmanager_job_downtime > 0 expr: flink_jobmanager_numRunningJobs <= 0 for: 1m labels: #severity: critical severity: 严重告警 annotations: summary: "Flink 作业已停止" description: "作业{{ $labels.job }}已停止,当前运行job个数为:{{ $value }}。" - alert: FlinkJobDown #expr: flink_jobmanager_job_downtime > 0 expr: flink_jobmanager_numRunningJobs <= 0 for: 1m labels: #severity: critical severity: 一般告警 annotations: summary: "Flink 作业已停止" description: "作业{{ $labels.job }}已停止,当前运行job个数为:{{ $value }}。" [root@prometheus_servers_131 ~]# cat /data/docker-compose/docker-compose.yaml version: '2' networks: monitor: driver: bridge services: prometheus: image: prom/prometheus container_name: prometheus hostname: prometheus restart: always volumes: - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - ./prometheus/rules.yml:/etc/prometheus/rules.yml - /etc/localtime:/etc/localtime - /data/docker-compose/data/prometheus/wal:/prometheus/wal ports: - "9090:9090" networks: - monitor alertmanager: image: prom/alertmanager container_name: alertmanager hostname: alertmanager restart: always volumes: - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml - ./alertmanager/email.tmpl:/etc/alertmanager/email.tmpl - /etc/localtime:/etc/localtime ports: - "9093:9093" networks: - monitor grafana: image: grafana/grafana container_name: grafana hostname: grafana restart: always volumes: - /etc/localtime:/etc/localtime - /data/docker-compose/grafana/data:/var/lib/grafana ports: - "3000:3000" networks: - monitor cadvisor: image: google/cadvisor:latest container_name: cadvisor hostname: cadvisor restart: always volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /etc/localtime:/etc/localtime ports: - "8899:8080" networks: - monitor 3.拉取镜像 docker pull prom/alertmanager docker pull prom/prometheus docker pull grafana/grafana docker pull google/cadvisor 4.创建容器 docker-compose up -d4.Prometheus监控nginx1.安装nginx rpm -ivh pcre2-10.23-2.el7.x86_64.rpm rpm -ivh nginx-1.26.1-1.el7.ngx.x86_64.rpm 2.设置开机自启 systemctl enable nginx systemctl start nginx 3.检查nginx是否启动成功 ps -ef|grep nginx ss -ntl |grep 80 4.启用Nginx的stub_status模块 nginx -V 2>&1 | grep -o with-http_stub_status_module 5.vim /etc/nginx/conf.d/status.conf [root@flink nginx]# cat /etc/nginx/conf.d/status.conf server { listen 80; server_name 10.0.0.100; location /nginx_status { stub_status on; access_log off; allow 10.0.0.1; # 允许客户端访问，按需修改 allow 10.0.0.100; allow 10.0.0.131; allow 127.0.0.1; # 仅允许本地访问，按需修改 deny all; } } 6.重启nginx systemctl restart nginx 或者 nginx -s reload 7.验证状态页 http://10.0.0.100/nginx_status 8.curl验证 [root@flink nginx]# curl http://10.0.0.100/nginx_status Active connections: 4 server accepts handled requests 11 11 7455 Reading: 0 Writing: 1 Waiting: 3 9.部署Nginx Prometheus Exporter （使用第三方Exporter将stub_status数据转换为Prometheus格式。） 下载Exporter：wget https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v0.11.0/nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz 运行Exporter：./nginx-prometheus-exporter -nginx.scrape-uri http://localhost:80/nginx_status 配置nginx-exporer.service [root@flink nginx]# cat /etc/systemd/system/nginx-exporter.service [Unit] Description=Linux Node Exporter Documentation=https://prometheus.io/docs/introduction/overview/ After=network.target [Service] ExecStart=/data/nginx/nginx-prometheus-exporter -nginx.scrape-uri http://10.0.0.100:80/nginx_status [Install] WantedBy=multi-user.target 重新加载：systemctl daemon-reload 开机自启：systemctl enable nginx-exporter 启动nginx-exporter：systemctl start nginx-exporter 检查nginx-exporter ps -ef|grep nginx root 8750 1 0 10:56 ? 00:01:12 /data/nginx/nginx-prometheus-exporter -nginx.scrape-uri http://10.0.0.100:80/nginx_status ss -ntl |grep 9113 #nginx-exporter 默认9113端口 10.修改Prometheus配置文件 vim /data/docker-compose/prometheus/prometheus.yml - job_name: 'nginx' static_configs: - targets: ['10.0.0.100:9113'] 11.重启Prometheus docker restart 容器id 12.登录Prometheus页面查看是否采集成功 curl http://10.0.0.100:9113/metrics|grep nginx 13，监控nginx服务状态指标nginx up nginx up 0 #代表服务可能停止运行，异常 nginx up 1 #服务正常 14.修改告警规则配置文件 vim /data/docker-compose/prometheus/rules.yml - alert: "nginx 服务异常" expr: nginx_up == 0 for: 30s labels: severity: "严重告警" annotations: summary: "Nginx 服务不可用" description: "Prometheus 无法连接到 Nginx 服务实例 {{ $labels.instance }}，Nginx 服务可能已停止运行。" 5.监控node-exporter1.部署node-exporter （https://github.com/prometheus/node_exporter/releases/download） 准备软件包node_exporter-1.8.2.linux-amd64.tar.gz 2.解压软件包 tar xf node_exporter-1.8.2.linux-amd64.tar.gz 3.配置systemcd服务node-exporter.service [root@prometheus_servers_131 ~]# cat /etc/systemd/system/node-exporter.service [Unit] Description=Linux Node Exporter Documentation=https://prometheus.io/docs/introduction/overview/ After=network.target [Service] ExecStart=/data/docker-compose/node_exporter-1.8.2.linux-amd64/node_exporter # --web.listen-address=:9100 --collector.textfile.directory=/data/docker-compose/node_exporter-1.8.2.linux-amd64/key/ [Install] WantedBy=multi-user.target 重新加载：systemctl daemon-reload 开机自启：systemctl enable node-exporter 启动：systemctl start node-exporter 4.Prometheus配置监控node-exporter cat /data/docker-compose/prometheus/prometheus.yml - job_name: 'node-export' static_configs: - targets: ['10.0.0.131:9100'] # 集群可以['10.0.0.141:9100','10.0.0.142:9100','10.0.0.143:9100'] 5.重启Prometheus docker restart 容器id 6.导入摸id=18606.监控zookeeper集群1.下载zookeeper_exporter （集群每一台执行） mv zookeeper_exporter /usr/local/sbin/ && chmod +x /usr/local/sbin/zookeeper_exporter 2.启动 （集群每一台执行） nohup /usr/local/sbin/zookeeper_exporter -zookeeper=10.0.0.108:2181 -zookeeper=10.0.0.109:2181 -zookeeper=10.0.0.110:2181 >> /data/exporter/logs/zookeeper.log 2>&1 & 3.访问 http://10.0.0.108:9141/metrics http://10.0.0.109:9141/metrics http://10.0.0.110:9141/metrics 4.修改zoo.cfg配置文件 4lw.commands.whitelist=srvr,ruok,mntr,stat 5.配置systemcd服务zookeeper-exporter.service [root@kafka108 ~]# vim /etc/systemd/system/zookeeper-exporter.service [Unit] Description=Linux Node Exporter Documentation=https://prometheus.io/docs/introduction/overview/ After=network.target [Service] ExecStart=/usr/local/sbin/zookeeper_exporter -zookeeper=10.0.0.108:2181 -zookeeper=10.0.0.109:2181 -zookeeper=10.0.0.110:2181 [Install] WantedBy=multi-user.target 重新加载：systemctl daemon-reload 开机自启：systemctl enable zookeeper-exporter 启动：systemctl start zookeeper-exporter 6.Prometheus配置监控zookeeper-exporter vim /data/docker-compose/prometheus/prometheus.yml ... - job_name: zookeeper-exporter static_configs: - targets: ['10.0.0.108:9141','10.0.0.109:9141','10.0.0.110:9141'] 7.重启Prometheus docker restart 容器id 8.granfan导入模版 id=150267.监控kafka集群1.下载kafka_exporter （集群每一台执行） tar xf kafka_exporter-1.8.0.linux-amd64.tar.gz -C /data/ 2.启动kafka-exporter nohup /data/kafka_exporter-1.8.0.linux-amd64/kafka_exporter --kafka.server=10.0.0.108:9092 --kafka.server=10.0.0.109:9092 --kafka.server=10.0.0.110:9092 --sasl.enabled --sasl.mechanism=PLAIN --sasl.username="admin" --sasl.password="admin@2024" >> /logs/kafka-exporter/kafka-exporter.log 2>&1 & 3.验证端口（默认端口9308） ss -ntl|grep 9308 http://10.0.0.108:9308/metrics 4.配置Prometheus vim /data/docker-compose/prometheus/prometheus.yml - job_name: kafka-exporter static_configs: - targets: ['10.0.0.108:9308','10.0.0.109:9308','10.0.0.110:9308'] 5.重启Prometheus docker restart 容器id 8.监控es集群1.下载elasticsearch_exporter https://github.com/prometheus-community/elasticsearch_exporter/releases 2.准备工作目录 mkdir /data/export/logs -p 3.解压软件包 tar xf elasticsearch_exporter-1.6.0.linux-amd64.tar.gz -C /data/exporter/ 4。需要配置环境变量 cat /etc/profile ... export ES_USERNAME=your_name #es用户 export ES_PASSWORD=your_password #es密码 5.启动es-exporter nohup ./elasticsearch_exporter --es.uri=http://10.0.0.105:9200 --es.ca=/zhuyang/app/elasticsearch-7.3.1/config/elasticsearch-certificates.p12 >> /data/exporter/logs/es.log 2>&1 & --es.uri=http://10.0.0.105:9200 #集群每台执行一次并且ip地址换成本机ip 6.配置Prometheus vim /data/docker-compose/prometheus/prometheus.yml - job_name: 'es-exporter' static_configs: - targets: ['10.0.0.105:9114','10.0.0.106:9114','10.0.0.107:9114'] 7.重启Prometheus docker restart 容器id 8.导入granfan 141919.监控flink集群1.修改配置文件 metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory 2.重启flink集群 stop-cluster.sh start-cluster.sh 3.检查端口默认端口9249 ss -ntl|grep 9249 4.Prometheus配置flink vim /data/docker-compose/prometheus/prometheus.yml - job_name: 'flink_job' static_configs: - targets: ['10.0.0.111:9249','10.0.0.112:9249','10.0.0.113:9249'] 5.重启Prometheus docker restart 容器id 6.导入granfan 11049 1491110.监控redis1.部署redis 下载redis离线包 redis-6.2.5-1.el7.remi.x86_64.rpm 2.安装redis rpm -ivh redis-6.2.5-1.el7.remi.x86_64.rpm 3.修改配置文件 sed -i '/ *#/d; /^$/d' #把所有注释项全部为空 vim /etc/redis/redis.conf ... daemonize yes bind 10.0.0.105 masterauth "Admin@2024" requirepass "Admin@2024" 4.启动redis /usr/bin/redis-server /etc/redis/redis.conf 5.停止redis /usr/bin/redis-cli -h 10.0.0.105 -p 7000 -a Admin@2024 shutdown 6.安装redis_exporter https://github.com/oliver006/redis_exporter/releases/download 7.解压软件包 tar xf redis_exporter-v1.69.0.linux-386.tar.gz -C /data/exporter 8.启动redis_exporter nohup /data/exporter/redis_exporter-v1.69.0.linux-386/redis_exporter -redis.addr 10.0.0.105:7000 -redis.password Admin@2024 >> /data/exporter/logs/redis.log 2>&1 & 9.验证端口默认9121 ss -ntl|grep 9121 10.Prometheus配置flink vim /data/docker-compose/prometheus/prometheus.yml - job_name: 'redis_exporter' static_configs: - targets: ['10.0.0.105:9121'] 11.重启Prometheus docker restart 容器id 12.导入granfan 763模版id或者 11835 id11.监控MySQL1.下载mysql_exporter https://github.com/prometheus/mysqld_exporter/releases/download/ mysqld_exporter-0.17.1.linux-amd64.tar.gz 2.创建mysql_exporter用户 -- 登录到 MySQL 服务器 mysql -u root -p -- 创建新用户 CREATE USER 'mysql_exporter'@'localhost' IDENTIFIED BY 'your_password'; -- 授予用户必要的权限 GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysql_exporter'@'localhost'; -- 刷新权限 FLUSH PRIVILEGES; 3. 配置并启动 MySQL Exporter 创建一个配置文件 my.cnf 来存储 MySQL 的连接信息： vim /data/exporter/mysqld_exporter-0.17.1.linux-amd64/my.cnf [root@elk107 mysqld_exporter-0.17.1.linux-amd64]# cat my.cnf [client] user = mysql_exporter password = your_password 启动 MySQL Exporter，并指定配置文件： 后台启动： nohup /data/exporter/mysqld_exporter-0.17.1.linux-amd64/mysqld_exporter --config.my-cnf="/data/exporter/mysqld_exporter-0.17.1.linux-amd64/my.cnf" >> /data/logs/mysql_exporter.log 2>&1 & 4.验证端口默认9104 ss -ntl|grep 9104 5.Prometheus配置mysql vim /data/docker-compose/prometheus/prometheus.yml - job_name: 'mysql_exporter' static_configs: - targets: ['10.0.0.107:9104'] 6.重启Prometheus docker restart 容器id 7.导入granfan 14031或者1894912.监控java -jar运行的微服务进程1.使用 node_exporter 的 textfile 收集器 创建目录： mkdir /data/exporter/node_exporter-1.8.2.linux-amd64/textfile_collector 2.自定义脚本 [root@dev-dmp-md-1 textfile_collector]# cat java_process.sh #!/bin/bash # 定义指标输出路径（需与 node_exporter 配置一致） OUTPUT_DIR="/data/exporter/node_exporter-1.8.2.linux-amd64/textfile_collector" METRIC_FILE="$OUTPUT_DIR/java_process_metrics.prom" # 清理旧指标文件 rm -f $METRIC_FILE # 检查 java -jar 进程是否存在 PROCESS_COUNT=$(pgrep -f 'java -jar' | wc -l) # 采集进程资源使用（示例：内存和CPU） if [ $PROCESS_COUNT -gt 0 ]; then # 获取第一个匹配进程的 PID PID=$(pgrep -f 'java -jar' | head -n 1) # 使用 ps 获取进程的 CPU 和内存占用 CPU_USAGE=$(ps -p $PID -o %cpu --no-headers) MEM_USAGE=$(ps -p $PID -o %mem --no-headers) RSS_MEM=$(ps -p $PID -o rss --no-headers) # 实际内存（KB） else CPU_USAGE=0 MEM_USAGE=0 RSS_MEM=0 fi # 生成 Prometheus 格式的指标 cat << EOF > $METRIC_FILE # HELP java_process_running Whether the Java process is running (1) or not (0) # TYPE java_process_running gauge java_process_running $PROCESS_COUNT # HELP java_process_cpu_usage CPU usage percentage of the Java process # TYPE java_process_cpu_usage gauge java_process_cpu_usage $CPU_USAGE # HELP java_process_mem_usage Memory usage percentage of the Java process # TYPE java_process_mem_usage gauge java_process_mem_usage $MEM_USAGE # HELP java_process_rss_mem_kb Resident Set Size memory of the Java process in KB # TYPE java_process_rss_mem_kb gauge java_process_rss_mem_kb $RSS_MEM EOF 创建文件 touch /data/exporter/node_exporter-1.8.2.linux-amd64/textfile_collector/java_process_metrics.prom 4.修改启动命令 [root@dev-dmp-md-1 textfile_collector]# cat /etc/systemd/system/node-exporter.service [Unit] Description=Linux Node Exporter Documentation=https://prometheus.io/docs/introduction/overview/ After=network.target [Service] ExecStart=/data/exporter/node_exporter-1.8.2.linux-amd64/node_exporter --collector.textfile.directory=/data/exporter/node_exporter-1.8.2.linux-amd64/textfile_collector --collector.textfile #指定node_exporter 的 textfile 收集器的目录路劲 [Install] WantedBy=multi-user.target 5.重启node-exporter systemctl restart node-exporter

MySQL部署 1.准备工作目录和环境1.创建目录 [root@mysql-114 ~]# mkdir /zy/{app,data,logs,bak} -p 2.查看防火墙是否关闭 [root@mysql-114 ~]# systemctl status firewalld 3.查看selinux安全策略是否关闭 [root@mysql-114 ~]# getenforce 4.清除系统自带的mariadb数据库服务相关的程序包 [root@mysql-114 ~]# rpm -qa|grep mariadb [root@mysql-114 ~]# yum remove -y mariadb-libs 5.安装数据库服务程序所需的依赖软件包 [root@mysql-114 ~]# yum install -y libaio-devel 6.创建数据库服务管理用户信息 [root@mysql-114 ~]# useradd mysql 7.创建数据库服务相关目录并进行目录信息进行授权 [root@mysql-114 ~]# chown -R mysql. /zy/2.上传mysql安装包，并解压[root@mysql-114 ~]# cd /zy/app/ [root@mysql-114 app]# ll total 650144 -rw-r--r-- 1 root root 665746960 May 29 14:26 mysql-commercial-8.0.35-linux-glibc2.12-x86_64.tar.xz [root@mysql-114 app]# tar xf mysql-commercial-8.0.35-linux-glibc2.12-x86_64.tar.xz -C /zy/app/3.创建软链接[root@mysql-114 app]# ln -s /zy/app/mysql-commercial-8.0.35-linux-glibc2.12-x86_64 /usr/local/mysql4.配置环境变量[root@mysql-114 ~]# vim /etc/profile #在最下面增加以下内容： export MYSQL_HOME=/usr/local/mysql export PATH=$PATH:$MYSQL_HOME/bin [root@mysql-114 ~]# source /etc/profile #重新加载系统环境变量5.查看是否配置成功[root@mysql-114 ~]# mysql --version mysql Ver 8.0.35-commercial for Linux on x86_64 (MySQL Enterprise Server - Commercial) [root@mysql-114 ~]# mysql -V mysql Ver 8.0.35-commercial for Linux on x86_64 (MySQL Enterprise Server - Commercial)6.修改my.cnf配置文件[root@mysql-114 ~]# cat /etc/my.cnf [mysql] socket=/tmp/mysql.sock [mysqld] user=mysql basedir=/usr/local/mysql datadir=/zy/data socket=/tmp/mysql.sock7.初始化mysql[root@mysql-114 ~]# mysqld --defaults-file=/etc/my.cnf --user=mysql --basedir=/usr/local/mysql --datadir=/zy/data --initialize 2024-08-13T13:58:59.334951Z 0 [System] [MY-013169] [Server] /zy/app/mysql-commercial-8.0.35-linux-glibc2.12-x86_64/bin/mysqld (mysqld 8.0.35-commercial) initializing of server in progress as process 1516 2024-08-13T13:58:59.340770Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-08-13T13:59:01.565838Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-08-13T13:59:08.618023Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: kV,DlQ/kw9ps #这里就是初始化默认管理员密码 #初始化操作参数： --initialize-insecure #表示不安全方式初始化（无管理员密码） --initialize #表示安全方式初始化（有随机默认管理员密码） # 数据库服务初始化拨错信息 [报错信息]： mysqld: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory [解决方法]： yum install -y libaio-devel [报错信息]： initialize specified but the data directory has files in it. Aborting. The designated data directory /zy/data/ is unusable. You can remove all files that the server added to it. [解决方法]： rm -rf /zy/data/*8.启动mysql服务1.前台启动 [root@mysql-114 ~]# mysqld_safe --defaults-file=/etc/my.cnf --user=mysql 2.后台启动 [root@mysql-114 ~]# nohup /usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql >> /zy/logs/local_start.log 2>&1 & [root@mysql-114 ~]# nohup /usr/local/mysql/bin/mysqladmin shutdown -uroot -p'数据库密码' --socket /tmp/mysql.sock >> /zy/logs/local_stop.log 2>&1 & #停止数据库 3.设置数据库服务程序启动运行文件 [root@mysql-114 ~]# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld 4.启动数据库服务程序利用脚本文件 [root@mysql-114 ~]# /etc/init.d/mysqld start [root@mysql-114 ~]# /etc/init.d/mysqld stop [root@mysql-114 ~]# service mysqld start [root@mysql-114 ~]# service mysqld stop 5.启动数据库服务程序利用脚本文件 (systemd方式) [root@mysql-114 ~]# systemctl enable mysqld mysqld.service is not a native service, redirecting to /sbin/chkconfig. Executing /sbin/chkconfig mysqld on #将service管理服务方式转换为systemd管理服务方式: [root@mysql-114 ~]# systemctl start mysqld [root@mysql-114 ~]# systemctl is-active mysqld9.验证数据库是否启动成功 [root@mysql-114 ~]# netstat -lntup|grep 3306 [root@mysql-114 ~]# ss -ntl|grep 3306 LISTEN 0 128 [::]:3306 [::]:* LISTEN 0 70 [::]:33060 [::]:* [root@mysql-114 ~]# ps -ef|grep mysqld #特殊说明：在启动mysql数据库服务时，可能看到33060端口信息，此端口信息主要实现mysqlx协议的通讯过程； #利用mysqlx协议可以实现利用mysql-shell功能组件，对数据进行key-value操作，即识别json文件信息，进行远程管理；10.修改默认密码并开启远程连接[root@mysql-114 ~]# mysql -u root -p Enter password: #第一次需输入初始化默认密码 ... mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Zy200211@'; #修改密码 Query OK, 0 rows affected (0.01 sec) mysql> use mysql; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> update user set host = '%' where user = 'root'; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> select host,user,plugin,authentication_string from mysql.user; #查询数据库服务用户信息11.主从复制（二进制日志文件和位置）1.修改主库配置文件 vim /etc/my.cnf [mysql] socket=/tmp/mysql.sock [mysqld] user=mysql basedir=/usr/local/mysql datadir=/zy/data socket=/tmp/mysql.sock server-id = 1 # 唯一 ID log_bin=binlog # 启用二进制日志 binlog-do-db = test_db # 需要复制的数据库（多个数据库可重复此参数） expire-logs-days = 7 # 二进制日志过期时间 binlog_format = ROW # 日志格式（推荐 ROW） 2.重启mysql systemctl restart mysqld 3.创建复制用户 mysql -u root -p mysql> CREATE USER 'slave1'@'%' IDENTIFIED WITH mysql_native_password BY 'your_slave_password'; mysql> GRANT REPLICATION SLAVE ON *.* TO 'slave1'@'%'; mysql> FLUSH PRIVILEGES; mysql> SHOW MASTER STATUS; +---------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +---------------+----------+--------------+------------------+-------------------+ | binlog.000008 | 823 | test_db | | | +---------------+----------+--------------+------------------+-------------------+ 1 row in set (0.00 sec) 4.修改从库配置文件 vim /etc/my.cnf [mysql] socket=/tmp/mysql.sock [mysqld] user=mysql basedir=/usr/local/mysql datadir=/zy/data socket=/tmp/mysql.sock server-id = 2 # 每个从库必须唯一 relay-log = mysql-relay-bin # 中继日志 log_bin=binlog # 可选：从库也可作为其他从库的主库 read-only = 1 # 从库设置为只读（超级用户除外） 5.重启mysql systemctl restart mysqld mysql -u root -p 6.配置主从复制 mysql> CHANGE MASTER TO -> MASTER_HOST='10.0.0.114', -> MASTER_USER='slave1', -> MASTER_PASSWORD='slave123456', -> MASTER_LOG_FILE='binlog.000008', -> MASTER_LOG_POS=823; Query OK, 0 rows affected, 8 warnings (0.02 sec) mysql> FLUSH PRIVILEGES; mysql> START SLAVE; #启动复制 mysql> SHOW SLAVE STATUS\G #检查复制状态 #关键状态检查： Slave_IO_Running: Yes # 必须为 Yes Slave_SQL_Running: Yes # 必须为 Yes Seconds_Behind_Master: 0 # 延迟时间，0 表示无延迟12.验证主从复制1. 在主库创建测试数据 CREATE DATABASE test_db; USE test_db; CREATE TABLE users (id INT PRIMARY KEY AUTO_INCREMENT, name VARCHAR(50)); INSERT INTO users (name) VALUES ('Alice'), ('Bob'); 2. 在从库验证数据同步 USE test_db; SELECT * FROM users; +----+-------+ | id | name | +----+-------+ | 1 | Alice | | 2 | Bob | +----+-------+ 2 rows in set (0.00 sec) 13.配置文件优化（生产环境根据业务需求）cat /etc/my.cnf [mysqld] #内存分配（关键）： innodb_buffer_pool_size = 2G # 主库建议较大值 innodb_buffer_pool_instances = 8 # 缓冲池实例数，建议为 CPU 核心数的一半 innodb_log_file_size = 256M # 单个日志文件大小 innodb_log_files_in_group = 3 # 日志文件数量 innodb_log_buffer_size = 16M # 日志缓冲区大小 #写入性能优化： sync_binlog = 1 # 确保二进制日志写入磁盘 innodb_flush_log_at_trx_commit = 2 # 性能与安全的平衡 # 性能与持久性的平衡（0=最快，2=中等，1=最安全） innodb_thread_concurrency = 0 # 0 表示不限制，建议设为 CPU 核心数*2 innodb_read_io_threads = 8 # 读 I/O 线程数 innodb_write_io_threads = 8 # 写 I/O 线程数 max_connections = 1000 # 根据业务需求调整 bind-address = 127.0.0.1 #限制远程访问，仅允许本地访问，通过中间件转发请求 （根据业务需求调整） general_log = 1 general_log_file = /var/log/mysql/mysql.log #审计日志 #查询优化： sort_buffer_size = 4M # 每个排序操作的缓冲区大小 join_buffer_size = 4M # 连接操作的缓冲区大小 slow_query_log = 1 # 启用慢查询日志 slow_query_log_file = /var/log/mysql/slow.log long_query_time = 1 # 超过 1 秒的查询记录 log_queries_not_using_indexes = 1 # 记录未使用索引的查询 #主从复制优化： （1）主库配置： # 二进制日志配置 log-bin = mysql-bin # 启用二进制日志 binlog_format = ROW # 推荐 ROW 格式（更安全，支持更多特性） expire_logs_days = 7 # 日志保留天数 max_binlog_size = 100M # 单个日志文件最大 size binlog_checksum = CRC32 # 日志校验 # GTID 模式（强烈推荐） gtid_mode = ON enforce_gtid_consistency = ON （2）从库配置 # 中继日志 relay_log = mysql-relay-bin relay_log_recovery = 1 # 从库崩溃恢复时自动清理未完成的中继日志 # 复制线程 slave_parallel_type = LOGICAL_CLOCK # 基于逻辑时钟的并行复制 slave_parallel_workers = 8 # 并行工作线程数 master_info_repository = TABLE # 主库信息存储在表中 relay_log_info_repository = TABLE # 中继日志信息存储在表中 # 从库只读（除超级用户外） read_only = 1 super_read_only = 1 # 严格只读，超级用户也无法写入 #安全加固配置： （1）网络与认证 # 绑定 IP，限制外部访问 bind-address = 127.0.0.1 # 仅允许本地访问，通过中间件或代理访问 skip-name-resolve = 1 # 禁用 DNS 解析，提高安全性 # 密码策略（MySQL 8.0 默认要求强密码） default_authentication_plugin = caching_sha2_password # 限制连接超时 wait_timeout = 28800 # 连接闲置超时时间（秒） interactive_timeout = 28800 （2）数据加密 # 开启 TLS（需要生成证书） ssl = ON ssl_ca = /etc/mysql/certs/ca.pem ssl_cert = /etc/mysql/certs/server-cert.pem ssl_key = /etc/mysql/certs/server-key.pem # 加密临时表和临时文件 internal_tmp_disk_storage_engine = InnoDB innodb_temp_data_file_path = ibtmp1:12M:autoextend:max:128M （3）文件权限 # 日志和数据目录权限 log-error = /var/log/mysql/error.log pid-file = /var/run/mysqld/mysqld.pid datadir = /var/lib/mysql log-error = /var/log/mysql/error.log # 错误日志 log_timestamps = SYSTEM # 使用系统时区记录时间 # 确保目录权限正确 # chown -R mysql:mysql /var/lib/mysql # chmod 700 /var/lib/mysql #审计日志： # 启用审计插件（MySQL 企业版或社区版通过插件实现） plugin-load-add = audit_log.so audit_log_format = JSON audit_log_file = /var/log/mysql/audit.log audit_log_policy = ALL # 记录所有操作 audit_log_exclude_accounts = 'repl_user@%' # 排除复制用户 #InnoDB 存储引擎配置 innodb_rollback_segments=128 # 回滚段数量 innodb_undo_log_truncate=1 # 启用 undo 日志截断 innodb_max_undo_log_size=4G # 最大 undo 日志大小 innodb_data_file_path=ibdata1:12M:autoextend # 数据文件路径和初始大小 innodb_log_files_in_group=4 # 日志文件数量 innodb_log_file_size=2G # 单个日志文件大小 innodb_temp_data_file_path=ibtmp1:16M:autoextend # 临时数据文件 innodb_open_files=409600 # 允许打开的 InnoDB 文件数量 innodb_change_buffering=all # 变更缓冲策略（提高写入性能） innodb_adaptive_hash_index=ON # 自适应哈希索引 innodb_autoinc_lock_mode=2 # 自增锁模式（减少锁争用） innodb_buffer_pool_size=128G # 缓冲池大小（关键性能参数） innodb_buffer_pool_instances=8 # 缓冲池实例数 innodb_max_dirty_pages_pct=50 # 脏页最大百分比 innodb_old_blocks_pct=25 # 旧块百分比（LRU 算法） innodb_old_blocks_time=3000 # 旧块停留时间（毫秒） innodb_log_buffer_size=64M # 日志缓冲区大小 innodb_buffer_pool_chunk_size=128M # 缓冲池块大小 innodb_page_cleaners=8 # 页面清理线程数 innodb_flush_method=O_DIRECT # 刷新方法（直接 I/O） innodb_use_native_aio # 使用原生 AIO innodb_adaptive_flushing # 自适应刷新 innodb_flush_log_at_trx_commit=1 # 事务提交时刷新日志（最安全） innodb_io_capacity=20000 # I/O 能力（根据存储性能调整） innodb_read_io_threads=16 # 读 I/O 线程数 innodb_write_io_threads=16 # 写 I/O 线程数 innodb_read_ahead_threshold=56 # 预读阈值 innodb_doublewrite # 双写缓冲区（提高数据安全性） innodb_purge_threads=1 # 清除线程数 innodb_purge_batch_size=20 # 清除批处理大小 innodb_stats_persistent=1 # 持久化统计信息 innodb_flush_neighbors=0 # 刷新相邻页（SSD 建议 0） innodb_file_per_table # 每个表独立文件 innodb_autoextend_increment=32 # 自动扩展增量（MB） innodb_fast_shutdown=1 # 快速关闭模式 autocommit=1 # 自动提交 transaction-isolation=READ-COMMITTED # 事务隔离级别 innodb_table_locks # 启用表锁 lock_wait_timeout=600 # 锁等待超时（秒） innodb_lock_wait_timeout=30 # InnoDB 锁等待超时（秒） innodb_spin_wait_delay=6 # 自旋等待延迟 innodb_sync_spin_loops=30 # 同步自旋循环次数 innodb_commit_concurrency=0 # 提交并发度（0 表示无限制） innodb_thread_concurrency=0 # 线程并发度（0 表示无限制）

部署ELK 1.单点部署es1.下载ES软件 https://www.elastic.co/cn/downloads/past-releases#elasticsearch 2.创建ES的工作目录 [root@elk105 ~]# mkdir /zhuyang/{data,logs,app} -p 3.解压软件包 [root@elk105 app]# tar xf elasticsearch-7.3.1-linux-x86_64.tar.gz 4.创建运行elasticsearch服务用户并修改权限 [root@elk105 ~]# useradd es [root@elk105 app]# chown es.es /zhuyang/app/elasticsearch-7.3.1 -R 5.修改文件描述符最大及进程数量打开上限及 [root@elk105 config]# vim /etc/security/limits.d/es.conf * soft nofile 65535 * hard nofile 131070 * hard nproc 8192 6.修改虚拟内存映射上限 临时修改内核参数(重启OS后配置丢失): [root@elk105 ~]# sysctl -w vm.max_map_count=262144 永久生效内核参数(重启OS后配置不丢失) 强烈推荐 [root@elk105 ~]# vim /etc/sysctl.d/es.conf vm.max_map_count=262144 加载文件，使得生效 [root@elk105 ~]# sysctl -f /etc/sysctl.d/es.conf 查看内核参数: [root@elk105 ~]# sysctl -q vm.max_map_count vm.max_map_count = 262144 7.修改配置文件 [root@elk105 ~]# vim/zhuyang/app/elasticsearch-7.3.1/config/elasticsearch.yml ... network.host: 0.0.0.0 ... discovery.seed_hosts: ["10.0.0.105"] cluster.initial_master_nodes: ["10.0.0.105"] 8.启动es 前台启动: [root@elk105]# su - es -c "/zhuyang/app/elasticsearch-7.3.1/bin/elasticsearch" 后台启动: [root@elk105]# su - es -c "/zhuyang/app/elasticsearch-7.3.1/bin/elasticsearch -d" 9.检查服务是否正常访问 [root@elk105]# curl 10.0.0.105:9200 { "name" : "elk105", "cluster_name" : "elasticsearch", "cluster_uuid" : "aMFqNy-VRi-13t_ZkC0cAg", "version" : { "number" : "7.3.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "4749ba6", "build_date" : "2019-08-19T20:19:25.651794Z", "build_snapshot" : false, "lucene_version" : "8.1.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } 10.编写ES服务的启动脚本 [root@elk105]# vim /usr/lib/systemd/system/es7.service [Unit] Description=www.zyshaby.com After=network.target [Service] User=es Group=es Type=simple ExecStart=/zhuyang/app/elasticsearch-7.3.1/bin/elasticsearch LimitNOFILE=131070 LimitNPROC=65535 [Install] WantedBy=multi-user.target 11.使用systemctl管理es [root@elk105 ~]# systemctl daemon-reload #加载文件 [root@elk105 ~]# systemctl enable --now es7 #开机自启es [root@elk105 ~]# systemctl status es7 #查看es状态 12.使用systemctl管理es前提得先把之前启动的es进程关掉 例如: [root@elk105 0]# ps -ef|grep elastic es 1879 1 2 00:14 ? 00:01:10 /zhuyang/app/elasticsearch-7.3.1/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-1126705440804840429 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Dio.netty.allocator.type=unpooled -XX:MaxDirectMemorySize=536870912 -Des.path.home=/zhuyang/app/elasticsearch-7.3.1 -Des.path.conf=/zhuyang/app/elasticsearch-7.3.1/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /zhuyang/app/elasticsearch-7.3.1/lib/* org.elasticsearch.bootstrap.Elasticsearch -d es 1894 1879 0 00:14 ? 00:00:00 /zhuyang/app/elasticsearch-7.3.1/modules/x-pack-ml/platform/linux-x86_64/bin/controller root 17234 1460 0 00:55 pts/0 00:00:00 grep --color=auto elastic [root@elk105 0]# kill -9 1879 [root@elk105 ~]# ps -ef|grep elastic root 17237 1460 0 00:55 pts/0 00:00:00 grep --color=auto elastic 然后再设置为开机自启 [root@elk105 ~]# systemctl daemon-reload [root@elk105 ~]# systemctl enable --now es72.部署es常见的报错1.java.lang.RuntimeException: can not run elasticsearch as root 错误原因： 运行ES服务不能以root身份运行。 解决方案: 创建普通用户运行即可。 2.could not find java in bundled JDK at /oldboyedu/softwares/elasticsearch-7.17.5/jdk/bin/java 错误原因: 找不到jkd路径，但实际上存在，可能是由于运行ES服务相关文件权限问题。 解决方案: 将ES相关的二进制文件修改为运行用户的身份即可。 3.ERROR: [3] bootstrap checks failed. You must address the points described in the following [3] lines before starting Elasticsearch. bootstrap check failure [1] of [3]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535] bootstrap check failure [2] of [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] bootstrap check failure [3] of [3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured 错误原因: 最大文件描述上限，虚拟机内存大小设置及服务发现未配置。 解决方案: 按照提示解决即可，具体解决方案可参考单点部署。3.jvm调优1.修改ES堆内存大小 [root@elk105 ~]# vim /zhuyang/app/elasticsearch-7.3.1/config/jvm.options ... -Xms256m -Xmx256m ... 2.重启es服务 [root@elk105 ~]# systemctl restart es7 3.验证堆内存是否修改成功 [root@elk105 ~]# ps -ef | grep java | grep -i xms es 17296 1 17 00:55 ? 00:00:59 /zhuyang/app/elasticsearch-7.3.1/jdk/bin/java -Xms256m -Xmx256m 4.安装Oracle JDK1.Oracle JDK下载地址: https://www.oracle.com/java/technologies/downloads/#java8-linux 2.解压软件包 [root@elk105 app]# tar xf jdk-8u191-linux-x64.tar.gz 3.配置JDK环境变量 [root@elk105 app]# cat /etc/profile.d/jdk.sh export JAVA_HOME=/zhuyang/app/jdk1.8.0_191 export PATH=$PATH:$JAVA_HOME/bin [root@elk105 app]# source /etc/profile.d/jdk.sh [root@elk105 app]# java -version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) 4.修改ES启动脚本传递JAVA_HOME变量 [root@elk105 app]# cat /usr/lib/systemd/system/es7.service [Unit] Description=www.zyshaby.com After=network.target [Service] User=es Group=es Type=simple ExecStart=/zhuyang/app/elasticsearch-7.3.1/bin/elasticsearch LimitNOFILE=131070 LimitNPROC=65535 Environment=JAVA_HOME=/zhuyang/app/jdk1.8.0_191 [Install] WantedBy=multi-user.target 5.重启es [root@elk105 app]# systemctl daemon-reload [root@elk105 app]# systemctl restart es7 [root@elk105 app]# systemctl status es7 6.验证命令 [root@elk105 app]# jps # 仅显示java程序的名称和PID 2164 Jps 2061 Elasticsearch [root@elk105 app]# jps -l # 查看java的包名 [root@elk105 app]# jps -v # 查看java程序的详细信息。5.日常查看补充1.查看服务状态 [root@elk105 ~]# systemctl status es7 -l 2.通过系统日志查看 [root@elk105 ~]# journalctl -u es7 3.查看启动日志 [root@elk105 ~]# tail -100f /zhuyang/app/elasticsearch-7.3.1/logs/elasticsearch.log 6.集群部署es1.创建数据目录和工作目录 [root@elk105 ~]# install -d /zhuyang/{data,logs}/es7 -o es -g es #递归创建目录并修改目录属主属组权限 2.修改es配置文件 [root@elk105 ~]# vim /zhuyang/app/elasticsearch-7.3.1/config/elasticsearch.yml # 指定集群名称 cluster.name: my-es7 # 指定数据目录 path.data: /zhuyang/data/es7 # 指定日志目录 path.logs: /zhuyang/logs/es7 # 对外暴露的IP地址 network.host: 0.0.0.0 # 指定集群外部提供的http端口,默认端口 http.port: 9200 # 指定集群内部数据传输端口，默认端口 transport.tcp.port: 9300 # 指定现有集群的主机地址 discovery.seed_hosts: ["10.0.0.105","10.0.0.106","10.0.0.107"] # 指定集群启动时，初始参与master选举的节点 cluster.initial_master_nodes: ["10.0.0.105","10.0.0.106","10.0.0.107"] 3.拷贝数据目录 [root@elk105 ~]# scp -r /zhuyang/ 10.0.0.106:/ [root@elk105 ~]# scp -r /zhuyang/ 10.0.0.107:/ [root@elk105 ~]# scp /etc/profile.d/jdk.sh 10.0.0.106:/etc/profile.d/ [root@elk105 ~]# scp /etc/profile.d/jdk.sh 10.0.0.107:/etc/profile.d/ [root@elk105 ~]# scp /etc/sysctl.d/es.conf 10.0.0.106:/etc/sysctl.d/ [root@elk105 ~]# scp /etc/sysctl.d/es.conf 10.0.0.107:/etc/sysctl.d/ [root@elk105 ~]# scp /usr/lib/systemd/system/es7.service 10.0.0.106:/usr/lib/systemd/system/ [root@elk105 ~]# scp /usr/lib/systemd/system/es7.service 10.0.0.107:/usr/lib/systemd/system/ 4.其他节点创建用户，修改权限，内核参数调优 [root@elk106 ~]# useradd -u 1000 es [root@elk106 ~]# chown -R es.es /zhuyang/app/elasticsearch-7.3.1 [root@elk106 ~]# install -d /zhuyang/{data,logs}/es7 -o es -g es [root@elk106 ~]# sysctl -f /etc/sysctl.d/es.conf [root@elk107 ~]# useradd -u 1000 es [root@elk107 ~]# chown -R es.es /zhuyang/app/elasticsearch-7.3.1 [root@elk107 ~]# install -d /zhuyang/{data,logs}/es7 -o es -g es [root@elk107 ~]# sysctl -f /etc/sysctl.d/es.conf 5.清空单点的数据 [root@elk105 ~]# systemctl stop es7 [root@elk105 ~]# pkill java [root@elk105 ~]# rm -rf /zhuyang/data/es7/* /zhuyang/logs/es7/* /tmp/* 6.所有节点同时启动服务 [root@elk105 ~]# systemctl enable --now es7 [root@elk106 ~]# systemctl enable --now es7 [root@elk107 ~]# systemctl enable --now es7 7.验证服务是否正常 [root@elk105 ~]# systemctl status es7 [root@elk106 ~]# systemctl status es7 [root@elk107 ~]# systemctl status es7 [root@elk105 ~]# ss -ntl | grep 9200 [root@elk106 ~]# ss -ntl | grep 9200 [root@elk107 ~]# ss -ntl | grep 9200 [root@elk105 ~]# curl 10.0.0.105:9200 { "name" : "elk105", "cluster_name" : "my-es7", "cluster_uuid" : "JyST2NtrQH6VNQl0ZHpq3g", "version" : { "number" : "7.3.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "4749ba6", "build_date" : "2019-08-19T20:19:25.651794Z", "build_snapshot" : false, "lucene_version" : "8.1.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } [root@elk105 ~]# curl 10.0.0.105:9200/_cat/nodes 10.0.0.105 34 96 0 0.11 0.16 0.11 dim - elk105 10.0.0.106 42 96 0 0.23 0.29 0.17 dim * elk106 10.0.0.107 42 96 0 0.07 0.16 0.11 dim - elk107 ES集群处于不健康状态 [root@elk105 ~]# curl 10.0.0.105:9200 { "name" : "elk105", "cluster_name" : "my-es7", "cluster_uuid" : "_na_", "version" : { "number" : "7.17.5", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84", "build_date" : "2022-06-23T21:57:28.736740635Z", "build_snapshot" : false, "lucene_version" : "8.11.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } [root@elk105 ~]# curl 10.0.0.105:9200/_cat/nodes {"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503} 解决方案: 重复集群部署笔记的5-7步骤即可7.配置集群免密登录及同步脚本1.修改主机列表 [root@elk105 ~]# vim /etc/hosts 10.0.0.105 elk105 10.0.0.106 elk106 10.0.0.107 elk107 2.elk105节点上生成密钥对 [root@elk105 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa -q 3.elk101配置所有集群节点的免密登录 [root@elk105 ~]# for ((host_id=105;host_id<=107;host_id++));do ssh-copy-id elk${host_id} ;done 4.链接测试 [root@elk105 ~]# ssh elk106 Last login: Tue Jun 18 21:07:12 2024 from 10.0.0.1 [root@elk106 ~]# exit logout Connection to elk106 closed. [root@elk105 ~]# ssh elk107 Last login: Tue Jun 18 21:07:16 2024 from 10.0.0.1 [root@elk107 ~]# 5.所有节点安装rsync数据同步工具 [root@elk105 ~]# yum -y install rsync [root@elk106 ~]# yum -y install rsync [root@elk107 ~]# yum -y install rsync 6.编写同步脚本 [root@elk105 ~]# vim /usr/local/sbin/data_rsync.sh #!/bin/bash # Auther: Zhu Yang if [ $# -ne 1 ];then echo "Usage: $0 /path/to/file(绝对路径)" exit fi # 判断文件是否存在 if [ ! -e $1 ];then echo "[ $1 ] dir or file not find!" exit fi # 获取父路径 fullpath=`dirname $1` # 获取子路径 basename=`basename $1` # 进入到父路径 cd $fullpath for ((host_id=106;host_id<=107;host_id++)) do # 使得终端输出变为绿色 tput setaf 2 echo ===== rsyncing elk${host_id}: $basename ===== # 使得终端恢复原来的颜色 tput setaf 7 # 将数据同步到其他两个节点 rsync -apz $basename `whoami`@elk${host_id}:$fullpath if [ $? -eq 0 ];then echo "命令执行成功!" fi done [root@elk105 ~]# chmod +x /usr/local/sbin/data_rsync.sh #给脚本授权8.配置ES集群加密1.生成证书文件 [root@elk105 ~]# cd /zhuyang/app/elasticsearch-7.3.1/ [root@elk105 elasticsearch-7.3.1]# ./bin/elasticsearch-certutil cert -out config/elasticsearch-certificates.p12 -pass "" 2.为证书文件修改属主和属组 [root@elk105 elasticsearch-7.3.1]# chown es:es config/elasticsearch-certificates.p12 3.同步证书文件到其他节点 [root@elk105 elasticsearch-7.3.1]# data_rsync.sh /zhuyang/app/elasticsearch-7.3.1/config/elasticsearch-certificates.p12 ===== rsyncing elk106: elasticsearch-certificates.p12 ===== 命令执行成功! ===== rsyncing elk107: elasticsearch-certificates.p12 ===== 命令执行成功! 4.修改ES集群的配置文件 [root@elk105 elasticsearch-7.3.1]# vim config/elasticsearch.yml ... # 在最后一行添加以下内容 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /zhuyang/app/elasticsearch-7.3.1/config/elasticsearch-certificates.p12 xpack.security.transport.ssl.truststore.path: /zhuyang/app/elasticsearch-7.3.1/config/elasticsearch-certificates.p12 #这里是指定证书的位置，可以用绝对路劲和当前路劲都可以 5.同步ES配置文件到其他节点 [root@elk105 elasticsearch-7.3.1]# data_rsync.sh /zhuyang/app/elasticsearch-7.3.1/config/elasticsearch.yml 6.所有节点重启ES集群 [root@elk105 ~]# systemctl restart es7 [root@elk106 ~]# systemctl restart es7 [root@elk107 ~]# systemctl restart es7 7.更改es集群密码 [root@elk105 bin]# ./elasticsearch-setup-passwords interactive 8.验证es集群 [root@elk105 bin]# curl -u elastic:Zy200211@ 10.0.0.105:9200/_cat/nodes 10.0.0.105 51 63 2 1.21 0.37 0.17 dim * elk105 10.0.0.107 43 62 1 0.49 0.19 0.13 dim - elk107 10.0.0.106 32 62 1 1.10 0.31 0.14 dim - elk106 #登录es-head访问部署kibana1.下载kibana软件https://www.elastic.co/cn/downloads/past-releases/kibana-7-3-12.解压软件包[root@elk105 app]# tar xf kibana-7.3.1-linux-x86_64.tar.gz3.修改配置文件[root@elk105 app]# vim kibana-7.3.1-linux-x86_64/config/kibana.yml ... server.host: "0.0.0.0" ... elasticsearch.hosts: ["http://10.0.0.105:9200","http://10.0.0.106:9200","http://10.0.0.107:9200"] ... elasticsearch.username: "elastic" elasticsearch.password: "Zy200211@" ... i18n.locale: "zh-CN"4.启动kibana[root@elk105 kibana-7.3.1-linux-x86_64]# ./bin/kibana #前台启动 [root@elk105 kibana-7.3.1-linux-x86_64]# nohup ./bin/kibana > /zhuyang/logs/kibana.log 2>&1 & #后台启动5.配置systemctl管理kibana1.创建kibana用户 [root@elk105 app]# useradd kibana 2.授权 [root@elk105 app]# chown -R kibana:kibana /zhuyang/app/kibana-7.3.1-linux-x86_64 3.配置kibana.service文件 [root@elk105 app]# cat /usr/lib/systemd/system/kibana.service [Unit] Description=Kibana Documentation=https://www.elastic.co/guide/en/kibana/current/index.html Wants=network-online.target After=network-online.target [Service] Type=simple User=kibana Group=kibana ExecStart=/zhuyang/app/kibana-7.3.1-linux-x86_64/bin/kibana # StandardOutput=/zhuyang/logs/kibana.log # StandardError=/zhuyang/logs/kibana.log Restart=on-failure [Install] WantedBy=multi-user.target 4.使用systemctl启动kibana [root@elk105 ~]# systemctl daemon-reload [root@elk105 ~]# systemctl start kibana 5.验证是否启动成功 [root@elk105 ~]# ss -ntl|grep 5601 [root@elk105 ~]# ps -ef|grep kibana 登录web访问UI: http://10.0.0.105:5601/部署logstash1.下载logstash软件https://artifacts.elastic.co/downloads/logstash/2.解压软件包[root@elk105 app]# tar xf logstash-7.3.1.tar.gz 3.修改配置文件[root@elk105 config]# vim logstash.conf # Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input{ tcp { mode => "server" host => "0.0.0.0" port => 4569 codec => "json" } } output{ elasticsearch { hosts => ["10.0.0.105:9200","10.0.0.106:9200","10.0.0.107:9200"] index => "logstash-rollover" user => "elastic" password => "Zy200211@" } } 4.启动logstash[root@elk105 bin]# ./logstash -f ../config #前台启动 [root@elk105 ~]# nohup /zhuyang/app/logstash-7.3.1/bin/logstash -f /zhuyang/app/logstash-7.3.1/config/logstash.conf > /zhuyang/logs/logstash/logstash.log 2>&1 & #后台启动5.验证logstash[root@elk105 bin]# ps -ef|grep logstash [root@elk105 bin]# ss -ntl |grep 4569