Centos 6.4 搭建ELK（1）-FinClip官网

Centos 6.4 搭建ELK（1）

网友投稿 612 2022-09-21

Centos 6.4 搭建ELK（1）

前段时间用ossec收集了一些系统的日志(syslog、secure、maillog等)，看了下elk这个架构，发现很适合ossec，也很好玩。

一、介绍：

elk官网它用来对日志进行收集、分割、集中日志平台

elasticsearch: 开源分布式搜索引擎,提供搜索功能，并用来存储最终的数据

kibana: web页面展示，支持各种查询、统计和展示

2、工作流程：

(1)、ossec client通过1514端口把日志发送给ossec server（存储在/var/logs/ossec/alerts/alerts.log），logstash-shipper把ossec server的所有日志分割，并将分割后的日志内容发给redis。

(2)、redis作为ossec server和logstash indexer之间的缓冲区，用来提升系统性能与可靠性,当logstash提取数据失败时,数据保存在redis中,不至于丢失。

(3)、logstash indexer提取redis的日志，将日志收集在一起(负责汇总数据)。

(4)、logstash indexer再把数据交给elasticsearch，elasticsearch存储最终的数据，并提供搜索功能。

(5)、最后通过kibana提供日志分析的web界面。

三、安装elk：

1、elk包

elk更新很快，版本众多，如果选择版本不一致，可能没办法使用。

如果安装最新版本elk，logstash2.x配置要更改，如果使用logstash1.52的配置，会报错。

elk有3种安装方式，我这里选择tar.gz包来安装。

logstash-1.5.2.tar.gz

elasticsearch-1.6.0.tar.gz

kibana-4.1.1-linux-x64.tar.gz

redis-3.0.6.tar.gz

2、服务器IP

ossec client：192.168.153.187

ossec server：192.168.153.172（安装ossec server和logstash，把这台服务器看成是logstash的client（即logstash-shipper）

elk+redis：192.168.153.200（这个logstash是server,即indexer）

3、安装过程

(1)、192.168.153.187

安装 ossec client，安装见之前的博客

(2)、192.168.153.172

安装 ossec server，安装见之前的博客

安装logstash

logstash依赖jdk的，安装jdk

[root@elk-redis ~]# yum install java-1.8.0-openjdk

[root@elk-redis ~]# java -version

openjdk version "1.8.0_91"

[root@ossec-server ~]# wget ~]# tar -xf logstash-1.5.2.tar.gz -C /usr/local/

后台运行logstash

[root@ossec-server ~]# /usr/local/logstash-1.5.2/bin/logstash -f /usr/local/logstash-1.5.2/logstash-200.conf &

Logstash startup completed

{

"@timestamp" => "2016-05-19T02:03:22.746Z",

"@version" => "1",

"ossec_group" => "pam,syslog,",

"reporting_ip" => "192.168.153.187",

"reporting_source" => "/var/log/secure",

"rule_number" => "5502",

"severity" => 3,

"signature" => "Login session closed.",

"@message" => "May 19 10:03:57 localhost sshd[4623]: pam_unix(sshd:session): session closed for user root",

"@fields.hostname" => "agent15",

"@fields.product" => "ossec",

"raw_message" => "** Alert 1463623401.3764: - pam,syslog,\n2016 May 19 10:03:21 (agent15) 192.168.153.187-

>/var/log/secure\nRule: 5502 (level 3) -> 'Login session closed.'\nMay 19 10:03:57 localhost sshd[4623]: pam_unix

(sshd:session): session closed for user root",

"ossec_server" => "ossec-server"

}

{

"@timestamp" => "2016-05-19T02:03:58.846Z",

"@version" => "1",

"ossec_group" => "syslog,sshd,authentication_success,",

"reporting_source" => "192.168.153.172",

"rule_number" => "5715",

"severity" => 3,

"signature" => "SSHD authentication success.",

"src_ip" => "192.168.153.1",

"acct" => "root",

"@message" => "May 19 10:03:57 ossec-server sshd[22805]: Accepted password for root from 192.168.153.1 port 31490

ssh2",

"@fields.hostname" => "ossec-server",

"@fields.product" => "ossec",

"raw_message" => "** Alert 1463623437.4008: - syslog,sshd,authentication_success,\n2016 May 19 10:03:57 ossec-server-

>192.168.153.172\nRule: 5715 (level 3) -> 'SSHD authentication success.'\nSrc IP: 192.168.153.1\nUser: root\nMay 19 10:03:57

ossec-server sshd[22805]: Accepted password for root from 192.168.153.1 port 31490 ssh2",

"ossec_server" => "ossec-server"

(3)、192.168.153.200

a、安装elasticsearch

elasticsearch是依赖jdk的，所以先安装jdk

[root@elk-redis ~]# yum install java-1.8.0-openjdk

[root@elk-redis ~]# java -version

openjdk version "1.8.0_91"

[root@elk-redis ~]# tar -xf elasticsearch-1.6.0.tar.gz -C /usr/local/

后台启动Elasticsearch

[root@elk-redis ~]# /usr/local/elasticsearch-1.6.0/bin/elasticsearch -d

访问192.168.153.200:9200端口，200表明es启动成功

[root@elk-redis ~]# curl : 200,

"name" : "elasticsearch-node01",

"cluster_name" : "elasticsearch",

"version" : {

"number" : "1.6.0",

"build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",

"build_timestamp" : "2015-06-09T13:36:34Z",

"build_snapshot" : false,

"lucene_version" : "4.10.4"

"tagline" : "You Know, for Search"

}

b、安装redis 3.0.6

[root@elk-redis ~]# tar zxvf redis-3.0.6.tar.gz

[root@elk-redis ~]# cd redis-3.0.6

[root@elk-redis ~]# make PREFIX=/usr/local/redis install

//这里纠结一下, redis如果不指定prefix路径,那么默认会在你这个解压的文件夹中编译生成bin文件

[root@elk-redis ~]# ln -sv /usr/local/redis/bin/redis-server /usr/bin/redis-server

[root@elk-redis ~]# ln -sv /usr/local/redis/bin/redis-cli /usr/bin/redis-cli

[root@elk-redis ~]# cp tmp/redis-3.0.6/utils/redis_init_script /etc/rc.d/init.d/redis

配置redis

[root@elk-redis ~]# vi /etc/rc.d/init.d/redis.conf

//然后在第二行插入chkconfig配置,然后修改EXEC和CLI,我的这个文件前几行是这样的

#!/bin/sh

# chkconfig: 2345 90 10

# Simple Redis init.d script conceived to work on Linux systems

# as it does use of the /proc filesystem.

REDISPORT=6379

EXEC=/usr/local/redis/bin/redis-server

CLIEXEC=/usr/local/redis/bin/redis-cli

PIDFILE=/var/run/redis_${REDISPORT}.pid

CONF="/etc/redis/${REDISPORT}.conf"

[root@elk-redis ~]# mkdir /etc/redis/

//这个目录用于放我们的配置文件

[root@elk-redis ~]# mkdir /var/rdb/

//这个目录存放redis的数据库文件

redis源码包中自带redis.conf,但这个只是模版,具体配置根据自己的环境设置

[root@elk-redis ~]# vi /etc/redis/redis.conf

启动redis

[root@elk-redis ~]# /etc/init.d/redis start

Starting Redis server...

1447:M 18 May 17:03:50.342 * Increased maximum number of open files to 10032 (it was originally set to 1024).

_._

_.-``__ ''-._

_.-`` `. `_. ''-._ Redis 3.0.6 (00000000/0) 64 bit

.-`` .-```. ```\/ _.,_ ''-._

( ' , .-` | `, ) Running in standalone mode

|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379

| `-._ `._ / _.-' | PID: 1447

`-._ `-._ `-./ _.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' | `-._`-.__.-'_.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' |

`-._ `-._`-.__.-'_.-' _.-'

`-._ `-.__.-' _.-'

`-._ _.-'

`-.__.-'

1447:M 18 May 17:03:50.345 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is

set to the lower value of 128.

1447:M 18 May 17:03:50.346 # Server started, Redis version 3.0.6

1447:M 18 May 17:03:50.346 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix

this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl

vm.overcommit_memory=1' for this to take effect.

1447:M 18 May 17:03:50.346 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create

latency and memory usage issues with Redis. To fix this issue run the command 'echo never >

/sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a

reboot. Redis must be restarted after THP is disabled.

1447:M 18 May 17:03:50.357 * DB loaded from disk: 0.011 seconds

1447:M 18 May 17:03:50.357 * The server is now ready to accept connections on port 6379

1447:M 18 May 17:21:03.197 * 1 changes in 900 seconds. Saving...

1447:M 18 May 17:21:03.198 * Background saving started by pid 1466

1466:C 18 May 17:21:03.202 * DB saved on disk

1466:C 18 May 17:21:03.202 * RDB: 0 MB of memory used by copy-on-write

1447:M 18 May 17:21:03.299 * Background saving terminated with success

1447:M 18 May 17:26:04.090 * 10 changes in 300 seconds. Saving...

1447:M 18 May 17:26:04.090 * Background saving started by pid 1468

1468:C 18 May 17:26:04.104 * DB saved on disk

[root@elk-redis]# redis-cli

127.0.0.1:6379> MONITOR

1463623574.234636 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623575.258853 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623575.453969 [0 192.168.153.172:36662] "rpush" "logstash:redis" "{\"@timestamp\":\"2016-05-19T02:03:58.848Z\",\"@version

\":\"1\",\"ossec_group\":\"pam,syslog,authentication_success,\",\"reporting_source\":\"192.168.153.172\",\"rule_number\":

\"5501\",\"severity\":3,\"signature\":\"Login session opened.\",\"@message\":\"May 19 10:03:57 ossec-server sshd[22805]:

pam_unix(sshd:session): session opened for user root by (uid=0)\",\"@fields.hostname\":\"ossec-server\",\"@fields.product\":

\"ossec\",\"raw_message\":\"** Alert 1463623437.4316: - pam,syslog,authentication_success,\\n2016 May 19 10:03:57 ossec-

server->192.168.153.172\\nRule: 5501 (level 3) -> 'Login session opened.'\\nMay 19 10:03:57 ossec-server sshd[22805]: pam_unix

(sshd:session): session opened for user root by (uid=0)\",\"ossec_server\":\"ossec-server\"}"

1463623575.456066 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623576.477031 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623601.018922 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623601.534860 [0 192.168.153.172:36662] "rpush" "logstash:redis" "{\"@timestamp\":\"2016-05-19T02:05:17.007Z\",\"@version

\":\"1\",\"ossec_group\":\"pam,syslog,\",\"reporting_source\":\"192.168.153.172\",\"rule_number\":\"5502\",\"severity\":3,

\"signature\":\"Login session closed.\",\"@message\":\"May 19 10:05:16 ossec-server sshd[22805]: pam_unix(sshd:session):

session closed for user root\",\"@fields.hostname\":\"ossec-server\",\"@fields.product\":\"ossec\",\"raw_message\":\"** Alert

1463623516.4585: - pam,syslog,\\n2016 May 19 10:05:16 ossec-server->192.168.153.172\\nRule: 5502 (level 3) -> 'Login session

closed.'\\nMay 19 10:05:16 ossec-server sshd[22805]: pam_unix(sshd:session): session closed for user root\",\"ossec_server\":

\"ossec-server\"}"

1463623601.542622 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623601.562655 [0 192.168.153.172:36662] "rpush" "logstash:redis" "{\"@timestamp\":\"2016-05-19T02:05:43.092Z\",\"@version

\":\"1\",\"ossec_group\":\"syslog,sshd,authentication_success,\",\"reporting_ip\":\"192.168.153.187\",\"reporting_source\":

\"/var/log/secure\",\"rule_number\":\"5715\",\"severity\":3,\"signature\":\"SSHD authentication success.\",\"src_ip\":

\"192.168.153.1\",\"acct\":\"root\",\"@message\":\"May 19 10:06:18 localhost sshd[4834]: Accepted password for root from

192.168.153.1 port 31537 ssh2\",\"@fields.hostname\":\"agent15\",\"@fields.product\":\"ossec\",\"raw_message\":\"** Alert

1463623542.4820: - syslog,sshd,authentication_success,\\n2016 May 19 10:05:42 (agent15) 192.168.153.187->/var/log/secure\

\nRule: 5715 (level 3) -> 'SSHD authentication success.'\\nSrc IP: 192.168.153.1\\nUser: root\\nMay 19 10:06:18 localhost sshd

[4834]: Accepted password for root from 192.168.153.1 port 31537 ssh2\",\"ossec_server\":\"ossec-server\"}"

c、redis设置密码访问

[root@elk-redis ~]# vi /etc/redis/redis.conf #此文件默认在根目录下。

# requirepass foobared去掉注释，foobared改为自己的密码，我在这里改为

requirepass xxxxxxxx

重启服务

[root@elk-redis ~]# /etc/init.d/redis restart

测试连接：./redis-cli -h 192.168.153.200 -p 6379

输入命令会提示(error) NOAUTH Authentication required. 这是属于正常现象。

我们输入 auth xxxxxxxx #你刚才设置的密码

d、安装logstash

[root@elk-redis ~]# wget ~]# tar -xf logstash-1.5.2.tar.gz -C /usr/local/

logstash配置文件

[root@elk-redis ~]# cat /usr/local/logstash-1.5.2/logstash-ossec.conf

input {

redis

{

host => "127.0.0.1"

data_type =>"list"

port => "6379"

key => "logstash:redis"

type => "ossec"

}

output {

stdout { codec => rubydebug }

if [type] == "ossec" {

elasticsearch {

host => "127.0.0.1"

port => "9300"

#cluster => "ossec"

index => "logstash-ossec-%{+YYYY.MM.dd}"

document_type => "ossec"

template_name => "template-ossec"

template => "/usr/local/share/logstash/elasticsearch_template.json"

template_overwrite => true

}

后台运行logstash

[root@elk-redis ~]# /usr/local/logstash-1.5.2/bin/logstash -f /usr/local/logstash-1.5.2/logstash-ossec.conf &

{

"@timestamp" => "2016-05-19T02:05:43.103Z",

"@version" => "1",

"ossec_group" => "pam,syslog,authentication_success,",

"reporting_ip" => "192.168.153.187",

"reporting_source" => "/var/log/secure",

"rule_number" => "5501",

"severity" => 3,

"signature" => "Login session opened.",

"@message" => "May 19 10:06:18 localhost sshd[4834]: pam_unix(sshd:session): session opened for user root by

(uid=0)",

"@fields.hostname" => "agent15",

"@fields.product" => "ossec",

"raw_message" => "** Alert 1463623542.5137: - pam,syslog,authentication_success,\n2016 May 19 10:05:42 (agent15)

192.168.153.187->/var/log/secure\nRule: 5501 (level 3) -> 'Login session opened.'\nMay 19 10:06:18 localhost sshd[4834]:

pam_unix(sshd:session): session opened for user root by (uid=0)",

"ossec_server" => "ossec-server",

"type" => "ossec"

e、安装kibana

[root@elk-redis ~]# tar -xf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/

[root@elk-redis ~]# nohup /usr/local/kibana-4.1.1-linux-x64/bin/kibana &

(4)、访问kibana

http://192.168.153.200:5601

elk安装参考文章

http://baidu.blog./71938/1676798

信创国产化如何推动技术自主创新与安全保障的未来发展

612 2022-09-21

Centos 6.4 搭建ELK（1）

信创国产化如何推动技术自主创新与安全保障的未来发展

信创国产化替换如何推动企业自主创新与市场竞争力提升

信创国产化政策如何推动企业技术转型与市场竞争力提升

最近发表

更多内容

小程序SDK

Finclip技术文档

小程序开发

小程序容器

小程序框架

Finclip小程序平台

Finclip用户投稿

车联网

推荐文章

小程序SDK是什么意思？小程序sdk和插件有什么区别？

小程序支付功能怎么实现？

企业app开发流程是什么？

app运营模式有哪些？

小程序多端引流怎么做？

小程序生态分析的机会和威胁

Flutter入门这一篇效率文章就够了

原生与跨平台解决方案分析,跨平台软件开发技术方案

热更新技术：让软件更新变得更加轻松快速

解决方案

银行解决方案

证券解决方案

互联网解决方案

政企OA解决方案

科技解决方案

loT解决方案

信任解决方案

热评文章

AppCan:基于混合模式的移动应用开发,移动混合模

Hybrid App混合模式开发的了解

小程序容器技术助力券商数字营销突围，小程序容器化的意

用mpvue开发微信小程序基础知识（vue.js开发

小程序多端框架全面测评对比，强烈推荐！

券商app架构 - 解析券商应用程序的构建与设计