Deploy ELK on CentOS 6.8
Adamhuan发表于 2016-07-18 05:47:38
本文仅会涉及到一台服务器。
如题所示，这台服务器为：CentOS 6.8。
[root@elk ~]# cat /etc/redhat-release CentOS release 6.8 (Final) [root@elk ~]#
关于这台服务器的其他相关信息如下：
[root@elk ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:CC:EC:E9
inet addr:192.168.232.151 Bcast:192.168.232.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fecc:ece9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13209 errors:0 dropped:0 overruns:0 frame:0
TX packets:6425 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:16349985 (15.5 MiB) TX bytes:396212 (386.9 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:960 (960.0 b) TX bytes:960 (960.0 b)
[root@elk ~]#
[root@elk ~]# sestatus
SELinux status: disabled
[root@elk ~]#
[root@elk ~]# service iptables status
iptables: Firewall is not running.
[root@elk ~]#
[root@elk ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.232.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0
0.0.0.0 192.168.232.2 0.0.0.0 UG 0 0 0 eth0
[root@elk ~]#
[root@elk ~]# mkdir /software
[root@elk ~]# mkdir /iso
[root@elk ~]# mkdir /backup
[root@elk ~]# mkdir /script
[root@elk ~]#
[root@elk ~]# hostname
elk
[root@elk ~]#
[root@elk ~]# cat /etc/hosts
# Local
127.0.0.1 localhost
# Pub
192.168.232.151 elk
[root@elk ~]#
[root@elk ~]# cat /etc/resolv.conf
# Generated by NetworkManager
domain localdomain
search localdomain
nameserver 192.168.232.2
[root@elk ~]#
[root@elk ~]# ping -c 3 baidu.com
PING baidu.com (111.13.101.208) 56(84) bytes of data.
64 bytes from 111.13.101.208: icmp_seq=1 ttl=128 time=36.8 ms
64 bytes from 111.13.101.208: icmp_seq=2 ttl=128 time=37.7 ms
64 bytes from 111.13.101.208: icmp_seq=3 ttl=128 time=48.4 ms
--- baidu.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 7706ms
rtt min/avg/max/mdev = 36.875/41.016/48.458/5.278 ms
[root@elk ~]#
我将会在这台服务器上搭建ELK架构：Elastic Search + Log Stash + Kibana。
下面开始：
——————————————————————
基础配置。
1. Java 7：
Java的最新版本的各个平台的安装包可以从Oracle的官方网站的下载页面获得：
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
下载完成后，上传介质包到服务器：
[root@elk ~]# ls /software -ltr total 156412 -rw-r--r-- 1 root root 160162581 Jul 11 23:06 jdk-8u91-linux-x64.rpm [root@elk ~]# [root@elk ~]# du -sh /software/* 153M /software/jdk-8u91-linux-x64.rpm [root@elk ~]#
执行安装：
[root@elk ~]# rpm -ivh /software/jdk-8u91-linux-x64.rpm
Preparing... ########################################### [100%]
1:jdk1.8.0_91 ########################################### [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
jfxrt.jar...
[root@elk ~]#安装后校验：
[root@elk ~]# rpm -qa | grep jdk jdk1.8.0_91-1.8.0_91-fcs.x86_64 [root@elk ~]# [root@elk ~]# rpm -ql jdk1.8.0_91-1.8.0_91-fcs.x86_64 | head -n 12 /usr /usr/java /usr/java/jdk1.8.0_91 /usr/java/jdk1.8.0_91/.java /usr/java/jdk1.8.0_91/.java/.systemPrefs /usr/java/jdk1.8.0_91/.java/.systemPrefs/.system.lock /usr/java/jdk1.8.0_91/.java/.systemPrefs/.systemRootModFile /usr/java/jdk1.8.0_91/.java/init.d/jexec /usr/java/jdk1.8.0_91/COPYRIGHT /usr/java/jdk1.8.0_91/LICENSE /usr/java/jdk1.8.0_91/README.html /usr/java/jdk1.8.0_91/THIRDPARTYLICENSEREADME-JAVAFX.txt [root@elk ~]# [root@elk ~]# whereis java java: /usr/bin/java /usr/share/man/man1/java.1 [root@elk ~]# [root@elk ~]# java -version java version "1.8.0_91" Java(TM) SE Runtime Environment (build 1.8.0_91-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode) [root@elk ~]#
这样，Java的支持，就安装成功了。
在ELK的架构中，需要Java支持的是：
Elasticsearch
Logstash
2. 配置elastic的YUM源：
ELK是Elastic公司全线提供的产品，所以配置了elastic的YUM源，就可以很方便的执行安装了。
[root@elk ~]# cat /etc/yum.repos.d/elastic.repo [logstash-2.1] name=Logstash repository for 2.1.x packages baseurl=http://packages.elastic.co/logstash/2.1/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 [elasticsearch-2.x] name=Elasticsearch repository for 2.x packages baseurl=http://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 [root@elk ~]#
3. 打开YUM缓存：
[root@elk ~]# cat /etc/yum.conf [main] cachedir=/var/cache/yum/$basearch/$releasever keepcache=1 debuglevel=2 logfile=/var/log/yum.log exactarch=1 obsoletes=1 gpgcheck=1 plugins=1 installonly_limit=5 bugtracker_url=http://bugs.centos.org/set_project.php?project_id=19&ref=http://bugs.centos.org/bug_report_page.php?category=yum distroverpkg=centos-release # This is the default, if you make this bigger yum won't see if the metadata # is newer on the remote and so you'll "gain" the bandwidth of not having to # download the new metadata and "pay" for it by yum not having correct # information. # It is esp. important, to have correct metadata, for distributions like # Fedora which don't keep old packages around. If you don't like this checking # interupting your command line usage, it's much better to have something # manually check the metadata once an hour (yum-updatesd will do this). # metadata_expire=90m # PUT YOUR REPOS HERE OR IN separate files named file.repo # in /etc/yum.repos.d [root@elk ~]#
4. 更新YUM本地数据库记录：
[root@elk ~]# yum repolist Loaded plugins: fastestmirror, refresh-packagekit, security Loading mirror speeds from cached hostfile * base: centos.ustc.edu.cn * extras: mirrors.sina.cn * updates: centos.ustc.edu.cn base | 3.7 kB 00:00 elasticsearch-2.x | 2.9 kB 00:00 elasticsearch-2.x/primary_db | 6.8 kB 00:00 extras | 3.4 kB 00:00 logstash-2.1 | 951 B 00:00 logstash-2.1/primary | 2.6 kB 00:00 logstash-2.1 4/4 updates | 3.4 kB 00:00 repo id repo name status base CentOS-6 - Base 6,696 elasticsearch-2.x Elasticsearch repository for 2.x packages 14 extras CentOS-6 - Extras 62 logstash-2.1 Logstash repository for 2.1.x packages 4 updates CentOS-6 - Updates 245 repolist: 7,021 [root@elk ~]#
5. 导入GPG校验秘钥：
[root@elk ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch [root@elk ~]# echo $? 0 [root@elk ~]#
6. 从官方网站获取安装介质：
如果你不想通过YUM的方式获取安装介质，你可以访问官网获得：
https://www.elastic.co/downloads
Elasticsearch：https://www.elastic.co/downloads/elasticsearch
Logstash：https://www.elastic.co/downloads/logstash
Kibana：https://www.elastic.co/downloads/kibana
7. 通过YUM安装：
elasticsearch
[root@elk ~]# yum list | grep --color elasticsearch elasticsearch.noarch 2.3.4-1 elasticsearch-2.x pcp-pmda-elasticsearch.x86_64 3.10.9-6.el6 base rsyslog7-elasticsearch.x86_64 7.4.10-5.el6 base [root@elk ~]#
logstash
[root@elk ~]# yum list | grep --color logstash logstash.noarch 1:2.1.3-1 logstash-2.1 [root@elk ~]#
YUM的安装很简单。上面已经找到了对应的包，剩下的事情只需要“yum install”即可。
本文呈现的方法是通过下载软件包一个个的部署ELK。
一、Elasticsearch 2.3.4
Download Link：
https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.4/elasticsearch-2.3.4.tar.gz
[root@elk ~]# cd /software [root@elk software]# ls jdk-8u91-linux-x64.rpm [root@elk software]# [root@elk software]# wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.4/elasticsearch-2.3.4.tar.gz --2016-07-17 19:28:53-- https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.4/elasticsearch-2.3.4.tar.gz Resolving download.elastic.co... 184.73.218.216, 107.22.236.1, 23.21.83.64, ... Connecting to download.elastic.co|184.73.218.216|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 27547169 (26M) [application/gzip] Saving to: “elasticsearch-2.3.4.tar.gz” 100%[====================================================>] 27,547,169 211K/s in 2m 46s 2016-07-17 19:31:42 (162 KB/s) - “elasticsearch-2.3.4.tar.gz” saved [27547169/27547169] [root@elk software]# [root@elk software]# ls -ltr * -rw-r--r-- 1 root root 27547169 Jul 7 06:05 elasticsearch-2.3.4.tar.gz -rw-r--r-- 1 root root 160162581 Jul 11 23:06 jdk-8u91-linux-x64.rpm [root@elk software]# [root@elk software]# du -sh * 27M elasticsearch-2.3.4.tar.gz 153M jdk-8u91-linux-x64.rpm [root@elk software]#
执行安装：
[root@elk software]# tar -xzf elasticsearch-2.3.4.tar.gz [root@elk software]# ls -ltr total 183320 -rw-r--r-- 1 root root 27547169 Jul 7 06:05 elasticsearch-2.3.4.tar.gz -rw-r--r-- 1 root root 160162581 Jul 11 23:06 jdk-8u91-linux-x64.rpm drwxr-xr-x 6 root root 4096 Jul 17 19:39 elasticsearch-2.3.4 [root@elk software]# [root@elk software]# mv elasticsearch-2.3.4 /opt/ [root@elk software]# ls elasticsearch-2.3.4.tar.gz jdk-8u91-linux-x64.rpm [root@elk software]# [root@elk software]# ls -ltr /opt total 8 drwxr-xr-x. 2 root root 4096 Mar 26 2015 rh drwxr-xr-x 6 root root 4096 Jul 17 19:39 elasticsearch-2.3.4 [root@elk software]# [root@elk software]# ls -ltr /opt/elasticsearch-2.3.4/ total 44 -rw-rw-r-- 1 1000 1000 8700 Jun 30 04:22 README.textile -rw-rw-r-- 1 1000 1000 150 Jun 30 04:22 NOTICE.txt -rw-rw-r-- 1 1000 1000 11358 Jun 30 04:22 LICENSE.txt drwxrwxr-x 5 1000 1000 4096 Jun 30 04:31 modules drwxr-xr-x 2 root root 4096 Jul 17 19:39 config drwxr-xr-x 2 root root 4096 Jul 17 19:39 bin drwxr-xr-x 2 root root 4096 Jul 17 19:39 lib [root@elk software]#
这样就安装完成了。
运行Elasticsearch：
[root@elk software]# /opt/elasticsearch-2.3.4/bin/elasticsearch
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
[root@elk software]#可以看到：
在Linux下，不允许使用root执行elasticsearch。
创建用户：
[root@elk software]# useradd elasticme [root@elk software]# echo "abcd1234" | passwd --stdin elasticme Changing password for user elasticme. passwd: all authentication tokens updated successfully. [root@elk software]#
分配权限：
[root@elk software]# id elasticme uid=501(elasticme) gid=501(elasticme) groups=501(elasticme) [root@elk software]# [root@elk software]# chown -R elasticme.elasticme /opt/elasticsearch-2.3.4/ [root@elk software]# ls -ld /opt/elasticsearch-2.3.4/ drwxr-xr-x 7 elasticme elasticme 4096 Jul 17 19:42 /opt/elasticsearch-2.3.4/ [root@elk software]#
再次执行elasticsearch：
[elasticme@elk ~]$ /opt/elasticsearch-2.3.4/bin/elasticsearch
[2016-07-17 19:46:05,820][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-07-17 19:46:06,086][INFO ][node ] [Hurricane] version[2.3.4], pid[25926], build[e455fd0/2016-06-30T11:24:31Z]
[2016-07-17 19:46:06,086][INFO ][node ] [Hurricane] initializing ...
[2016-07-17 19:46:06,720][INFO ][plugins ] [Hurricane] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-07-17 19:46:06,747][INFO ][env ] [Hurricane] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [21.6gb], net total_space [26.1gb], spins? [possibly], types [ext4]
[2016-07-17 19:46:06,747][INFO ][env ] [Hurricane] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-07-17 19:46:06,747][WARN ][env ] [Hurricane] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-07-17 19:46:09,194][INFO ][node ] [Hurricane] initialized
[2016-07-17 19:46:09,194][INFO ][node ] [Hurricane] starting ...
[2016-07-17 19:46:09,292][INFO ][transport ] [Hurricane] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2016-07-17 19:46:09,300][INFO ][discovery ] [Hurricane] elasticsearch/YQvlEybHS5uLkiBsCpZKtQ
[2016-07-17 19:46:12,412][INFO ][cluster.service ] [Hurricane] new_master {Hurricane}{YQvlEybHS5uLkiBsCpZKtQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-07-17 19:46:12,441][INFO ][http ] [Hurricane] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2016-07-17 19:46:12,441][INFO ][node ] [Hurricane] started
[2016-07-17 19:46:12,483][INFO ][gateway ] [Hurricane] recovered [0] indices into cluster_state通过上面的运行日志，你可以看到，elasticsearch监听了：9200。
[root@elk software]# netstat -tupln | grep --color 9200 tcp 0 0 ::ffff:127.0.0.1:9200 :::* LISTEN 25926/java tcp 0 0 ::1:9200 :::* LISTEN 25926/java [root@elk software]#
访问elasticsearch的WEB应用：
http://localhost:9200/
Shell：
[root@elk software]# curl http://localhost:9200/
{
"name" : "Hurricane",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.3.4",
"build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
"build_timestamp" : "2016-06-30T11:24:31Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
[root@elk software]#
http://localhost:9200/_cat/health?v
Shell：
[root@elk software]# curl http://localhost:9200/_cat/health?v epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1468810439 19:53:59 elasticsearch green 1 1 0 0 0 0 0 0 - 100.0% [root@elk software]#
http://localhost:9200/_cat/nodes?v
[root@elk software]# curl http://localhost:9200/_cat/nodes?v host ip heap.percent ram.percent load node.role master name 127.0.0.1 127.0.0.1 5 60 0.06 d * Hurricane [root@elk software]#
http://localhost:9200/_cat/allocation?v
[root@elk software]# curl http://localhost:9200/_cat/allocation?v
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 4.5gb 21.5gb 26.1gb 17 127.0.0.1 127.0.0.1 Hurricane
[root@elk software]#这样，Elasticsearch就安装完成了。
二、Kibana 4.5.3
Download Link：
https://download.elastic.co/kibana/kibana/kibana-4.5.3-1.x86_64.rpm
[root@elk software]# wget https://download.elastic.co/kibana/kibana/kibana-4.5.3-1.x86_64.rpm --2016-07-17 19:58:58-- https://download.elastic.co/kibana/kibana/kibana-4.5.3-1.x86_64.rpm Resolving download.elastic.co... 23.23.240.27, 184.73.171.200, 184.73.218.216, ... Connecting to download.elastic.co|23.23.240.27|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 33920908 (32M) [application/x-rpm] Saving to: “kibana-4.5.3-1.x86_64.rpm” 100%[=======================================================================================================================================>] 33,920,908 167K/s in 3m 0s 2016-07-17 20:01:59 (184 KB/s) - “kibana-4.5.3-1.x86_64.rpm” saved [33920908/33920908] [root@elk software]# [root@elk software]# ls -ltr * -rw-r--r-- 1 root root 27547169 Jul 7 06:05 elasticsearch-2.3.4.tar.gz -rw-r--r-- 1 root root 160162581 Jul 11 23:06 jdk-8u91-linux-x64.rpm -rw-r--r-- 1 root root 33920908 Jul 14 08:02 kibana-4.5.3-1.x86_64.rpm [root@elk software]# [root@elk software]# du -sh * 27M elasticsearch-2.3.4.tar.gz 153M jdk-8u91-linux-x64.rpm 33M kibana-4.5.3-1.x86_64.rpm [root@elk software]#
执行安装：
[root@elk software]# rpm -ivh kibana-4.5.3-1.x86_64.rpm Preparing... ########################################### [100%] 1:kibana ########################################### [100%] [root@elk software]# [root@elk software]# rpm -qa | grep kibana kibana-4.5.3-1.x86_64 [root@elk software]# [root@elk software]# rpm -ql kibana-4.5.3-1.x86_64 | head -n 16 /etc/default/kibana /etc/init.d/kibana /lib/systemd/system/kibana.service /opt/kibana/LICENSE.txt /opt/kibana/README.txt /opt/kibana/bin/kibana /opt/kibana/bin/kibana.bat /opt/kibana/config/kibana.yml /opt/kibana/installedPlugins /opt/kibana/node/CHANGELOG.md /opt/kibana/node/LICENSE /opt/kibana/node/README.md /opt/kibana/node/bin/node /opt/kibana/node/bin/npm /opt/kibana/node/include/node/android-ifaddrs.h /opt/kibana/node/include/node/ares.h [root@elk software]# [root@elk software]# ls -ltr /opt total 12 drwxr-xr-x. 2 root root 4096 Mar 26 2015 rh drwxr-xr-x 9 elasticme elasticme 4096 Jul 17 19:46 elasticsearch-2.3.4 drwxr-xr-x 10 root root 4096 Jul 17 20:02 kibana [root@elk software]# [root@elk software]# chown -R elasticme.elasticme /opt/kibana/ [root@elk software]# [root@elk software]# ls -ld /opt/kibana/ drwxr-xr-x 10 elasticme elasticme 4096 Jul 17 20:02 /opt/kibana/ [root@elk software]# [root@elk software]# ls -ltr /opt/kibana/ total 44 -rw-rw-r-- 1 elasticme elasticme 2462 Jul 13 17:03 README.txt -rw-rw-r-- 1 elasticme elasticme 563 Jul 13 17:03 LICENSE.txt drwxrwxr-x 2 elasticme elasticme 4096 Jul 13 17:03 installedPlugins -rw-rw-r-- 1 elasticme elasticme 700 Jul 13 17:03 package.json drwxr-xr-x 3 elasticme elasticme 4096 Jul 17 20:02 optimize drwxr-xr-x 82 elasticme elasticme 4096 Jul 17 20:02 node_modules drwxr-xr-x 8 elasticme elasticme 4096 Jul 17 20:02 src drwxr-xr-x 2 elasticme elasticme 4096 Jul 17 20:02 config drwxr-xr-x 2 elasticme elasticme 4096 Jul 17 20:02 bin drwxr-xr-x 6 elasticme elasticme 4096 Jul 17 20:02 node drwxr-xr-x 2 elasticme elasticme 4096 Jul 17 20:02 webpackShims [root@elk software]#
配置Kinbana：
[root@elk software]# cd /opt/kibana/config/ [root@elk config]# ls -ltr * -rw-rw-r-- 1 elasticme elasticme 3040 Jul 13 17:03 kibana.yml [root@elk config]# [root@elk config]# cat kibana.yml | grep -v ^# | strings [root@elk config]# [root@elk config]# vi kibana.yml [root@elk config]# cat kibana.yml | grep -v ^# | strings elasticsearch.url: "http://localhost:9200" [root@elk config]#
启动Kinbana：
[root@elk ~]# su - elasticme [elasticme@elk ~]$ /opt/kibana/bin/kibana log [20:09:01.478] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready log [20:09:01.514] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [20:09:01.529] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready log [20:09:01.537] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready log [20:09:01.543] [info][status][plugin:metric_vis] Status changed from uninitialized to green - Ready log [20:09:01.555] [info][status][plugin:spyModes] Status changed from uninitialized to green - Ready log [20:09:01.568] [info][status][plugin:statusPage] Status changed from uninitialized to green - Ready log [20:09:01.574] [info][status][plugin:table_vis] Status changed from uninitialized to green - Ready log [20:09:01.582] [info][listening] Server running at http://0.0.0.0:5601 log [20:09:06.583] [info][status][plugin:elasticsearch] Status changed from yellow to yellow - No existing Kibana index found log [20:09:09.911] [info][status][plugin:elasticsearch] Status changed from yellow to green - Kibana index ready
可以看到，Kinbana开启了对端口的监听：5601。
[root@elk bin]# netstat -tupln | grep 5601 tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 26259/node [root@elk bin]#
访问Kinba的WEB应用：
http://localhost:5601/
http://localhost:5601/status
三、Logstash 2.3.4
Download Link：
https://download.elastic.co/logstash/logstash/logstash-2.3.4.zip
https://download.elastic.co/logstash/logstash/packages/centos/logstash-5.0.0-alpha4.rpm
下载并安装：
[root@elk software]# wget https://download.elastic.co/logstash/logstash/logstash-2.3.4.zip --2016-07-17 21:24:03-- https://download.elastic.co/logstash/logstash/logstash-2.3.4.zip Resolving download.elastic.co... 184.73.171.200, 23.23.240.27, 184.73.218.216, ... Connecting to download.elastic.co|184.73.171.200|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 85757162 (82M) [application/zip] Saving to: “logstash-2.3.4.zip” 100%[=======================================================================>] 85,757,162 175K/s in 10m 19s 2016-07-17 21:34:24 (135 KB/s) - “logstash-2.3.4.zip” saved [85757162/85757162] [root@elk software]# [root@elk software]# ls -ltr * -rw-r--r-- 1 root root 89820815 Jun 28 13:36 logstash-5.0.0-alpha4.rpm -rw-r--r-- 1 root root 85757162 Jul 6 17:35 logstash-2.3.4.zip -rw-r--r-- 1 root root 27547169 Jul 7 06:05 elasticsearch-2.3.4.tar.gz -rw-r--r-- 1 root root 160162581 Jul 11 23:06 jdk-8u91-linux-x64.rpm -rw-r--r-- 1 root root 33920908 Jul 14 08:02 kibana-4.5.3-1.x86_64.rpm [root@elk software]# [root@elk software]# du -sh * 27M elasticsearch-2.3.4.tar.gz 153M jdk-8u91-linux-x64.rpm 33M kibana-4.5.3-1.x86_64.rpm 82M logstash-2.3.4.zip 86M logstash-5.0.0-alpha4.rpm [root@elk software]# [root@elk software]# unzip logstash-2.3.4.zip ... Much more output. inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/uri_formatter.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/user_interaction.rb creating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/util/ inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/util.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/util/list.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/util/stringio.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/validator.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/version.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/rubygems/version_option.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/securerandom.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/syslog.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/tempfile.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/tmpdir.rb inflating: logstash-2.3.4/vendor/jruby/lib/ruby/shared/ubygems.rb creating: logstash-2.3.4/vendor/jruby/tool/ creating: logstash-2.3.4/vendor/jruby/tool/nailgun/ inflating: logstash-2.3.4/vendor/jruby/tool/nailgun/Makefile.in inflating: logstash-2.3.4/vendor/jruby/tool/nailgun/README.txt inflating: logstash-2.3.4/vendor/jruby/tool/nailgun/configure inflating: logstash-2.3.4/vendor/jruby/tool/nailgun/ng.exe creating: logstash-2.3.4/vendor/jruby/tool/nailgun/src/ creating: logstash-2.3.4/vendor/jruby/tool/nailgun/src/c/ inflating: logstash-2.3.4/vendor/jruby/tool/nailgun/src/c/ng.c inflating: logstash-2.3.4/vendor/bundle/jruby/1.9/gems/jrjackson-0.3.9-java/.mvn/extensions.xml inflating: logstash-2.3.4/vendor/bundle/jruby/1.9/gems/ruby-maven-3.3.12/.mvn/extensions.xml inflating: logstash-2.3.4/Gemfile inflating: logstash-2.3.4/Gemfile.jruby-1.9.lock [root@elk software]# [root@elk software]# ls -ltr total 387912 -rw-r--r-- 1 root root 89820815 Jun 28 13:36 logstash-5.0.0-alpha4.rpm -rw-r--r-- 1 root root 85757162 Jul 6 17:35 logstash-2.3.4.zip -rw-r--r-- 1 root root 27547169 Jul 7 06:05 elasticsearch-2.3.4.tar.gz -rw-r--r-- 1 root root 160162581 Jul 11 23:06 jdk-8u91-linux-x64.rpm -rw-r--r-- 1 root root 33920908 Jul 14 08:02 kibana-4.5.3-1.x86_64.rpm drwxr-xr-x 5 root root 4096 Jul 17 21:48 logstash-2.3.4 [root@elk software]# [root@elk software]# mv logstash-2.3.4 /opt/ [root@elk software]# [root@elk software]# ls -ltr total 387908 -rw-r--r-- 1 root root 89820815 Jun 28 13:36 logstash-5.0.0-alpha4.rpm -rw-r--r-- 1 root root 85757162 Jul 6 17:35 logstash-2.3.4.zip -rw-r--r-- 1 root root 27547169 Jul 7 06:05 elasticsearch-2.3.4.tar.gz -rw-r--r-- 1 root root 160162581 Jul 11 23:06 jdk-8u91-linux-x64.rpm -rw-r--r-- 1 root root 33920908 Jul 14 08:02 kibana-4.5.3-1.x86_64.rpm [root@elk software]# [root@elk software]# ls -ltr /opt total 16 drwxr-xr-x. 2 root root 4096 Mar 26 2015 rh drwxr-xr-x 9 elasticme elasticme 4096 Jul 17 19:46 elasticsearch-2.3.4 drwxr-xr-x 10 elasticme elasticme 4096 Jul 17 20:02 kibana drwxr-xr-x 5 root root 4096 Jul 17 21:48 logstash-2.3.4 [root@elk software]# [root@elk software]# chown -R elasticme.elasticme /opt/logstash-2.3.4/ [root@elk software]# [root@elk software]# ls -ltr /opt total 16 drwxr-xr-x. 2 root root 4096 Mar 26 2015 rh drwxr-xr-x 9 elasticme elasticme 4096 Jul 17 19:46 elasticsearch-2.3.4 drwxr-xr-x 10 elasticme elasticme 4096 Jul 17 20:02 kibana drwxr-xr-x 5 elasticme elasticme 4096 Jul 17 21:48 logstash-2.3.4 [root@elk software]# [root@elk software]# ls -ltr /opt/logstash-2.3.4/ total 156 -rw-rw-r-- 1 elasticme elasticme 149 Jul 7 02:02 NOTICE.TXT -rw-rw-r-- 1 elasticme elasticme 589 Jul 7 02:02 LICENSE -rw-rw-r-- 1 elasticme elasticme 2249 Jul 7 02:02 CONTRIBUTORS -rw-rw-r-- 1 elasticme elasticme 104936 Jul 7 02:02 CHANGELOG.md -rw-rw-r-- 1 elasticme elasticme 21773 Jul 7 02:02 Gemfile.jruby-1.9.lock -rw-rw-r-- 1 elasticme elasticme 3885 Jul 7 02:02 Gemfile drwxr-xr-x 2 elasticme elasticme 4096 Jul 17 21:48 bin drwxr-xr-x 4 elasticme elasticme 4096 Jul 17 21:48 lib drwxr-xr-x 4 elasticme elasticme 4096 Jul 17 21:48 vendor [root@elk software]#
测试：
[root@elk software]# /opt/logstash-2.3.4/bin/logstash -e 'input { stdin { } } output { stdout {} }'
Settings: Default pipeline workers: 1
Pipeline main started
Hello world - adamhuan
2016-07-18T05:16:16.479Z elk Hello world - adamhuan在这个界面，你输入“Hello world”，便会得到格式化的输出。
想要结束：Ctrl + C。
[root@elk software]# /opt/logstash-2.3.4/bin/logstash -e 'input { stdin { } } output { stdout {} }'
Settings: Default pipeline workers: 1
Pipeline main started
Hello world - adamhuan
2016-07-18T05:16:16.479Z elk Hello world - adamhuan
2016-07-18T05:17:17.611Z elk
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
Pipeline main has been shutdown
[root@elk software]#
编写Logstash配置文件：
[root@elk script]# pwd
/script
[root@elk script]# ls -ltr
total 4
-rw-r--r-- 1 root root 63 Jul 17 22:20 logstash-sample.conf
[root@elk script]#
[root@elk script]# cat logstash-sample.conf
input { stdin { } }
output {
stdout { codec=> rubydebug }
}
[root@elk script]#
以上面的配置文件，做启动logstash之前的检查：
[root@elk script]# time /opt/logstash-2.3.4/bin/logstash -f logstash-sample.conf --configtest Configuration OK real 0m11.857s user 0m11.299s sys 0m0.422s [root@elk script]#
启动logstash：
[root@elk script]# /opt/logstash-2.3.4/bin/logstash -f logstash-sample.conf
Settings: Default pipeline workers: 1
Pipeline main started
This is Alice.
{
"message" => "This is Alice.",
"@version" => "1",
"@timestamp" => "2016-07-18T05:24:24.783Z",
"host" => "elk"
}
Hello World.
{
"message" => "Hello World.",
"@version" => "1",
"@timestamp" => "2016-07-18T05:24:46.040Z",
"host" => "elk"
}上面的：
This is Alice.
Hello World.
是自己手动敲入的。
和之前的差别是，这次使用了配置文件，于是输出信息被格式化了。
此外：
启动logstash，也可以用以下命令：
logstash agent -f logstash-es-simple.conf
退出：Ctrl + C。
[root@elk script]# /opt/logstash-2.3.4/bin/logstash -f logstash-sample.conf
Settings: Default pipeline workers: 1
Pipeline main started
This is Alice.
{
"message" => "This is Alice.",
"@version" => "1",
"@timestamp" => "2016-07-18T05:24:24.783Z",
"host" => "elk"
}
Hello World.
{
"message" => "Hello World.",
"@version" => "1",
"@timestamp" => "2016-07-18T05:24:46.040Z",
"host" => "elk"
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss. {:level=>:warn}
Pipeline main has been shutdown
[root@elk script]#
将Logstash的数据，存储到Elasticsearch中。
修改配置文件：
[root@elk script]# cat logstash-sample.conf
input { stdin { } }
output {
elasticsearch { hosts=>"localhost:9200" }
stdout { codec=> rubydebug }
}
[root@elk script]#注意：
使用 hosts 来指定ES的位置，老版使用的是 host ，如果在这里使用 host 会报错
可以使用 hosts => [“IP Address 1:port1”, “IP Address 2:port2”, “IP Address 3”] 的方式指定多个进行冗余，和负载均衡
如果ES使用的 9200 端口，是可以在配置里省略的
确定elasticsearch的运行状态：
[root@elk script]# curl http://localhost:9200
{
"name" : "Hurricane",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.3.4",
"build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
"build_timestamp" : "2016-06-30T11:24:31Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
[root@elk script]#
以修改的配置文件，启动logstash：
[root@elk script]# /opt/logstash-2.3.4/bin/logstash -f logstash-sample.conf
Settings: Default pipeline workers: 1
Pipeline main started
a. this is for testing.
{
"message" => "a. this is for testing.",
"@version" => "1",
"@timestamp" => "2016-07-18T05:30:12.962Z",
"host" => "elk"
}
b. let's started it.
{
"message" => "b. let's started it.",
"@version" => "1",
"@timestamp" => "2016-07-18T05:30:37.121Z",
"host" => "elk"
}
c. store data in elasticsearch
{
"message" => "c. store data in elasticsearch",
"@version" => "1",
"@timestamp" => "2016-07-18T05:30:53.753Z",
"host" => "elk"
}启动后，做了一些录入。
下面看看elasticsearch的状态：
[root@elk conf.d]# curl http://localhost:9200/_search?pretty
{
"took" : 13,
"timed_out" : false,
"_shards" : {
"total" : 6,
"successful" : 6,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : 1.0,
"hits" : [ {
"_index" : ".kibana",
"_type" : "config",
"_id" : "4.5.3",
"_score" : 1.0,
"_source" : {
"buildNum" : 9910
}
}, {
"_index" : "logstash-2016.07.18",
"_type" : "logs",
"_id" : "AVX8e7b8utPYJ8HnaL1U",
"_score" : 1.0,
"_source" : {
"message" : "a. this is for testing.",
"@version" : "1",
"@timestamp" : "2016-07-18T05:30:12.962Z",
"host" : "elk"
}
}, {
"_index" : "logstash-2016.07.18",
"_type" : "logs",
"_id" : "AVX8fBPAutPYJ8HnaL1V",
"_score" : 1.0,
"_source" : {
"message" : "b. let's started it.",
"@version" : "1",
"@timestamp" : "2016-07-18T05:30:37.121Z",
"host" : "elk"
}
}, {
"_index" : "logstash-2016.07.18",
"_type" : "logs",
"_id" : "AVX8fFTFutPYJ8HnaL1W",
"_score" : 1.0,
"_source" : {
"message" : "c. store data in elasticsearch",
"@version" : "1",
"@timestamp" : "2016-07-18T05:30:53.753Z",
"host" : "elk"
}
} ]
}
}
[root@elk conf.d]#可以看到elasticsearch中，已经有数据了。
Kibana配置：
如上，可以通过Kibana看到logstash捕捉到的数据了。
——————————————————————
Logstash：从文件中获取数据。
修改配置文件，并启动logstash：
[root@elk script]# cat logstash-sample.conf
input {
stdin {}
file {
type=>"syslog"
path=>"/var/log/messages"
start_position=>beginning
}
}
output {
elasticsearch { hosts=>"localhost:9200" }
stdout { codec=>rubydebug }
}
[root@elk script]#
[root@elk script]# /opt/logstash-2.3.4/bin/logstash -f logstash-sample.conf
Settings: Default pipeline workers: 1
Pipeline main started
（... ... 过多的输出。）
{
"message" => "Jul 14 10:44:50 elk tpvmlpd2[2459]: device type not supported",
"@version" => "1",
"@timestamp" => "2016-07-18T05:43:58.557Z",
"path" => "/var/log/messages",
"host" => "elk",
"type" => "syslog"
}
{
"message" => "Jul 14 10:44:53 elk seahorse-daemon[2506]: DNS-SD initialization failed: Daemon not running",
"@version" => "1",
"@timestamp" => "2016-07-18T05:43:58.557Z",
"path" => "/var/log/messages",
"host" => "elk",
"type" => "syslog"
}
{
"message" => "Jul 14 10:44:53 elk seahorse-daemon[2506]: init gpgme version 1.1.8",
"@version" => "1",
"@timestamp" => "2016-07-18T05:43:58.557Z",
"path" => "/var/log/messages",
"host" => "elk",
"type" => "syslog"
}如上，全部加载完“/var/log/messages”之后，logstash会等待，当messages文件有变化，会自动的被logstash加载。
再次查看kibana：
尝试给/var/log/messages中插入一些记录：
[root@elk conf.d]# tail -n 1 /var/log/messages Jul 17 22:50:13 elk dhclient[27019]: bound to 192.168.232.151 -- renewal in 814 seconds. [root@elk conf.d]# [root@elk conf.d]# echo "Hey, Kibana." >> /var/log/messages [root@elk conf.d]# [root@elk conf.d]# tail -n 1 /var/log/messages Hey, Kibana. [root@elk conf.d]# [root@elk conf.d]# echo "Now you see me." >> /var/log/messages [root@elk conf.d]# [root@elk conf.d]# tail -n 1 /var/log/messages Now you see me. [root@elk conf.d]#
然后，查看下Kibana的变化：
至此，关于ELK架构的初步尝试已经完成。
——————————————————————
参考文档：
http://soft.dog/2015/12/22/elk-basic/
http://baidu.blog.51cto.com/71938/1676798
——————————————————————
Done。