Logstash使用介紹
一 應(yīng)用場(chǎng)景描述
創(chuàng)新互聯(lián)建站主營(yíng)白山網(wǎng)站建設(shè)的網(wǎng)絡(luò)公司,主營(yíng)網(wǎng)站建設(shè)方案,app開發(fā)定制,白山h5小程序設(shè)計(jì)搭建,白山網(wǎng)站營(yíng)銷推廣歡迎白山等地區(qū)企業(yè)咨詢
最近在研究日志平臺(tái)解決方案。最終選擇使用目前比較流行的ELK框架，即Elasticsearch，Logstash，Kibana三個(gè)開源軟件的組合來構(gòu)建日志平臺(tái)。其中Elasticsearch用于日志搜索，Logstash用于日志的收集，過濾，處理等，Kibana用于日志的界面展示。最核心的就是要先了解Logstash的工作原理。
二 Logstash介紹
Logstash是一款用于接收，處理并輸出日志的工具。Logstash可以處理各種各樣的日志，包括系統(tǒng)日志，WEB容器日志如Apache日志和Nginx日志和Tomcat日志等，各種應(yīng)用日志等。
三 Logstash簡(jiǎn)單使用
Logstash是用ruby語(yǔ)言編寫，Jruby作為ruby解釋器。所以運(yùn)行Logstash只需要安裝Java就行。
在CentOS上安裝Java
yum -y install java-1.7.0-openjdk*
$ java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-2.5.4.0.el6_6-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar zxvf logstash-1.4.2.tar.gz
cd logstash-1.4.2
使用bin/logstash agent --help 查看參數(shù)說明
-e 后面直接跟配置信息，而不通過-f 參數(shù)指定配置文件?？梢杂糜诳焖贉y(cè)試
在命令行運(yùn)行
$ bin/logstash -e 'input {stdin {} } output {stdout {} }'
然后再輸入一些信息
$ bin/logstash -e 'input {stdin {} } output {stdout {} }'
hello world
2015-01-31T12:02:20.438+0000 xxxxx hello world
這里通過stdin輸入信息，然后通過stdout輸出信息。在輸入hello world后Logstash將處理后的信息輸出到屏幕
$ bin/logstash -e 'input {stdin {} } output {stdout { codec => rubydebug } }'
goodnight moon
{
"message" => "goodnight moon",
"@version" => "1",
"@timestamp" => "2015-01-31T12:09:38.564Z",
"host" => "xxxx-elk-log"
}存儲(chǔ)日志到Elasticsearch
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.zip
unzip elasticsearch-1.4.2.zip
cd elasticsearch-1.4.2
./bin/elasticsearch
Logstash和Elasticsearch的版本要一致
$bin/logstash -e 'input { stdin {} } output { elasticsearch { host => localhost }}'
you know,for logs這里logstash從屏幕接收信息，然后將輸出結(jié)果發(fā)送到Elasticsearch，然后驗(yàn)證Elasticsearch是否從Logstash接收了數(shù)據(jù)
$ curl 'http://localhost:9200/_search?pretty'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logstash-2015.01.31",
"_type" : "logs",
"_id" : "W6HMXGx2Tw25sTX7OwZPug",
"_score" : 1.0,
"_source":{"message":"you know,for logs","@version":"1","@timestamp":"2015-01-31T12:43:53.630Z","host":"jidong-elk-log"}
} ]
}
}另外可以通過Elasticearch-kopf插件訪問查看Logstash數(shù)據(jù)
使用一下方式安裝
bin/plugin -install lmenezes/elasticsearch-kopf
然后通過
http://localhost:9200/_plugin/kopf 訪問
使用多種輸出方式
$bin/logstash -e 'input { stdin {} } output { elasticsearch { host => localhost } stdout {} }'
multiple outputs
2015-01-31T13:03:43.426+0000 jidong-elk-log multiple outputs這里除了將從鍵盤輸入的內(nèi)容輸出到Elasticsearch外，還輸出到屏幕
$ curl 'http://localhost:9200/_search?pretty'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logstash-2015.01.31",
"_type" : "logs",
"_id" : "W6HMXGx2Tw25sTX7OwZPug",
"_score" : 1.0,
"_source":{"message":"you know,for logs","@version":"1","@timestamp":"2015-01-31T12:43:53.630Z","host":"jidong-elk-log"}
}, {
"_index" : "logstash-2015.01.31",
"_type" : "logs",
"_id" : "kMXKoQglQNCDYOEyOmAnhg",
"_score" : 1.0,
"_source":{"message":"multiple outputs","@version":"1","@timestamp":"2015-01-31T13:03:43.426Z","host":"jidong-elk-log"}
} ]
}
}Elasticsearch默認(rèn)是根據(jù)日期來創(chuàng)建索引，每天創(chuàng)建一個(gè)索引，如logstash-2015.01.31
Logstash事件的生命周期 The life of an event
Inputs,Outputs,Codecs,Filters是Logstash配置的核心。
Inputs 傳送日志數(shù)據(jù)到Logstash，主要有以下幾個(gè)插件可以使用
file 從一個(gè)文件中讀入日志數(shù)據(jù)
syslog 默認(rèn)監(jiān)聽514端口，接收來自syslog的日志，并根據(jù)RFC3164格式解析
redis 從redis讀入日志數(shù)據(jù)，通常redis在一個(gè)集中Logstash部署架構(gòu)中作為一個(gè)broker來緩沖來自Logstash agent或其他方式發(fā)送過來的日志。
lumberjack 處理使用lumberjack協(xié)議發(fā)送過來的日志?，F(xiàn)在叫做logstash-forwarder
Filters 用于根據(jù)各種匹配條件對(duì)Logstash事件進(jìn)行過濾處理，主要有以下幾個(gè)插件
grok 解析任意文本并將它結(jié)構(gòu)化
mutate 對(duì)事件進(jìn)行添加，刪除，移動(dòng)，替換，修改等更改操作
drop 丟掉特定事件
clone 克隆事件
geoip 添加IP地址的物理位置信息
Outputs 是Logstash pipeline的最后一個(gè)階段。一個(gè)事件可以有多種輸出。常用的有以下幾個(gè)插件
elasticsearch 將事件數(shù)據(jù)寫入到Elasticsearch
file 將事件數(shù)據(jù)寫入到磁盤文件
Codecs 是用于流過濾，可以添加到input或output。主要有plain,json等
使用配置文件
conf/logstash-simple.conf
input {
stdin {}
}
output {
elasticsearch {
host => localhost
}
stdout {
codec => rubydebug
}
}$ sudo bin/logstash -f conf/logstash-simple.conf
config file
{
"message" => "config file",
"@version" => "1",
"@timestamp" => "2015-02-01T02:38:15.347Z",
"host" => "xxxxxx"
}curl 'http://localhost:9200/_search?pretty'
"_index" : "logstash-2015.02.01",
"_type" : "logs",
"_id" : "NW2e8LdWSwuNE-aJZNtd-w",
"_score" : 1.0,
"_source":{"message":"config file","@version":"1","@timestamp":"2015-02-01T02:38:15.347Z","host":"xxxxxx"}
} ]
}Filter測(cè)試
$ cat conf/logstash-filter.conf
input {
stdin {}
}
filter {
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
date {
match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
host => localhost
}
stdout {
codec => rubydebug
}
}在屏幕輸入一行Apache日志
$ sudo bin/logstash -f conf/logstash-filter.conf
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
{
"message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
"@version" => "1",
"@timestamp" => "2013-12-11T08:01:45.000Z",
"host" => "xxxxxxx",
"clientip" => "127.0.0.1",
"ident" => "-",
"auth" => "-",
"timestamp" => "11/Dec/2013:00:01:45 -0800",
"verb" => "GET",
"request" => "/xampp/status.php",
"httpversion" => "1.1",
"response" => "200",
"bytes" => "3891",
"referrer" => "\"http://cadenza/xampp/navi.php\"",
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\""
} "_index" : "logstash-2013.12.11",
"_type" : "logs",
"_id" : "QusW5lY5T8a9wqgCcottnA",
"_score" : 1.0,
"_source":{"message":"127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\
" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gec
ko/20100101 Firefox/25.0\"","@version":"1","@timestamp":"2013-12-11T08:01:45.000Z","host":"jidong-elk-lo
g","clientip":"127.0.0.1","ident":"-","auth":"-","timestamp":"11/Dec/2013:00:01:45 -0800","verb":"GET","
request":"/xampp/status.php","httpversion":"1.1","response":"200","bytes":"3891","referrer":"\"http://ca
denza/xampp/navi.php\"","agent":"\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101
Firefox/25.0\""}案例一，使用Logstash處理Apache日志
$ cat /tmp/access.log 71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" 134.39.72.245 - - [18/May/2011:12:40:18 -0700] "GET /favicon.ico HTTP/1.1" 200 1189 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)" 98.83.179.51 - - [18/May/2011:19:35:08 -0700] "GET /css/main.css HTTP/1.1" 200 1837 "http://www.safesand.com/information.htm" "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
$ cat conf/logstash-apache.conf
input {
file {
path => "/tmp/access_log"
start_position => beginning
}
}
filter {
if [path] =~"access" {
mutate {
replace => {
"type" => "apache_access"
}
}
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}啟動(dòng)Logstash后可以看到Logstash將/tmp/access_log的日志數(shù)據(jù)處理了
$ sudo bin/logstash -f conf/logstash-apache.conf
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
{
"message" => "71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] \"GET /admin HTTP/1.1\" 301 566 \"-\" \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"",
"@version" => "1",
"@timestamp" => "2011-05-18T08:48:10.000Z",
"host" => "jidong-elk-log",
"path" => "/tmp/access_log",
"type" => "apache_access",
"clientip" => "71.141.244.242",
"ident" => "-",
"auth" => "kurt",
"timestamp" => "18/May/2011:01:48:10 -0700",
"verb" => "GET",
"request" => "/admin",
"httpversion" => "1.1",
"response" => "301",
"bytes" => "566",
"referrer" => "\"-\"",
"agent" => "\"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\""
}
{
"message" => "134.39.72.245 - - [18/May/2011:12:40:18 -0700] \"GET /favicon.ico HTTP/1.1\" 200 1189 \"-\" \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)\"",
"@version" => "1",
"@timestamp" => "2011-05-18T19:40:18.000Z",
"host" => "jidong-elk-log",
"path" => "/tmp/access_log",
"type" => "apache_access",
"clientip" => "134.39.72.245",
"ident" => "-",
"auth" => "-",
"timestamp" => "18/May/2011:12:40:18 -0700",
"verb" => "GET",
"request" => "/favicon.ico",
"httpversion" => "1.1",
"response" => "200",
"bytes" => "1189",
"referrer" => "\"-\"",
"agent" => "\"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)\""
}
{
"message" => "98.83.179.51 - - [18/May/2011:19:35:08 -0700] \"GET /css/main.css HTTP/1.1\" 200 1837 \"http://www.safesand.com/information.htm\" \"Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1\"",
"@version" => "1",
"@timestamp" => "2011-05-19T02:35:08.000Z",
"host" => "jidong-elk-log",
"path" => "/tmp/access_log",
"type" => "apache_access",
"clientip" => "98.83.179.51",
"ident" => "-",
"auth" => "-",
"timestamp" => "18/May/2011:19:35:08 -0700",
"verb" => "GET",
"request" => "/css/main.css",
"httpversion" => "1.1",
"response" => "200",
"bytes" => "1837",
"referrer" => "\"http://www.safesand.com/information.htm\"",
"agent" => "\"Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1\""
}查看Elasticsearch
curl 'http://localhost:9200/_search?pretty'
案例二，使用Logstash處理來自syslog的日志
$ cat conf/logstash-syslog.conf
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"}
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "receieved_from", "%{host}" ]
}
syslog_pri {}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss","MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug}
}啟動(dòng)logstash
$ sudo bin/logstash -f conf/logstash-syslog.conf
通過telnet連接到5000端口，然后發(fā)送日志信息給Logstash
$ telnet localhost 5000 Trying ::1... Connected to localhost. Escape character is '^]'. Dec 23 12:11:43 louis postfix/smtpd[31499]: connect from unknown[95.75.93.154] Dec 23 14:42:56 louis named[16000]: client 199.48.164.7#64817: query (cache) 'amsterdamboothuren.com/MX/IN' denied Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log) Dec 22 18:28:06 louis rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="2253" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
查看Logstash屏幕輸出
$ sudo bin/logstash -f conf/logstash-syslog.conf
{
"message" => "Dec 23 12:11:43 louis postfix/smtpd[31499]: connect from unknown[95.75.93.154]\r",
"@version" => "1",
"@timestamp" => "2015-12-23T04:11:43.000Z",
"host" => "0:0:0:0:0:0:0:1:34337",
"type" => "syslog",
"syslog_timestamp" => "Dec 23 12:11:43",
"syslog_hostname" => "louis",
"syslog_program" => "postfix/smtpd",
"syslog_pid" => "31499",
"syslog_message" => "connect from unknown[95.75.93.154]\r",
"received_at" => "2015-02-01 05:01:48 UTC",
"receieved_from" => "0:0:0:0:0:0:0:1:34337",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}
{
"message" => "Dec 23 14:42:56 louis named[16000]: client 199.48.164.7#64817: query (cache) 'amsterdamboothuren.com/MX/IN' denied\r",
"@version" => "1",
"@timestamp" => "2015-12-23T06:42:56.000Z",
"host" => "0:0:0:0:0:0:0:1:34337",
"type" => "syslog",
"syslog_timestamp" => "Dec 23 14:42:56",
"syslog_hostname" => "louis",
"syslog_program" => "named",
"syslog_pid" => "16000",
"syslog_message" => "client 199.48.164.7#64817: query (cache) 'amsterdamboothuren.com/MX/IN' denied\r",
"received_at" => "2015-02-01 05:01:48 UTC",
"receieved_from" => "0:0:0:0:0:0:0:1:34337",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}
{
"message" => "Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)\r",
"@version" => "1",
"@timestamp" => "2015-12-23T06:30:01.000Z",
"host" => "0:0:0:0:0:0:0:1:34337",
"type" => "syslog",
"syslog_timestamp" => "Dec 23 14:30:01",
"syslog_hostname" => "louis",
"syslog_program" => "CRON",
"syslog_pid" => "619",
"syslog_message" => "(www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)\r",
"received_at" => "2015-02-01 05:01:48 UTC",
"receieved_from" => "0:0:0:0:0:0:0:1:34337",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}
{
"message" => "Dec 22 18:28:06 louis rsyslogd: [origin software=\"rsyslogd\" swVersion=\"4.2.0\" x-pid=\"2253\" x-info=\"http://www.rsyslog.com\"] rsyslogd was HUPed, type 'lightweight'.\r",
"@version" => "1",
"@timestamp" => "2015-12-22T10:28:06.000Z",
"host" => "0:0:0:0:0:0:0:1:34337",
"type" => "syslog",
"syslog_timestamp" => "Dec 22 18:28:06",
"syslog_hostname" => "louis",
"syslog_program" => "rsyslogd",
"syslog_message" => "[origin software=\"rsyslogd\" swVersion=\"4.2.0\" x-pid=\"2253\" x-info=\"http://www.rsyslog.com\"] rsyslogd was HUPed, type 'lightweight'.\r",
"received_at" => "2015-02-01 05:01:53 UTC",
"receieved_from" => "0:0:0:0:0:0:0:1:34337",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}參考文檔
http://logstash.net/docs/1.4.2/
http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash
當(dāng)前文章：Logstash使用介紹
網(wǎng)站地址：http://www.muchs.cn/article48/jpccep.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供Google、、商城網(wǎng)站、App開發(fā)、定制開發(fā)、品牌網(wǎng)站制作
聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來源： 創(chuàng)新互聯(lián)
- 品牌網(wǎng)站設(shè)計(jì)：如何制作設(shè)計(jì)一個(gè)好的網(wǎng)站？ 2016-10-06
- 品牌網(wǎng)站設(shè)計(jì)中大小標(biāo)題之間的色彩關(guān)系 2022-12-03
- 品牌網(wǎng)站設(shè)計(jì)制作-是企業(yè)對(duì)外的名片和形象 2020-11-28
- 深圳品牌網(wǎng)站設(shè)計(jì)公司 2014-05-13
- 吳江高端網(wǎng)站建設(shè)制作品牌網(wǎng)站設(shè)計(jì)如何布局 2020-11-23
- 怎樣做深圳品牌網(wǎng)站設(shè)計(jì)才有效果 2021-10-18
- 品牌網(wǎng)站設(shè)計(jì)定制解決方案 2021-08-16
- 品牌網(wǎng)站設(shè)計(jì)-好的用戶體驗(yàn)更容易留住用戶 2022-04-30
- 我要做的是品牌網(wǎng)站設(shè)計(jì)，實(shí)際上是品牌網(wǎng)頁(yè)優(yōu)化設(shè)計(jì)，可懂? 2022-05-23
- 優(yōu)秀品牌網(wǎng)站設(shè)計(jì)欣賞 2014-06-25
- 品牌網(wǎng)站設(shè)計(jì)對(duì)企業(yè)來說有什么意義 2021-10-27
- 品牌網(wǎng)站設(shè)計(jì)應(yīng)該考慮的問題有哪些？ 2023-03-23