Logstash and icingaweb2 integration


(Baidurjya Choudhury) #1

Hello Dear Team, I have installed latest version of icingaweb2, nginx and logstash.

My requirement is to capture nginx error log (http status more than 500) in Logstash and then forward error message from Logstash to Icingaweb2 so that customer can see any server error in icingaweb2 portal.

I need your expertise to develop the logstash.conf file to send logs to icingaweb2 portal.

if anyone could give me a documentation link, that would be really grateful
Thank you for reading and helping out.


#3

Only way I currently know so far is, to store the logstash events in elastic search and use the elasticsearch module for icingaweb2.


(Wolfgang Winter) #4

Hi, you need this output module:


(Baidurjya Choudhury) #5

Dear Wolfgang, many thanks for your reply.

In this scenario, Logstash is running in docker container. I am trying to open one link in nginx http server which does not exist and yields 404 error, for testing purpose, I want this 404 error must change the stutus of nginx service in icinga2.

I have followed the instruction from the link. but still logstash is not able to send logs to icinga2.

I have got following error in logstash log.

[2019-01-17T17:22:08,492][WARN ][logstash.outputs.icinga ] Request failed {:host=>“10.0.2.4”, :port=>5665, :path=>"/v1/actions/process-check-result?service=kmaster%21nginx_errorlog", :body=>"{“exit_status”:“2”,“plugin_output”:"<27>Jan 17 18:21:29 e1b4f8b46ab1[861]: 2019/01/17 17:21:29 [error] 5#5: *3 open() \"/usr/share/nginx/html/super\" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: \“GET /super HTTP/1.1\”, host: \“localhost:8888\”"}", :error=>#<IOError: Broken pipe>}
[2019-01-17T17:22:10,775][WARN ][logstash.outputs.icinga ] Request failed {:host=>“10.0.2.4”, :port=>5665, :path=>"/v1/actions/process-check-result?service=kmaster%21nginx_errorlog", :body=>"{“exit_status”:“2”,“plugin_output”:"<30>Jan 17 18:21:29 e1b4f8b46ab1[861]: 10.0.2.2 - - [17/Jan/2019:17:21:29 +0000] \“GET /super HTTP/1.1\” 404 555 \"-\" \“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36\” \"-\""}", :error=>#<Net::OpenTimeout: Net::OpenTimeout>}
[2019-01-17T17:22:11,730][WARN ][logstash.outputs.icinga ] Request failed {:host=>“10.0.2.4”, :port=>5665, :path=>"/v1/actions/process-check-result?service=kmaster%21nginx_errorlog", :body=>"{“exit_status”:“2”,“plugin_output”:"<27>Jan 17 18:22:01 e1b4f8b46ab1[861]: 2019/01/17 17:22:01 [error] 5#5: *5 open() \"/usr/share/nginx/html/super\" failed (2: No such file or directory), client: 10.0.2.2, server: localhost, request: \“GET /super HTTP/1.1\”, host: \“localhost:8888\”"}", :error=>#<IOError: Connection reset by peer>}
[2019-01-17T17:22:12,727][WARN ][logstash.outputs.icinga ] Request failed {:host=>“10.0.2.4”, :port=>5665, :path=>"/v1/actions/process-check-result?service=kmaster%21nginx_errorlog", :body=>"{“exit_status”:“2”,“plugin_output”:"<30>Jan 17 18:22:01 e1b4f8b46ab1[861]: 10.0.2.2 - - [17/Jan/2019:17:22:01 +0000] \“GET /super HTTP/1.1\” 404 555 \"-\" \“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36\” \"-\""}", :error=>#<IOError: Connection reset by peer>}

but when I am sending same message to icinga2 by CURL command (executed inside logstash docker container) then it’s working. which proves no connectivity issue from inside docker container to icinga2. command below.

curl -k -s -u root:0418dc8e3735aa37 -H ‘Accept: application/json’ -X POST https://10.0.2.4:5665/v1/actions/process-check-result?service=kmaster!nginx_errorlog -d “{“exit_status”:“2”,“plugin_output”:“http 500 error”,“check_source”:“kmaster”}”

Below is the logstash.conf file.

input {
tcp {
type => syslog
port => 5000
add_field => { “[@metadata][input]” => “tcp”}
}
}

filter {
grok {
match => { “message” => “%{IPORHOST:remote_ip} - %{DATA:user_name} [%{HTTPDATE:access_time}] “%{WORD:http_method} %{DATA:url} HTTP/%{NUMBER:http_version}” %{NUMBER:response_code} %{NUMBER:body_sent_bytes} “%{DATA:referrer}” “%{DATA:agent}”” }
}
mutate{
add_field => {
“actual_host_is” => “${ACTUAL_HOST}”
}
}
mutate {
replace => { “exit_status” => “2” }
}
}

output {
elasticsearch{
hosts=>[“elasticsearch:9200”]
index => “logstash-%{+YYYY.MM.dd}”
}
icinga {
host => ‘10.0.2.4’
user => ‘root’
password => ‘0418dc8e3735aa37’
ssl_verify => false
action => ‘process-check-result’
action_config => {
exit_status => “%{exit_status}”
plugin_output => “%{message}”
}
icinga_host => ‘kmaster’
icinga_service => ‘nginx_errorlog’
}
}

do you have any idea ? thanks kindly, Baidurjya