"Failed connect to 127.0.0.1:2003; Connection refused" for all services and hosts

icingaweb2

(Matt Herzog) #1

New to carbon, graphite and icinga-graphite-web and yes, I
I can telnet to port 2003. I’m using Icinga2 2.6.3 and icingaweb2-2.4.2-1.el7.icinga.noarch on CentOS 7.4.

https://paste.ubuntu.com/p/npfCpbjNgw/ <-- carbon.conf

Apache error log:
[Tue Mar 13 20:30:00.493319 2018] [:error] [pid 26322] mod_wsgi (pid=26322): Target WSGI script ‘/usr/share/graphite/graphite-web.wsgi’ cannot be loaded as Python module.
[Tue Mar 13 20:30:00.493347 2018] [:error] [pid 26322] mod_wsgi (pid=26322): Exception occurred processing WSGI script ‘/usr/share/graphite/graphite-web.wsgi’.
[Tue Mar 13 20:30:00.493379 2018] [:error] [pid 26322] Traceback (most recent call last):
[Tue Mar 13 20:30:00.493403 2018] [:error] [pid 26322] File “/usr/share/graphite/graphite-web.wsgi”, line 22, in
[Tue Mar 13 20:30:00.493468 2018] [:error] [pid 26322] import graphite.metrics.search # noqa
[Tue Mar 13 20:30:00.493481 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/graphite/metrics/search.py”, line 6, in
[Tue Mar 13 20:30:00.493538 2018] [:error] [pid 26322] from graphite.storage import is_pattern, match_entries
[Tue Mar 13 20:30:00.493550 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/graphite/storage.py”, line 9, in
[Tue Mar 13 20:30:00.493658 2018] [:error] [pid 26322] from graphite.remote_storage import RemoteStore
[Tue Mar 13 20:30:00.493672 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/graphite/remote_storage.py”, line 8, in
[Tue Mar 13 20:30:00.493740 2018] [:error] [pid 26322] from graphite.util import unpickle
[Tue Mar 13 20:30:00.493754 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/graphite/util.py”, line 81, in
[Tue Mar 13 20:30:00.493822 2018] [:error] [pid 26322] defaultUser = User.objects.get(username=‘default’)
[Tue Mar 13 20:30:00.493838 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/models/manager.py”, line 151, in get
[Tue Mar 13 20:30:00.493916 2018] [:error] [pid 26322] return self.get_queryset().get(*args, **kwargs)
[Tue Mar 13 20:30:00.493929 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/models/query.py”, line 304, in get
[Tue Mar 13 20:30:00.494267 2018] [:error] [pid 26322] num = len(clone)
[Tue Mar 13 20:30:00.494281 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/models/query.py”, line 77, in len
[Tue Mar 13 20:30:00.494300 2018] [:error] [pid 26322] self._fetch_all()
[Tue Mar 13 20:30:00.494310 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/models/query.py”, line 857, in _fetch_all
[Tue Mar 13 20:30:00.494325 2018] [:error] [pid 26322] self._result_cache = list(self.iterator())
[Tue Mar 13 20:30:00.494335 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/models/query.py”, line 220, in iterator
[Tue Mar 13 20:30:00.494348 2018] [:error] [pid 26322] for row in compiler.results_iter():
[Tue Mar 13 20:30:00.494358 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 713, in results_iter
[Tue Mar 13 20:30:00.494605 2018] [:error] [pid 26322] for rows in self.execute_sql(MULTI):
[Tue Mar 13 20:30:00.494624 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 786, in execute_sql
[Tue Mar 13 20:30:00.494643 2018] [:error] [pid 26322] cursor.execute(sql, params)
[Tue Mar 13 20:30:00.494653 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/backends/util.py”, line 53, in execute
[Tue Mar 13 20:30:00.494729 2018] [:error] [pid 26322] return self.cursor.execute(sql, params)
[Tue Mar 13 20:30:00.494742 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/utils.py”, line 99, in exit
[Tue Mar 13 20:30:00.494835 2018] [:error] [pid 26322] six.reraise(dj_exc_type, dj_exc_value, traceback)
[Tue Mar 13 20:30:00.494854 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/backends/util.py”, line 53, in execute
[Tue Mar 13 20:30:00.494875 2018] [:error] [pid 26322] return self.cursor.execute(sql, params)
[Tue Mar 13 20:30:00.494885 2018] [:error] [pid 26322] File “/usr/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py”, line 452, in execute
[Tue Mar 13 20:30:00.495026 2018] [:error] [pid 26322] return Database.Cursor.execute(self, query, params)
[Tue Mar 13 20:30:00.495057 2018] [:error] [pid 26322] OperationalError: no such table: auth_user


(Matt Herzog) #2

icinga2/features-available/graphite.conf

library “perfdata”

object GraphiteWriter “graphite” {
host = “127.0.0.1”
port = 2003
}


(Matt Herzog) #3

Carbon listener.log:
13/03/2018 23:13:03 :: MetricLineReceiver connection with 127.0.0.1:40430 established
13/03/2018 23:13:03 :: invalid line (GET /metrics/expand?query=icinga2.prod_polywog_net.services.Cluster_Health_Check.cluster.perfdata.%2A.va) received from client 127.0.0.1:40430, ignoring
) received from client 127.0.0.1:40430, ignoring icingaweb2-module-graphite
) received from client 127.0.0.1:40430, ignoring.0.1:2003
) received from client 127.0.0.1:40430, ignoring


(Chris) #4

seems like the databse was not set up.
What guide did you use to set it up ?
Maybe you can explain what you did already to set up graphite


(Matt Herzog) #5

I found a doc that explained how to set up the database but something somewhere had the wrong charset. When I click on Hosts or Services under Graphite I get:
SQLSTATE[42000]: Syntax error or access violation: 1253 COLLATION ‘latin1_general_ci’ is not valid for CHARACTER SET ‘utf8’, query was: SELECT ho.name1 AS host_name, h.display_name COLLATE latin1_general_ci AS host_display_name, hs.check_command AS host_check_command, hcv_check_command.varvalue AS _host_check_command FROM icinga_objects AS ho
INNER JOIN icinga_hosts AS h ON h.host_object_id = ho.object_id AND ho.is_active = 1 AND ho.objecttype_id = 1
INNER JOIN icinga_hoststatus AS hs ON hs.host_object_id = ho.object_id
LEFT JOIN icinga_customvariablestatus AS hcv_check_command ON h.host_object_id = hcv_check_command.object_id AND hcv_check_command.varname = ‘check_command’ COLLATE latin1_general_ci ORDER BY h.display_name COLLATE latin1_general_ci ASC LIMIT 25


(Matt Herzog) #6

Also, do I need an entry in /etc/icingaweb2/resources.ini?


(Michael Friedrich) #7

I’d be interested in the resources.ini and the ido section (remove sensitive credentials).

Note aside - the Graphite module for Icinga Web 2 needs at least 2.5.1, your setup uses 2.4.2.


(Matt Herzog) #8

I have upgraded. Sorry I failed to mention this. Here is what I had installed before I ran ‘python /usr/lib/python2.7/site-packages/graphite/manage.py syncdb

icinga2-selinux-2.8.1-1.el7.icinga.x86_64
icingaweb2-vendor-lessphp-2.5.1-1.el7.icinga.noarch
icinga2-common-2.8.1-1.el7.icinga.x86_64
icingaweb2-vendor-zf1-2.5.1-1.el7.icinga.noarch
icinga-rpm-release-7-2.el7.centos.noarch
vim-icinga2-2.8.1-1.el7.icinga.x86_64
icingacli-2.5.1-1.el7.icinga.noarch
icingaweb2-vendor-JShrink-2.5.1-1.el7.icinga.noarch
icingaweb2-vendor-HTMLPurifier-2.5.1-1.el7.icinga.noarch
icinga2-bin-2.8.1-1.el7.icinga.x86_64
php-Icinga-2.5.1-1.el7.icinga.noarch
icinga2-libs-2.8.1-1.el7.icinga.x86_64
icingaweb2-common-2.5.1-1.el7.icinga.noarch
icingaweb2-vendor-Parsedown-2.5.1-1.el7.icinga.noarch
icinga2-ido-mysql-2.8.1-1.el7.icinga.x86_64
icingaweb2-vendor-dompdf-2.5.1-1.el7.icinga.noarch
icingaweb2-2.5.1-1.el7.icinga.noarch
icinga2-2.8.1-1.el7.icinga.x86_64


(Matt Herzog) #9

resources.ini.

[icinga_ido]
type = "db"
db = "mysql"
host = "localhost"
dbname = "icinga"
username = "root"
password = "notrealpass"
charset = "utf8"
persistent = "0"
use_ssl = "0"

[icingaweb_db]
type = "db"
db = "mysql"
host = "localhost"
dbname = "icingaweb_db"
username = "mrzaug"
password = "notrealpass"
charset = "utf8"
persistent = "0"
use_ssl = "0"  

[directordb]
type = "db"
db = "mysql"
host = "localhost"
dbname = "mrzaug"
username = "zogman"
password = "nothingreal"
charset = "utf8"
persistent = "0"
use_ssl = "0"

(Michael Friedrich) #10

That’s not gonna work within the IDO resource, and schema hardcoded latin1_cs collation for several columns. Explains the error above.


(Matt Herzog) #11

I’m really confused. If I change the charset to latin1_cs, the config can’t be validated in the web interface.


(Matt Herzog) #12

Are you saying the graphite mysql db needs to to be created with latin1_cs ?


(Michael Friedrich) #13

Did I say to change the value to latin_cs? I was explaining why it won’t work.

Remove the config setting and leave it to its default.


(Matt Herzog) #14

That’s confusing af. Why is there a field to select charset if one should not specify a charset? But you’re right of course.


(Michael Friedrich) #15

The database resource itself is a free form, where one can set the field. This not only serves as base for the “ido” resource, but any datasource which can be configured. E.g. one which is fully utf8, and provides import data for the Director module.

The documentation doesn’t tell you to explicitly set utf8 for the IDO resource, so ticking everything out of curiosity isn’t a good idea :wink:


(Matt Herzog) #16

On my production 2.6.3 system at work, the charset is not specified, and never has been. I should have referred to that instance.