JsonRpcConnection error in satellite log file ​

This forum was archived to /woltlab and is now in read-only mode.
  • Getting JsonRpcConnection error in satellite log file



    Refered this thread Certificate Issue for a Icinga Rookie

    But in this node wizard is being used.


    In our project we are using top down approach with 1 master and presently there are 2 satellite-zones which are per VPC. So, 1 satellite is in AWS-Mumbai other is in on-premise . Behind each satellite there are child nodes which DO NOT talk to the master directly. Child nodes send data to satellite end point from where it is sent to Master node. Communication port is 5665.


    Icinga2 is installed on all nodes and ca.crt was generated only once on master node. Master node contains all crt,csr & key files for the satellite as well as child nodes. Now the issue I am facing here is error on child node::



    [2017-09-24 04:36:31 +0530] information/ApiListener: Reconnecting to endpoint 'AWS_Satellite_Mumbai' via host 'yy.yy.yy.yy' and port '5665'

    [2017-09-24 04:36:31 +0530] information/ApiListener: New client connection for identity 'AWS_Satellite_Mumbai' to [yy.yy.yy.yy]:5665

    [2017-09-24 04:36:31 +0530] information/ApiListener: Finished reconnecting to endpoint 'AWS_Satellite_Mumbai' via host 'yy.yy.yy.yy' and port '5665'

    [2017-09-24 04:36:31 +0530] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Sending config updates for endpoint 'AWS_Satellite_Mumbai' in zone 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Finished sending config file updates for endpoint 'AWS_Satellite_Mumbai' in zone 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Syncing runtime objects to endpoint 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Finished syncing runtime objects to endpoint 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Finished sending runtime config updates for endpoint 'AWS_Satellite_Mumbai' in zone 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Sending replay log for endpoint 'AWS_Satellite_Mumbai' in zone 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Finished sending replay log for endpoint 'AWS_Satellite_Mumbai' in zone 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:36:31 +0530] information/ApiListener: Finished syncing endpoint 'AWS_Satellite_Mumbai' in zone 'AWS_Satellite_Mumbai'.

    [2017-09-24 04:37:32 +0530] information/JsonRpcConnection: No messages for identity 'AWS_Satellite_Mumbai' have been received in the last 60 seconds.

    [2017-09-24 04:37:32 +0530] warning/JsonRpcConnection: API client disconnected for identity 'AWS_Satellite_Mumbai'

    [2017-09-24 04:37:32 +0530] warning/JsonRpcConnection: API client disconnected for identity 'AWS_Satellite_Mumbai'

    [2017-09-24 04:37:32 +0530] warning/ApiListener: Removing API client for endpoint 'AWS_Satellite_Mumbai'. 0 API clients left.

    [2017-09-24 04:37:32 +0530] warning/ApiListener: Removing API client for endpoint 'AWS_Satellite_Mumbai'. 0 API clients left.


    And error on Satellite for this child node


    [2017-09-25 03:21:11 +0530] information/WorkQueue: #4 (ApiListener, RelayQueue) items: 0, rate: 24.6833/s (1481/min 7447/5min 22350/15min);

    [2017-09-25 03:21:12 +0530] information/WorkQueue: #9 (JsonRpcConnection, #1) items: 0, rate: 0.833333/s (50/min 260/5min 804/15min);

    [2017-09-25 03:21:12 +0530] information/WorkQueue: #8 (JsonRpcConnection, #0) items: 0, rate: 1.8/s (108/min 491/5min 1469/15min);

    [2017-09-25 03:21:13 +0530] information/ApiListener: New client connection for identity 'In_premise_VM' from [xx.xx.xx.xx]:44264

    [2017-09-25 03:21:13 +0530] information/ApiListener: Sending config updates for endpoint 'In_premise_VM' in zone 'In_premise_VM'.

    [2017-09-25 03:21:13 +0530] critical/SSL: Could not open CA key file '/var/lib/icinga2/ca/ca.key': 151441516, "error:0906D06C:PEM routines:PEM_read_bio:no start line"

    [2017-09-25 03:21:13 +0530] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'In_premise_VM'.

    [2017-09-25 03:21:13 +0530] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/ca/ca.crt': 537346050, "error:20074002:BIO routines:FILE_CTRL:system lib"

    [2017-09-25 03:21:13 +0530] warning/JsonRpcConnection: Error while processing message for identity 'In_premise_VM'

    Error: std::exception


    (0) /usr/lib64/icinga2/libbase.so.2.7.1() [0x34f5ca2977]

    (1) libbase.so.2.7.1: icinga::GetX509Certificate(icinga::String const&) (+0x3cf) [0x34f5ceb79f]

    (2) /usr/lib64/icinga2/libremote.so.2.7.1() [0x34f82eae7a]

    (3) /usr/lib64/icinga2/libremote.so.2.7.1() [0x34f8240baf]

    (4) libremote.so.2.7.1: icinga::ApiFunction::Invoke(boost::intrusive_ptr const&, boost::intrusive_ptr const&) (+0x1d) [0x34f82c693d]

    (5) libremote.so.2.7.1: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x47f) [0x34f82dbbbf]

    (6) libremote.so.2.7.1: icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) (+0x43) [0x34f82dc5b3]

    (7) libbase.so.2.7.1: icinga::WorkQueue::WorkerThreadProc() (+0x4bc) [0x34f5cbfc5c]

    (8) /usr/lib64/libboost_thread.so.1.53.0() [0x34f8e0c5c3]

    (9) /lib64/libpthread.so.0() [0x3f69207aa1]

    (10) libc.so.6: clone (+0x6d) [0x3f68ee8bcd]



    [2017-09-25 03:21:13 +0530] information/ApiListener: Finished sending config file updates for endpoint 'In_premise_VM' in zone 'In_premise_VM'.

    [2017-09-25 03:21:13 +0530] information/ApiListener: Syncing runtime objects to endpoint 'In_premise_VM'.

    [2017-09-25 03:21:13 +0530] information/ApiListener: Finished syncing runtime objects to endpoint 'In_premise_VM'.

    [2017-09-25 03:21:13 +0530] information/ApiListener: Finished sending runtime config updates for endpoint 'In_premise_VM' in zone 'In_premise_VM'.

    [2017-09-25 03:21:17 +0530] warning/JsonRpcConnection: API client disconnected for identity 'In_premise_VM'

    [2017-09-25 03:21:17 +0530] warning/JsonRpcConnection: API client disconnected for identity 'In_premise_VM'

    [2017-09-25 03:21:17 +0530] warning/ApiListener: Removing API client for endpoint 'In_premise_VM'. 1 API clients left.

    [2017-09-25 03:21:17 +0530] warning/ApiListener: Removing API client for endpoint 'In_premise_VM'. 1 API clients left.