Client certificate validation failed

icinga2
(Alexandr) #1

Hi!

I add icinga clients via ansible playbook with pki and node commands. I use this a long time. But now i receive error with client certificate validation while node setup:

information/ApiListener: New client connection for identity 'client.com' from [123.123.123.123]:35344 (certificate validation failed: code 18: self signed certificate)
warning/ApiListener: No data received on new API connection for identity 'client.com'. Ensure that the remote endpoints are properly configured in a cluster setup.

I reproduce this manually but same error.

Manual configure

Generate new cert for client.com (executed on client.com)

icinga2 pki new-cert --cn client.com --key /etc/icinga2/pki/client.com.key --cert /etc/icinga2/pki/client.com.crt

Save new cert (executed on client.com)

icinga2 pki save-cert --key /etc/icinga2/pki/client.com.key --cert /etc/icinga2/pki/client.com.crt --trustedcert /etc/icinga2/pki/trusted-master.crt --host server.com

Get ticket on server for client (executed on server.com)

icinga2 pki ticket --cn client.com

Add client to server (executed on client.com, deprecated master_host parameter has changed to parent_host )

icinga2 node setup --ticket ticket_from_previous_step --endpoint server.com --zone client.com --parent_host server.com --trustedcert /etc/icinga2/pki/trusted-master.crt

I have error on last step.

Environment

Icinga version: r2.10.2-1 (in all servers)
Operating System and version: Debian 9.6 amd64 (in all servers)

1 Like
(Alexandr) #2

I reinstalled icinga in server and generate new certs via icinga2 node wizard . Now i have this error while generate certs on all client servers (including using wizard).

#3

Hi,

does this also occur if you use the node wizard on the client?

Please provide full logs from client (agent) and server (master), also include the zones.conf from both nodes after you can the manual configure steps.

Kind regards
Michael

1 Like
(Alexandr) #4

yes

Now i test with new server as client (agent).

There is only below strings related with setup:

information/ApiListener: New client connection for identity 'client.com' from [123.123.123.123]:35344 (certificate validation failed: code 18: self signed certificate)
warning/ApiListener: No data received on new API connection for identity 'client.com'. Ensure that the remote endpoints are properly configured in a cluster setup.

zones.conf on server (master):

object Endpoint NodeName {
}
object Zone ZoneName {
	endpoints = [ NodeName ]
}
object Zone "global-templates" {
	global = true
}
object Zone "director-global" {
	global = true
}

No logs yet and default zones.conf on client (agent).

Node setup and wizard on clients (agents) ends with below errors (in server (agent) logs):

critical/cli: Could not fetch valid response. Please check the master log.
critical/cli: Failed to fetch signed certificate from master 'server.com, 5665'. Please try again.

Wizard ask ticket again after errors.

Other old clients (agents) also cannot connect to the server (master) with new certificates. I tried icinga2 node setup and icinga node wizard on old clients (agents). Have same errors. But connected fine before regenerate certs on server (master).

Сonnection of clients to server worked perfectly earlier (last client setup 2 months ago).

(Michael Friedrich) #5

Please add timestamps to the logs to get a better picture about the workflow.

(Alexandr) #6

Now i do below steps

  • stop icinga in all old connected clients (agents),
  • restart icinga on server (master),
  • execute wizard on new client (agent)

All logs entries after restarting icinga on master:

[2019-01-22 09:22:57 +0300] information/FileLogger: 'main-log' started.
[2019-01-22 09:22:57 +0300] information/ApiListener: 'api' started.
[2019-01-22 09:22:57 +0300] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2019-01-22 09:22:57 +0300] information/ExternalCommandListener: 'command' started.
[2019-01-22 09:22:57 +0300] information/InfluxdbWriter: 'influxdb' started.
[2019-01-22 09:22:57 +0300] information/CheckerComponent: 'checker' started.
[2019-01-22 09:22:57 +0300] information/NotificationComponent: 'notification' started.
[2019-01-22 09:22:57 +0300] information/DbConnection: 'ido-mysql' started.
[2019-01-22 09:22:57 +0300] information/ConfigItem: Activated all objects.
[2019-01-22 09:22:57 +0300] information/cli: Closing console log.
[2019-01-22 09:22:57 +0300] information/DbConnection: Resuming IDO connection: ido-mysql
[2019-01-22 09:22:57 +0300] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2019-01-22 09:22:57 +0300] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2019-01-22 09:22:57 +0300] information/IdoMysqlConnection: Finished reconnecting to MySQL IDO database in 0.0464492 second(s).
[2019-01-22 09:23:07 +0300] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.366667/s (22/min 22/5min 22/15min);
[2019-01-22 09:23:07 +0300] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 0, rate: 0.0333333/s (2/min 2/5min 2/15min);
[2019-01-22 09:23:07 +0300] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2019-01-22 09:23:07 +0300] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 0, rate: 0.383333/s (23/min 23/5min 23/15min);
[2019-01-22 09:23:47 +0300] information/ApiListener: New client connection from [123.123.123.123]:29390 (no client certificate)
[2019-01-22 09:23:47 +0300] information/ApiListener: No data received on new API connection. Ensure that the remote endpoints are properly configured in a cluster setup.
[2019-01-22 09:23:47 +0300] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 3, rate: 2.65/s (159/min 159/5min 159/15min);
[2019-01-22 09:24:09 +0300] information/ApiListener: New client connection for identity 'client.com' from [123.123.123.123]:29392 (certificate validation failed: code 18: self signed certificate)
[2019-01-22 09:24:19 +0300] warning/ApiListener: No data received on new API connection for identity 'client.com'. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
	(0) Handling new API client connection
(Alexandr) #7

Now i tried manually copy new ca.crt to one of old client (agent). Working fine:

[2019-01-22 09:42:54 +0300] information/ApiListener: New client connection for identity 'old-client.com' from [123.123.123.123]:62572
[2019-01-22 09:42:54 +0300] information/ApiListener: Sending config updates for endpoint 'old-client.com' in zone 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/ApiListener: Finished sending config file updates for endpoint 'old-client.com' in zone 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/ApiListener: Syncing runtime objects to endpoint 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/ApiListener: Finished syncing runtime objects to endpoint 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/ApiListener: Finished sending runtime config updates for endpoint 'old-client.com' in zone 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/ApiListener: Sending replay log for endpoint 'old-client.com' in zone 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/ApiListener: Finished sending replay log for endpoint 'old-client.com' in zone 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/ApiListener: Finished syncing endpoint 'old-client.com' in zone 'old-client.com'.
[2019-01-22 09:42:54 +0300] information/JsonRpcConnection: Received certificate request for CN 'old-client.com' signed by our CA.
[2019-01-22 09:42:54 +0300] information/JsonRpcConnection: The certificate for CN 'old-client.com' is valid and uptodate. Skipping automated renewal.
[2019-01-22 09:43:04 +0300] information/WorkQueue: #12 (JsonRpcConnection, #0) items: 0, rate: 573.9/s (34434/min 34434/5min 34434/15min);
[2019-01-22 09:43:04 +0300] information/WorkQueue: #13 (JsonRpcConnection, #1) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2019-01-22 09:44:17 +0300] information/ExternalCommandListener: Executing external command: [1548139457] SCHEDULE_FORCED_SVC_CHECK;old-client.com;apt;1548139457

I think server (master) can’t verify ticket from client (agent) during node setup and node wizard although ticket and certificates are valid.

(Alexandr) #8

Icinga can’t start after complete configuring on new client (agent) with errors:

Jan 22 17:21:11 client.com systemd[1]: Starting Icinga host/service/network monitoring system...
-- Subject: Unit icinga2.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit icinga2.service has begun starting up.
Jan 22 17:21:11 client.com icinga2[5848]: [2019-01-22 17:21:11 +0500] information/cli: Icinga application loader (version: r2.10.2-1)
Jan 22 17:21:11 client.com icinga2[5848]: [2019-01-22 17:21:11 +0500] information/cli: Loading configuration file(s).
Jan 22 17:21:11 client.com icinga2[5848]: [2019-01-22 17:21:11 +0500] information/ConfigItem: Committing config item(s).
Jan 22 17:21:11 client.com icinga2[5848]: [2019-01-22 17:21:11 +0500] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//client.com.crt': 33558541, "error:0200100D:system library:fopen:Permission denied"
Jan 22 17:21:11 client.com icinga2[5848]: [2019-01-22 17:21:11 +0500] critical/config: Error: Cannot get certificate from cert path: '/var/lib/icinga2/certs//client.com.crt'.
Jan 22 17:21:11 client.com icinga2[5848]: Location: in /etc/icinga2/features-enabled/api.conf: 6:1-6:24
Jan 22 17:21:11 client.com icinga2[5848]: /etc/icinga2/features-enabled/api.conf(4):  */
Jan 22 17:21:11 client.com icinga2[5848]: /etc/icinga2/features-enabled/api.conf(5):
Jan 22 17:21:11 client.com icinga2[5848]: /etc/icinga2/features-enabled/api.conf(6): object ApiListener "api" {
Jan 22 17:21:11 client.com icinga2[5848]:                                            ^^^^^^^^^^^^^^^^^^^^^^^^
Jan 22 17:21:11 client.com icinga2[5848]: /etc/icinga2/features-enabled/api.conf(7):   accept_config = true
Jan 22 17:21:11 client.com icinga2[5848]: /etc/icinga2/features-enabled/api.conf(8):   accept_commands = true
Jan 22 17:21:11 client.com icinga2[5848]: [2019-01-22 17:21:11 +0500] critical/config: 1 error
Jan 22 17:21:11 client.com systemd[1]: icinga2.service: Main process exited, code=exited, status=1/FAILURE
Jan 22 17:21:11 client.com systemd[1]: Failed to start Icinga host/service/network monitoring system.

No problem if disable api feature. But i need API)

Then i generate new client sert on old client and get same error. Perhaps error is related to the generation of server certificate.

# ls -la /var/lib/icinga2/certs
drw------- 2 nagios nagios 4096 Jan 22 16:55 .
drwxr-x--- 4 nagios nagios 4096 Jan 22 16:48 ..
-rw-r--r-- 1 nagios nagios 1761 Jan 22 16:55 client.com.crt
-rw------- 1 nagios nagios 3243 Jan 22 16:55 client.com.key
-rw-r--r-- 1 nagios nagios 1719 Jan 22 16:55 ca.crt
-rw-r--r-- 1 root   root   1749 Jan 22 16:55 trusted-master.crt
#9

Honestly, I am confused. Which CLI commands did you use? What did you copy from where to where? Are client.com and old-client.com the same machine?

Let’s take a step back and do one thing after the other.

Since you are working on a production machines I would create a full backup before (including /var/lib/icinga2/* and /etc/icinga2/*).

  1. Verify that the master is operational

Run icinga2 node wizard on the master and (re-) run the master setup. It should check if the necessary certificates are in place, if not the wizard will create the missing certificates.

  1. Manual client setup

If the master is good, you can now try to run the client setup manually. Run icinga2 node wizard and choose the client setup.

  1. Verify the connection now works

Restart both nodes and monitor the logs for anomalies.

  1. If the manual client setup works you can begin with automating it.

Otherwise please show us the entire output of the single CLI commands you ran (from master and client) with the respective logs. Also include the zones.conf from both nodes.

Also you shouldn’t manually copy things to the /var/lib/icinga2 directories. This isn’t necessary under normal conditions and can often cause problems. Use instead the respective CLI commands to deal with certificates.

Regards
Michael

(Alexandr) #10

I use only node wizard on master. I tried use node wizard and node setup on agents. Ands with same errors.

On agents while agent setup:

critical/cli: Could not fetch valid response. Please check the master log.
critical/cli: Failed to fetch signed certificate from master 'server.com, 5665'. Please try again.

On master while agent setup (in logs):

information/ApiListener: New client connection for identity 'client.com' from [123.123.123.123]:35344 (certificate validation failed: code 18: self signed certificate)
warning/ApiListener: No data received on new API connection for identity 'client.com'. Ensure that the remote endpoints are properly configured in a cluster setup.

I copied only ca.crt from master to agents.

  1. I did that but problem stay.

  2. It crash on node wizard and node setup.

  3. I did that but problem stay.

Previously connected and working agents now works perfectly after manually copy new ca.crt from master. node wizard and node setup crash on any client nodes. Any client node does not work if i generate new certificates for it and manually copy ca.crt from master. Icinga can’t start. I tried node wizard on previously connected and working agent. It also crash as on new client.

As a result, the problem is in new generated clients certificates. No problem with old clents certificates. It seems the problem is in generation of broken certificates.

#11

What do you mean with new ca.crt? Did you re-create the Icinga CA on the master?

(Alexandr) #12

Yes. I reconfigured certs on master via node wizard.

#13

Can you please try the steps in the troubleshooting steps for certificate issues.

https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#certificate-troubleshooting

I am especially interested if the master ca.crt matches with the received ca.crt on the client. Also check if all necessary attributes are set.

How does the network look between the two nodes (master and client)? Is there any firewall, filter, etc in place what may can interfere the connection?

Can you please also provide a full output from the client running node wizard until you hit the error, e.g.:

root@testing-deb9-icinga2-client:/# icinga2 node wizard

Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]: 

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [testing-deb9-icinga2-client]: 

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): testing-deb9-icinga2-master

Do you want to establish a connection to the parent node from this node? [Y/n]: 
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 172.17.0.2
Master/Satellite endpoint port [5665]: 

Add more master/satellite endpoints? [y/N]: 
Parent certificate information:

 Subject:     CN = testing-deb9-icinga2-master
 Issuer:      CN = Icinga CA
 Valid From:  Jan 23 18:47:17 2019 GMT
 Valid Until: Jan 19 18:47:17 2034 GMT
 Fingerprint: 88 9A 08 E6 7B BE E2 E0 BB 91 85 7C 7F E6 07 AA 58 8B D0 27 

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'testing-deb9-icinga2-client'): 7d726179199c84c41a71874689b9194f35bfddf5
Please specify the API bind host/port (optional):
Bind Host []: 
Bind Port []: 

Accept config from parent node? [y/N]: 
Accept commands from parent node? [y/N]: 

Reconfiguring Icinga...

Local zone name [testing-deb9-icinga2-client]: 
Parent zone name [master]: 

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: 

Do you want to disable the inclusion of the conf.d directory [Y/n]: 
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!

We need to know at which exact point the error happens.

(Alexandr) #14

System create not valid client cert.

Configuring client (agent)

root@client # icinga2 node wizard 
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]: 

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [client.com]: 

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): server.com

Do you want to establish a connection to the parent node from this node? [Y/n]: 
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): server.com
Master/Satellite endpoint port [5665]: 

Add more master/satellite endpoints? [y/N]: 
Parent certificate information:

 Subject:     CN = server.com
 Issuer:      CN = Icinga CA
 Valid From:  Jan 21 15:58:51 2019 GMT
 Valid Until: Jan 17 15:58:51 2034 GMT
 Fingerprint: E7 A6 75 6E C2 B0 B1 62 1B 54 5E B9 76 90 BC 60 CA 35 44 38 

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'client.com'): 5c31f5df26fc5aff2e585daf2160a1f6776cc1d9
critical/cli: Could not fetch valid response. Please check the master log.
critical/cli: Failed to fetch signed certificate from master 'server.com, 5665'. Please try again.

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'client.com'): 5c31f5df26fc5aff2e585daf2160a1f6776cc1d9
critical/cli: Could not fetch valid response. Please check the master log.
critical/cli: Failed to fetch signed certificate from master 'server.com, 5665'. Please try again.

Strings from server (master) logs:

[2019-01-24 08:48:17 +0300] information/ApiListener: New client connection for identity 'client.com' from [123.123.123.123]:33950 (certificate validation failed: code 18: self signed certificate)
[2019-01-24 08:48:27 +0300] warning/ApiListener: No data received on new API connection for identity 'client.com'. Ensure that the remote endpoints are properly configured in a cluster setup.

Certs checking

CA certificate

root@server # openssl x509 -in ca.crt -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            41:77:b2:e8:fb:2f:ef:c9:4b:3c:81:2b:78:2c:df:ea:f5:75:40:40
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Icinga CA
        Validity
            Not Before: Jan 21 15:58:51 2019 GMT
            Not After : Jan 17 15:58:51 2034 GMT
        Subject: CN = Icinga CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    .........
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         ...........
-----BEGIN CERTIFICATE-----
.........
-----END CERTIFICATE-----

Client public certificate

root@client # openssl x509 -in client.com.crt -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            e6:d1:9e:73:1f:23:16:97:96:63:a5:f7:3e:4c:9f:2c:7a:27:ea:36
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = client.com
        Validity
            Not Before: Jan 22 12:34:48 2019 GMT
            Not After : Jan 18 12:34:48 2034 GMT
        Subject: CN = client.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    .......
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:client.com
    Signature Algorithm: sha256WithRSAEncryption
         ........
-----BEGIN CERTIFICATE-----
.........
-----END CERTIFICATE-----

Verify server cert with CA cert

root@server # openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/server.com.crt 
/var/lib/icinga2/certs/server.com.crt: OK

Verify client cert with CA cert (ca.crt copied manually via scp, because wizard has not finished setup and not copy it)

root@client # openssl verify -verbose -CAfile /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/certs/client.com.crt 
CN = client.com
error 18 at 0 depth lookup: self signed certificate
error /var/lib/icinga2/certs/client.com.crt: verification failed

P.S. Real server (master) FQDN changed to server.com, real client (agent) FQDN changed to client.com, real IP changed to 123.123.123.123.

(Alexandr) #15

Now i tried downgrade openssl and regenerate certs. Same errors.

#16

That the last check (verifying the client.crt against the ca.crt) failed is right, since the client couldn’t fetch a signed certificate it is indeed self signed, so this is correct.

What happens if you use the on demand signing method? Leave the ticket blank during the node wizard on the client and use ca list and ca sign <fingerprint> on the master to sign the certificate on demand.

https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#on-demand-csr-signing

How are the nodes (master <-> client) connected? Is anything on the network layer that could interfere the connection between them?

(Alexandr) #17

List is empty.

These are ordinary public vps. Only iptables rules can interfere, but I tried to reset iptables rules to standard. Another previously connected client can’t connect after certificate regeneration via node wizard and node setup. All other clients with old certificates and with new CA certificate are connected and work fine with master server.

#18

One last thing you can try is to create and sign the certificate manually.

  1. Create key and CSR on the client
icinga2 pki new-cert --cn hostname --key hostname.key --csr hostname.csr
  1. Copy CSR to the master & sign it

After you copied hosname.csr to the master yo need to sign it.

icinga2 pki sign-csr --csr hostname.csr --cert hostname.crt
  1. Copy the signed certificate & ca.crt to the client and move it to the right directory

The files needs to be placed in /var/lib/icinga2/certs

ls -la /var/lib/icinga2/certs/          

drwxr-xr-x 2 nagios nagios 4096 Jan 27 18:31 .
drwxr-x--- 4 nagios nagios 4096 Jan 27 18:34 ..
-rw-r--r-- 1 nagios nagios 1720 Jan 27 18:31 ca.crt
-rw-r--r-- 1 nagios nagios 1794 Jan 27 18:24 hostname.crt
-rw-r--r-- 1 nagios nagios 1687 Jan 27 18:24 hostname.csr
-rw------- 1 nagios nagios 3243 Jan 27 18:24 hostname.key

Ensure the permissions are probably set.

  1. Ensure the api feature is enabled and the zones.conf is valid
icinga2 feature enable api
object Endpoint "yourmaster" {
	host = "000.000.000.000"
	port = "5665"
}

object Zone "master" {
	endpoints = [ "yourmaster" ]
}

object Endpoint "hostname" {
}

object Zone "hostname" {
	endpoints = [ "hostname" ]
	parent = "master"
}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}

Ensure that you have a corresponding Endpoint & Zone object in your masters zones.conf.

If this works there is a problem to transport the CSR and/or signed CRT to the master/client during the node wizard. This would need further analysis on the network layer.

(Alexandr) #19

I did it, but some errors:

-- Unit icinga2.service has begun starting up.
Jan 28 12:59:29 client.com icinga2[2721]: [2019-01-28 12:59:29 +0300] information/cli: Icinga application loader (version: r2.10.2-1)
Jan 28 12:59:29 client.com icinga2[2721]: [2019-01-28 12:59:29 +0300] information/cli: Loading configuration file(s).
Jan 28 12:59:29 client.com icinga2[2721]: [2019-01-28 12:59:29 +0300] information/ConfigItem: Committing config item(s).
Jan 28 12:59:29 client.com icinga2[2721]: [2019-01-28 12:59:29 +0300] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//client.com.crt': 33558541, "error:0200100D:system library:fopen:Permission denied"
Jan 28 12:59:29 client.com icinga2[2721]: [2019-01-28 12:59:29 +0300] critical/config: Error: Cannot get certificate from cert path: '/var/lib/icinga2/certs//client.com.crt'.
Jan 28 12:59:29 client.com icinga2[2721]: Location: in /etc/icinga2/features-enabled/api.conf: 6:1-6:24
Jan 28 12:59:29 client.com icinga2[2721]: /etc/icinga2/features-enabled/api.conf(4):  */
Jan 28 12:59:29 client.com icinga2[2721]: /etc/icinga2/features-enabled/api.conf(5):
Jan 28 12:59:29 client.com icinga2[2721]: /etc/icinga2/features-enabled/api.conf(6): object ApiListener "api" {
Jan 28 12:59:29 client.com icinga2[2721]:                                            ^^^^^^^^^^^^^^^^^^^^^^^^
Jan 28 12:59:29 client.com icinga2[2721]: /etc/icinga2/features-enabled/api.conf(7):   accept_config = true
Jan 28 12:59:29 client.com icinga2[2721]: /etc/icinga2/features-enabled/api.conf(8):   accept_commands = true
Jan 28 12:59:29 client.com icinga2[2721]: [2019-01-28 12:59:29 +0300] critical/config: 1 error
Jan 28 12:59:29 client.com systemd[1]: icinga2.service: Main process exited, code=exited, status=1/FAILURE
Jan 28 12:59:29 client.com systemd[1]: Failed to start Icinga host/service/network monitoring system.
ls -la /var/lib/icinga2/certs/

drw------- 2 nagios nagios 4096 Jan 28 12:49 .
drwxr-x--- 4 nagios nagios 4096 Jan 28 12:02 ..
-rw-r--r-- 1 nagios nagios 1719 Jan 28 12:01 ca.crt
-rw-r--r-- 1 nagios nagios 1765 Jan 28 12:49 client.com.crt
-rw-r--r-- 1 nagios nagios 1655 Jan 28 12:01 client.com.csr
-rw------- 1 nagios nagios 3247 Jan 28 12:01 client.com.key
#20

This is now a permission issue:

[2019-01-28 12:59:29 +0300] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//client.com.crt': 33558541, "error:0200100D:system library:fopen:Permission denied"

How did you install Icinga 2? Which user runs the icinga2 daemon?