Troubleshooting
===============



Server installation errors
--------------------------

RHEL 8
^^^^^^

1. While installing the PEM server on RHEL 8, if you see this error:

.. code:: text

   [root@etpgxlt firstuser]# dnf install edb-pem
   Updating Subscription Management repositories.
   Last metadata expiration check: 0:01:33 ago on Wed 30 Mar 2022 01:28:16 AM EDT.
   Error: 
    Problem: problem with installed package python3-mod_wsgi-4.6.4-4.el8.s390x
     - package python39-mod_wsgi-4.7.1-4.module+el8.4.0+9822+20bf1249.s390x conflicts with python3-mod_wsgi provided by python3-mod_wsgi-4.6.4-4.el8.s390x
     - package python39-mod_wsgi-4.7.1-4.module+el8.4.0+9822+20bf1249.s390x conflicts with python3-mod_wsgi provided by python3-mod_wsgi-4.6.4-3.el8.s390x
     - package edb-pem-server-8.4.0-7.rhel8.s390x requires python39-mod_wsgi >= 4.7, but none of the providers can be installed
     - package edb-pem-8.4.0-7.rhel8.s390x requires edb-pem-server = 8.4.0-7.rhel8, but none of the providers can be installed
     - cannot install the best candidate for the job
   (try to add --allowerasing to command line to replace conflicting packages or --skip-broken to skip uninstallable packages or --nobest to use not only best candidate packages)
   [root@etpgxlt firstuser]# 

Remove the ``python3-mod_wsgi`` package first:

.. code:: shell

   dnf remove python3-mod_wsgi

Try installing the PEM server again.

2. On RHEL 8, if you see this error in the worker.log after configuring
   the PEM server:

.. code:: text

   Tue Nov 28 03:02:19 2023 WARNING: Error clearing zombies: ERROR:  failed to JIT module: Added modules have incompatible data layouts: E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-a:8:16-n32:64 (module) vs E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-v128:64-a:8:16-n32:64 (jit)
   CONTEXT:  SQL statement "WITH running_agent_job AS (
           SELECT j.jobid, j.agent_id
           FROM pem.job j LEFT JOIN pem.joblog jl ON (j.jobid = jl.jlgjobid)
           WHERE jl.jlgstatus = r AND agent_id = $1 AND j.jobarid != agent_runtime_id
           ORDER BY j.jobid, jl.jlgjobid
           FOR UPDATE SKIP LOCKED
       ), joblog_status_update AS (
           UPDATE pem.joblog jl SET jlgstatus=d
           FROM running_agent_job r
           WHERE r.jobid = jl.jlgjobid AND jl.jlgstatus=r
           RETURNING r.agent_id, jl.jlgjobid, jl.jlgid
       ), jobsteplog_status_update AS (
           UPDATE pem.jobsteplog js SET jslstatus=d
           FROM joblog_status_update jl
           WHERE js.jsljlgid = jl.jlgid AND js.jslstatus=r
           RETURNING jl.agent_id, jl.jlgjobid AS job_id
       )
       UPDATE pem.job j SET jobprocessid=NULL, jobnextrun=NULL, jobarid=NULL
       FROM (SELECT DISTINCT agent_id, job_id FROM jobsteplog_status_update) js
       WHERE js.job_id = j.jobid"
   PL/pgSQL function pem.clear_job_zombies(integer) line 8 at SQL statement

To resolve the error, set the ``jit`` parameter to ``off`` in
``postgresql.conf`` file of the backend database server:

.. code:: ini

     jit=off

Restart the backend database server.

Reconfiguring the PEM server
----------------------------

In some situations, you might need to uninstall the PEM server,
reinstall it, and then configure the server again. To do so:

1. Remove the PEM server configuration and uninstall:

.. code:: shell

       /usr/edb/pem/bin/configure-pem-server.sh –un

2. Remove the PEM packages:

.. code:: shell

       yum erase edb-pem-server

3. Drop the ``pem`` database:

.. code:: sql

       DROP DATABASE pem

4. Move the certificates from ``/root/.pem/`` to another location:

.. code:: shell

       mv /root/.pem/* <new_location>

5. Move the ``agent.cfg`` file from ``/usr/edb/pem/agent/etc/agent.cfg``
   to another location:

.. code:: shell

       mv /usr/edb/pem/agent/etc/agent.cfg <new_location>

6. Then, configure the PEM server again:

.. code:: shell

       /usr/edb/pem/bin/configure-pem-server.sh

PEM web application not loading
-------------------------------

If the PEM web application isn’t loading, :ref:`check the web services are running <Managing a PEM server>`  . If they are
running, check the web server and application error logs.

If you are using NGINX as the web server, the web server logs are
located at ``/var/log/nginx/error.log`` and the application logs are at
``/var/log/edb-uwsgi/pem.log``

If you are using Apache HTTPD The log is located at
``/var/log/httpd/error_log`` . This file contains both web server and
application error messages.

If you are using Apache HTTPD as the web server, and see the following
message in the log:

.. code:: text

   Truncated or oversized response headers received from daemon process edbpem: /usr/edb/pem/web/pem.wsgi

Add the following statement at the bottom of the Apache ``httpd.conf``
file located in the ``/etc/httpd/conf`` folder:

.. code:: shell

   WSGIApplicationGroup %{GLOBAL}

Restart the HTTPD server after adding the statement:

.. code:: shell

   sudo systemctl restart httpd.service

Error connecting to PostgreSQL server
-------------------------------------

When connecting to a PostgreSQL server, you might get one of these error
messages. Review the message carefully. Each error message attempts to
incorporate the information you need to resolve the problem.

- **Connection to the server has been lost**: This error message
  indicates that the connection attempt took longer than the specified
  threshold. There might be a problem with the connection properties
  provided on the Server dialog box, network connectivity issues, or the
  server might not be running.

- **Could not connect to Server: Connection refused**: There are two
  possible reasons for this error:

  - The database server isn’t running. Start the server.
  - The server isn’t configured to accept TCP/IP requests on the address
    shown.

For security reasons, a PostgreSQL server “out of the box” doesn’t
listen on TCP/IP ports. Instead, you must enable it to listen for TCP/IP
requests. Add ``tcpip = true`` to the ``postgresql.conf`` file for
Versions 7.3.x and 7.4.x. Add ``listen_addresses='*'`` for Version 8.0.x
and above. These additions make the server accept connections on any IP
interface.

For more information, refer to the PostgreSQL documentation about
`runtime configuration <http://www.postgresql.org/docs/current/interactive/runtime-config.html>`_  .

- **FATAL: no pg_hba.conf entry**: If PEM displays this message when
  connecting, your server can be contacted correctly over the network,
  but it isn’t configured to accept your connection. Your client wasn’t
  detected as a legal user for the database.

To connect to a server, configure the ``pg_hba.conf`` file on the
database server to accept connections from the host of the PEM client.
Modify the ``pg_hba.conf`` file on the database server host, and add an
entry in the form:

- ``host template1 postgres 192.168.0.0/24 md5`` for an IPV4 network
- ``host template1 postgres ::ffff:192.168.0.0/120 md5`` for an IPV6
  network

For more information, see the PostgreSQL documentation about
`client authentication <http://www.postgresql.org/docs/current/interactive/client-authentication.html>`_  .

- **FATAL: password authentication failed**: The
  ``password authentication failed for user`` error message indicates
  there might be a problem with the password you entered. Retry the
  password to confirm you entered it correctly. If the error message
  returns, make sure that you have the correct password, that you are
  authorized to access the server, and that the access was correctly
  configured in the server’s ``postgresql.conf`` configuration file.

PEM web server connectivity check
---------------------------------

Run this command to check whether your client can reach the PEM server
and obtain a response:

.. code:: shell

   curl https://<SERVER_ADDR>:8443/pem/misc/ping -k -i
   __OUTPUT__
   PING

Where ``SERVER_ADDR`` is the IP address of your PEM server. The output
``PING`` confirms the PEM web server is up and running.
