Troubleshooting
===============

.. container:: registered_link
   :name: Troubleshooting

Server installation errors
--------------------------

RHEL 8
^^^^^^

1. While installing the PEM server on RHEL 8, if you see this error:

.. code:: text

   [root@etpgxlt firstuser]# dnf install edb-pem
   Updating Subscription Management repositories.
   Last metadata expiration check: 0:01:33 ago on Wed 30 Mar 2022 01:28:16 AM EDT.
   Error: 
    Problem: problem with installed package python3-mod_wsgi-4.6.4-4.el8.s390x
     - package python39-mod_wsgi-4.7.1-4.module+el8.4.0+9822+20bf1249.s390x conflicts with python3-mod_wsgi provided by python3-mod_wsgi-4.6.4-4.el8.s390x
     - package python39-mod_wsgi-4.7.1-4.module+el8.4.0+9822+20bf1249.s390x conflicts with python3-mod_wsgi provided by python3-mod_wsgi-4.6.4-3.el8.s390x
     - package edb-pem-server-8.4.0-7.rhel8.s390x requires python39-mod_wsgi >= 4.7, but none of the providers can be installed
     - package edb-pem-8.4.0-7.rhel8.s390x requires edb-pem-server = 8.4.0-7.rhel8, but none of the providers can be installed
     - cannot install the best candidate for the job
   (try to add --allowerasing to command line to replace conflicting packages or --skip-broken to skip uninstallable packages or --nobest to use not only best candidate packages)
   [root@etpgxlt firstuser]# 

Remove the ``python3-mod_wsgi`` package first:

.. code:: shell

   dnf remove python3-mod_wsgi

Try installing the PEM server again.

2. On RHEL 8, if you see this error in the worker.log after configuring
   the PEM server:

.. code:: text

   Tue Nov 28 03:02:19 2023 WARNING: Error clearing zombies: ERROR:  failed to JIT module: Added modules have incompatible data layouts: E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-a:8:16-n32:64 (module) vs E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-v128:64-a:8:16-n32:64 (jit)
   CONTEXT:  SQL statement "WITH running_agent_job AS (
           SELECT j.jobid, j.agent_id
           FROM pem.job j LEFT JOIN pem.joblog jl ON (j.jobid = jl.jlgjobid)
           WHERE jl.jlgstatus = r AND agent_id = $1 AND j.jobarid != agent_runtime_id
           ORDER BY j.jobid, jl.jlgjobid
           FOR UPDATE SKIP LOCKED
       ), joblog_status_update AS (
           UPDATE pem.joblog jl SET jlgstatus=d
           FROM running_agent_job r
           WHERE r.jobid = jl.jlgjobid AND jl.jlgstatus=r
           RETURNING r.agent_id, jl.jlgjobid, jl.jlgid
       ), jobsteplog_status_update AS (
           UPDATE pem.jobsteplog js SET jslstatus=d
           FROM joblog_status_update jl
           WHERE js.jsljlgid = jl.jlgid AND js.jslstatus=r
           RETURNING jl.agent_id, jl.jlgjobid AS job_id
       )
       UPDATE pem.job j SET jobprocessid=NULL, jobnextrun=NULL, jobarid=NULL
       FROM (SELECT DISTINCT agent_id, job_id FROM jobsteplog_status_update) js
       WHERE js.job_id = j.jobid"
   PL/pgSQL function pem.clear_job_zombies(integer) line 8 at SQL statement

To resolve the error, set the ``jit`` parameter to ``off`` in
``postgresql.conf`` file of the backend database server:

.. code:: ini

     jit=off

Restart the backend database server.

Reconfiguring the PEM server
----------------------------

In some situations, you might need to uninstall the PEM server,
reinstall it, and then configure the server again. To do so:

1. Remove the PEM server configuration and uninstall:

.. code:: shell

       /usr/edb/pem/bin/configure-pem-server.sh –un

2. Remove the PEM packages:

.. code:: shell

       yum erase edb-pem-server

3. Drop the ``pem`` database:

.. code:: sql

       DROP DATABASE pem

4. Move the certificates from ``/root/.pem/`` to another location:

.. code:: shell

       mv /root/.pem/* <new_location>

5. Move the ``agent.cfg`` file from ``/usr/edb/pem/agent/etc/agent.cfg``
   to another location:

.. code:: shell

       mv /usr/edb/pem/agent/etc/agent.cfg <new_location>

6. Then, configure the PEM server again:

.. code:: shell

       /usr/edb/pem/bin/configure-pem-server.sh

PEM web client not loading
--------------------------

If the PEM web client isn’t loading, check the HTTPD log in the
``/var/log/httpd/error_log`` file.

If you see the following message in the log:

.. code:: text

   Truncated or oversized response headers received from daemon process edbpem: /usr/edb/pem/web/pem.wsgi

Add the following statement at the bottom of the Apache ``httpd.conf``
file located in the ``/etc/httpd/conf`` folder:

.. code:: shell

   WSGIApplicationGroup %{GLOBAL}

Restart the HTTPD server after adding the statement:

.. code:: shell

   sudo systemctl restart httpd.service
   ```  ## Error connecting to PostgreSQL server

   When connecting to a PostgreSQL server, you might get one of these error messages. Review the message carefully. Each error message attempts to incorporate the information you need to resolve the problem.

   -  **Connection to the server has been lost**: This error message indicates that the connection attempt took longer than the specified threshold. There might be a problem with the connection properties provided on the Server dialog box, network connectivity issues, or the server might not be running. 

   -  **Could not connect to Server: Connection refused**: There are two possible reasons for this error:  

      -    The database server isn't running. Start the server.
      -    The server isn't configured to accept TCP/IP requests on the address shown. 

   For security reasons, a PostgreSQL server "out of the box" doesn't listen on TCP/IP ports. Instead, you must enable it to listen for TCP/IP requests. Add `tcpip = true`  to the `postgresql.conf`  file for Versions 7.3.x and 7.4.x. Add `listen_addresses='*'`  for Version 8.0.x and above. These additions make the server accept connections on any IP interface.

   For more information, refer to the PostgreSQL documentation about  `runtime configuration <http://www.postgresql.org/docs/current/interactive/runtime-config.html>`_  .

   -  **FATAL: no pg_hba.conf entry**: If PEM displays this message when connecting, your server can be contacted correctly over the network, but it isn't configured to accept your connection. Your client wasn't detected as a legal user for the database. 

   To connect to a server, configure the `pg_hba.conf`  file on the database server to accept connections from the host of the PEM client. Modify the `pg_hba.conf`  file on the database server host, and add an entry in the form:

      -    `host template1 postgres 192.168.0.0/24 md5`  for an IPV4 network
      -    `host template1 postgres ::ffff:192.168.0.0/120 md5`  for an IPV6 network

   For more information, see the PostgreSQL documentation about  `client authentication <http://www.postgresql.org/docs/current/interactive/client-authentication.html>`_  .

   -  **FATAL: password authentication failed**: The `password authentication failed for user`  error message indicates there might be a problem with the password you entered. Retry the password to confirm you entered it correctly. If the error message returns, make sure that you have the correct password, that you are authorized to access the server, and that the access was correctly configured in the server's `postgresql.conf`  configuration file. 

   ##  PEM web server status check

   Run this command to check whether the PEM webserver is up and running:

   ```shell
   curl https://<SERVER_ADDR>:8443/pem/misc/ping -k -i
   __OUTPUT__
   PING

Where ``SERVER_ADDR`` is the IP address of your PEM server. The output
``PING`` confirms the PEM web server is up and running.
