Troubleshooting

Agent Test

To check whether or not the agent is able to send WireGuard data to the Pro Custodibus servers, run the following command on the host:

$ sudo /opt/venvs/procustodibus-agent/bin/procustodibus-agent --test
... 1 wireguard interfaces found ...
... 192.0.2.3 is pro custodibus ip address ...
... healthy pro custodibus api ...
... can access host record on api for My Test Host ...
All systems go :)

(To run this test command on Windows, from the Start menu, right-click the Pro Custodibus Agent > Pro Custodibus Agent Test menu item, and select Run as Administrator.)

Make sure you run this command as root. If the output ends All systems go, then it should be good. Try restarting the agent as directed by the Agent Status instructions.

Missing conf

If the output of the test is Missing conf for Agent ID and Host ID, then either:

  1. You did not run the test as root; try re-running the test as root.

  2. The agent configuration file is missing required settings (or missing altogether); try re-downloading the configuration file as directed by the Set Up Host instructions.

  3. The agent configuration file or directory has the wrong SELinux context label.

  4. The agent configuration file or directory does not have the correct permissions.

If the agent configuration file is present at /etc/wireguard/procustodibus.conf, and you examine it, it should look similar to the following:

$ sudo cat /etc/wireguard/procustodibus.conf
# procustodibus.conf generated 1/1/2020, 12:34:56 PM PDT
[Procustodibus]
# Alice's Laptop Agent
Agent = ABC123def45
# Alice's Laptop
Host = DEF456ghi78

And if you view the permissions on the /etc/wireguard/ directory, as well as the files within it, the owner’s permission bits should look like the following:

$ sudo ls -ld /etc/wireguard
drwx------ 2 root root 4096 May 25 21:39 /etc/wireguard
$ sudo ls -l /etc/wireguard
total 12
-rw-r--r-- 1 root root 181 May 27 22:52 procustodibus.conf
-rw-r----- 1 root root 275 May 27 22:52 procustodibus-credentials.conf
-rw-r----- 1 root root 554 May 25 21:43 wg0.conf

Missing code

If the output of the test includes Missing code in setup file, then the agent setup file is missing required settings; try re-downloading the setup file as directed by the Set Up Host instructions.

If you examine the agent setup file, it should look similar to the following:

$ sudo cat /etc/wireguard/procustodibus-setup.conf
# procustodibus-setup.conf generated 1/1/2020, 12:34:56 PM PDT
[Procustodibus.Setup]
# Alice's Laptop Agent
Agent = ABC123def45
Code = GHI789JKL01
Expires = 2020-01-03T20:34:56Z

Setup code has expired

If the output of the test includes Setup code has expired, then the code in the agent setup file has expired; download a new setup file as directed by the Set Up Host instructions.

Could not read credentials

If the output of the test includes Could not read credentials file, then the agent credentials file is missing; download a new setup file as directed by the Set Up Host instructions (a new setup file will allow the agent to generate a new credentials file).

Missing private key

If the output of the test includes Missing private key in credentials file, then the agent credentials file is missing required settings; try downloading a new setup file as directed by the Set Up Host instructions (a new setup file will allow the agent to generate a new credentials file).

If you examine the agent credentials file, it should look similar to the following:

$ sudo cat /etc/wireguard/procustodibus-credentials.conf
# procustodibus-credentials.conf generated 2020-01-01T20:34:56Z
# for agent ABC123def45 on a123.corp.example.com
[Procustodibus.Credentials]
PublicKey = O2onvM62pC1io6jQKm8Nc2UyFXcd4kOmOsBIoYtZ2ik=
PrivateKey = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

No wg executable

If the output of the command includes no wg executable found, then you may not have installed WireGuard on the host yet. Try installing WireGuard and adding a WireGuard interface.

No wireguard interfaces

If the output of the command includes no wireguard interfaces found, then you may not have configured any WireGuard interfaces on the host yet. Try adding one.

DNS error

If the output of the command includes dns error resolving api.custodib.us, then the agent is not able to resolve the Pro Custodibus servers' DNS entries. Access to the public DNS servers on which the agent relies by default may be blocked.

Try accessing the Quad9 DoH servers from the host:

$ curl -I https://dns.quad9.net/
HTTP/2 404
server: h2o/dnsdist
date: Thu, 26 Sep 2024 02:27:25 GMT
content-type: text/plain; charset=utf-8
content-length: 9

If this hangs or fails, you either need to open up access to the Quad9 servers, or provide the agent another way to resolve DNS queries.

To provide the agent with a private DNS server (for example, one at 192.168.123.53), use the following settings in the procustodibus.conf configuration file:

Dns = 192.168.123.53
DnsProtocol = udp

To rely on the host’s own network stack for DNS resolution, use the following setting in the procustodibus.conf configuration file:

Dns = off

To prefer the use of IPv4 addresses for DNS resolution, add the following setting to the procustodibus.conf configuration file:

ResolveHostnames = ipv4

Server unavailable

If the output of the command includes server unavailable, check whether or not the output also includes dns error. If it does, see the DNS error issue above.

If the output does not include dns error, the output will include the IP address of your active Pro Custodibus API endpoint in a phrase like 192.0.2.3 is pro custodibus ip address (the IP address will be something other than 192.0.2.3, however). Try running the following command in a terminal on the host (replacing 192.0.2.3 with the actual IP address from the output):

$ ping -c1 192.0.2.3
PING 192.0.2.3 (192.0.2.3): 56 data bytes
64 bytes from 192.0.2.3: icmp_seq=0 ttl=37 time=10.138 ms
--- 192.0.2.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 10.138/10.138/10.138/0.000 ms

If the ping command hangs or the output of the ping command ends with 100% packet loss, network access from the host to the Pro Custodibus servers may be blocked. You may need to update your firewall or other networking settings to allow the host to access the Pro Custodibus servers.

If the output of the ping command includes 0% packet loss, your connection to the Pro Custodibus servers is good. The Pro Custodibus servers may be down; email support@procustodibus.com to check.

Unhealthy pro custodibus api

If the output of the command includes unhealthy pro custodibus api, then there is a problem with the Pro Custodibus servers; email support@procustodibus.com for details.

Cannot access host record

If the output of the command includes cannot access host record, then the agent does not have permission to access the host’s record on Pro Custodibus. Either:

  1. The host record on Pro Custodibus has been deleted

  2. The agent record on Pro Custodibus has been deleted

  3. The agent’s credentials for Pro Custodibus have been revoked

  4. The agent’s access to the host on Pro Custodibus has been revoked

  5. The host’s clock is not accurate (the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate)

If the host’s clock is off by more than a few minutes, fix the clock and restart the agent. Otherwise, try re-downloading the procustodibus.conf and procustodibus-setup.conf files as directed by the Set Up Host instructions, and then restart the agent as directed by the Agent Status instructions.

Agent Status

To check the status of the agent on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ systemctl status procustodibus-agent
● procustodibus-agent.service - Pro Custodibus Agent
     Loaded: loaded (/etc/systemd/system/procustodibus-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-07-09 12:45:36 PDT; 17s ago
   Main PID: 2770504 (procustodibus-a)
      Tasks: 1 (limit: 38335)
     Memory: 15.0M
     CGroup: /system.slice/procustodibus-agent.service
             └─2770504 /opt/venvs/procustodibus-agent/bin/python3 /opt/venvs/procustodibus-agent/bin/procustodibus-agent --loop=60

Jul 09 12:45:36 localhost systemd[1]: Started Pro Custodibus Agent.

If the agent is running, this command output should include active (running). Otherwise, try starting the agent with this command:

$ sudo systemctl start procustodibus-agent

Substitute restart for start to restart an already running agent.

Linux With OpenRC

Run the following command in a terminal on the host:

$ rc-service procustodibus-agent status
procustodibus-agent started

If the agent is running, this command output should end with started. Otherwise, try starting the agent with this command:

$ sudo rc-service procustodibus-agent start

Substitute restart for start to restart an already running agent.

Linux With Sysvinit

Run the following command in a terminal on the host:

$ /etc/init.d/procustodibus-agent status
started

If the agent is running, this command should output started. Otherwise, try starting the agent with this command:

$ sudo /etc/init.d/procustodibus-agent start
starting Pro Custodibus Agent ...

Substitute restart for start to restart an already running agent.

FreeBSD

Run the following command in a terminal on the host as root:

# service procustodibus-agent status
procustodibus_agent is running as pid 1234.

If the agent is running, this command output should contain is running. Otherwise, try starting the agent with this command as root:

# service procustodibus-agent start
Starting procustodibus_agent.

Substitute restart for start to restart an already running agent.

macOS

Run the following command in a terminal on the host:

% launchctl print system/procustodibus-agent
system/procustodibus-agent = {
        active count = 1
        path = /Library/LaunchDaemons/procustodibus-agent.plist
        type = LaunchDaemon
        state = running
...

If the agent is running, this command output should contain state = running. Otherwise, try starting the agent with this command as root:

% sudo launchctl kickstart system/procustodibus-agent

Windows

Run the following command in a command prompt:

> sc query ProCustodibusAgentService

  SERVICE_NAME: ProCustodibusAgentService
          TYPE               : 10  WIN32_OWN_PROCESS
          STATE              : 4  RUNNING
                                  (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
          WIN32_EXIT_CODE    : 0  (0x0)
          SERVICE_EXIT_CODE  : 0  (0x0)
          CHECKPOINT         : 0x0
          WAIT_HINT          : 0x0

If the agent is running, this command output should contain STATE : 4 RUNNING. Otherwise, try starting the agent with this command as Administrator:

> sc start ProCustodibusAgentService

  SERVICE_NAME: ProCustodibusAgentService
          TYPE               : 10  WIN32_OWN_PROCESS
          STATE              : 2  START_PENDING
                                  (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
          WIN32_EXIT_CODE    : 0  (0x0)
          SERVICE_EXIT_CODE  : 0  (0x0)
          CHECKPOINT         : 0x0
          WAIT_HINT          : 0x7d0
          PID                : 1234
          FLAGS

Run the stop command and then the start command to restart an already running agent.

Agent Logs

To check the log files of the agent on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ journalctl -u procustodibus-agent

Linux With OpenRC

Run the following command in a terminal on the host:

$ less /var/log/procustodibus-agent.log

Also check the error log if present:

$ less /var/log/procustodibus-agent.err

Linux With Sysvinit

Run the following command in a terminal on the host:

$ less /var/log/procustodibus-agent.log

FreeBSD

Run the following command in a terminal on the host, then search for procustodibus-agent:

# less /var/log/messages
/procustodibus-agent

macOS

Run the following command in a terminal on the host:

% less /var/log/procustodibus-agent.log

Also check the error log if present:

% less /var/log/procustodibus-agent.err

Windows

Open up the following file with Notepad:

> notepad "C:\Program Files\Pro Custodibus Agent\log\stdout.log"

Also check the following log files (which normally contain only a few messages like “starting logging at level X” and “service config is Y”):

> notepad "C:\Program Files\Pro Custodibus Agent\log\init.log"
> notepad "C:\Program Files\Pro Custodibus Agent\ProCustodibusAgentService.log"

Log Messages

Unauthorized for url

If the agent log ends with the following error message:

401 Client Error: Unauthorized for url: https://api.custodib.us/users/ABC123def45/credentials/signature

Try re-downloading the procustodibus.conf and procustodibus-setup.conf files as directed by the Set Up Host instructions, and then restart the agent as directed by the Agent Status instructions.

Also, check the host’s clock drift — the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate.

No module named procustodibus_agent

If after upgrading the OS on a host you see the following error message in the agent logs:

ModuleNotFoundError: No module named 'procustodibus_agent'

This probably means the OS is now using a newer version of Python than the agent was originally installed under.

To fix, download the latest version of the agent, and reinstall it with the --install flag:

$ cd procustodibus-agent-*/
$ sudo ./install.sh --install
running as root
install 1.3.1
agent configuration found at /etc/wireguard/procustodibus.conf
agent credentials found at /etc/wireguard/procustodibus-credentials.conf
/etc/wireguard/procustodibus.conf mode ok (-rw-rw-r--)
/etc/wireguard/procustodibus.conf owner is root
/etc/wireguard/procustodibus.conf group is root
/etc/wireguard/procustodibus-credentials.conf mode ok (-rw-r-----)
/etc/wireguard/procustodibus-credentials.conf owner is root
/etc/wireguard/procustodibus-credentials.conf group is root
libsodium found at /lib/aarch64-linux-gnu/libsodium.so.23
python 3.10.0 found at /usr/bin/python3
python includes all packages needed for venv
WARNING python virtualenv broken at /opt/venvs/procustodibus-agent
recreate virtualenv? ([y]es, [q]uit): y
OK will delete virtualenv
deleted virtualenv
OK will create virtualenv
created virtualenv
WARNING agent package not installed
install agent from /root/procustodibus-agent-1.3.1? ([y]es, [q]uit): y
OK will install agent
...
installed agent
systemd daemon found at /etc/systemd/system/procustodibus-agent.service
WARNING daemon failed
start daemon? ([y]es, [q]uit): y
OK will start daemon
started daemon
install SUCCESS

SELinux

If you’re using SELinux, try running the ls -lZ command to check the context labels for the files in the /etc/wireguard/ directory. If the files are labeled correctly, you should see output similar to the following:

$ sudo ls -lZ /etc/wireguard
total 12
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0 181 May 27 22:52 procustodibus.conf
-rw-r-----. 1 root root system_u:object_r:etc_t:s0 275 May 27 22:52 procustodibus-credentials.conf
-rw-r-----. 1 root root system_u:object_r:etc_t:s0 554 May 25 21:43 wg0.conf

If not, try running the restorecon command to relabel them:

$ sudo restorecon -Fv /etc/wireguard/*
Relabeled /etc/wireguard/procustodibus-credentials.conf from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:etc_t:s0
Relabeled /etc/wireguard/procustodibus.conf from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:etc_t:s0