Troubleshooting

Agent Test

To check whether or not the agent is able to send WireGuard data to the Pro Custodibus servers, run the following command on the host:

$ sudo /opt/venvs/procustodibus-agent/bin/procustodibus-agent --test
... 1 wireguard interfaces found ...
... 192.0.2.3 is pro custodibus ip address ...
... healthy pro custodibus api ...
... can access host record on api for My Test Host ...
All systems go :)

Make sure you run this command as root. If the output ends All systems go, then it should be good. Try restarting the agent as directed by the Agent Status instructions.

Missing conf

If the output of the test is Missing conf for Agent ID and Host ID, then either:

  1. You did not run the test as root; try re-running the test as root.

  2. The agent configuration file is missing required settings (or missing altogether); try re-downloading the configuration file as directed by the Set Up Host instructions.

  3. The agent configuration file has the wrong SELinux context label.

If the agent configuration file is present at /etc/wireguard/procustodibus.conf, and you examine it, it should look similar to the following:

$ sudo cat /etc/wireguard/procustodibus.conf
# procustodibus.conf generated 1/1/2020, 12:34:56 PM PDT
[Procustodibus]
# Alice's Laptop Agent
Agent = ABC123def45
# Alice's Laptop
Host = DEF456ghi78

Missing code

If the output of the test includes Missing code in setup file, then the agent setup file is missing required settings; try re-downloading the setup file as directed by the Set Up Host instructions.

If you examine the agent setup file, it should look similar to the following:

$ sudo cat /etc/wireguard/procustodibus-setup.conf
# procustodibus-setup.conf generated 1/1/2020, 12:34:56 PM PDT
[Procustodibus.Setup]
# Alice's Laptop Agent
Agent = ABC123def45
Code = GHI789JKL01
Expires = 2020-01-03T20:34:56Z

Setup code has expired

If the output of the test includes Setup code has expired, then the code in the agent setup file has expired; download a new setup file as directed by the Set Up Host instructions.

Missing credentials

If the output of the test ends with No such file or directory: '/etc/wireguard/procustodibus-credentials.conf', then the agent credentials file is missing; download a new setup file as directed by the Set Up Host instructions (a new setup file will allow the agent to generate a new credentials file).

Missing private key

If the output of the test includes Missing private key in credentials file, then the agent credentials file is missing required settings; try downloading a new setup file as directed by the Set Up Host instructions (a new setup file will allow the agent to generate a new credentials file).

If you examine the agent credentials file, it should look similar to the following:

$ sudo cat /etc/wireguard/procustodibus-credentials.conf
# procustodibus-credentials.conf generated 2020-01-01T20:34:56Z
# for agent ABC123def45 on a123.corp.example.com
[Procustodibus.Credentials]
PublicKey = O2onvM62pC1io6jQKm8Nc2UyFXcd4kOmOsBIoYtZ2ik=
PrivateKey = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

No wg executable

If the output of the command includes no wg executable found, then you may not have installed WireGuard on the host yet. Try installing WireGuard and adding a WireGuard interface.

No wireguard interfaces

If the output of the command includes no wireguard interfaces found, then you may not have configured any WireGuard interfaces on the host yet. Try adding one.

Cannot lookup ip address

If the output of the command includes cannot lookup ip address, then the host may not be able to resolve the Pro Custodibus servers' DNS entries. Make sure that:

  1. The host’s DNS resolver is working

  2. The host’s DNS resolver can resolve public DNS entries

Try running the following command in a terminal on the host:

$ dig +short api.custodib.us
192.0.2.3

If the dig command outputs nothing, likely either a) or b) above is not true.

Server unavailable

If the output of the command includes server unavailable, check whether or not the output also includes cannot lookup ip address. If it does, see the Cannot lookup ip address issue above.

If the output does not include cannot lookup ip address, the output will include the IP address of your active Pro Custodibus API endpoint in a phrase like 192.0.2.3 is pro custodibus ip address (the IP address will be something other than 192.0.2.3, however). Try running the following command in a terminal on the host (replacing 192.0.2.3 with the actual IP address from the output):

$ ping -c1 192.0.2.3
PING 192.0.2.3 (192.0.2.3): 56 data bytes
64 bytes from 192.0.2.3: icmp_seq=0 ttl=37 time=10.138 ms
--- 192.0.2.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 10.138/10.138/10.138/0.000 ms

If the ping command hangs or the output of the ping command ends with 100% packet loss, network access from the host to the Pro Custodibus servers may be blocked. You may need to update your firewall or other networking settings to allow the host to access the Pro Custodibus servers.

If the output of the ping command includes 0% packet loss, your connection to the Pro Custodibus servers is good. The Pro Custodibus servers may be down; email support@procustodibus.com to check.

Unhealthy pro custodibus api

If the output of the command includes unhealthy pro custodibus api, then there is a problem with the Pro Custodibus servers; email support@procustodibus.com for details.

Cannot access host record

If the output of the command includes cannot access host record, then the agent does not have permission to access the host’s record on Pro Custodibus. Either:

  1. The host record on Pro Custodibus has been deleted

  2. The agent record on Pro Custodibus has been deleted

  3. The agent’s credentials for Pro Custodibus have been revoked

  4. The agent’s access to the host on Pro Custodibus has been revoked

  5. The host’s clock is not accurate (the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate)

If the host’s clock is off by more than a few minutes, fix the clock and restart the agent. Otherwise, try re-downloading the procustodibus.conf and procustodibus-setup.conf files as directed by the Set Up Host instructions, and then restart the agent as directed by the Agent Status instructions.

Agent Status

To check the status of the agent on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ systemctl status procustodibus-agent
● procustodibus-agent.service - Pro Custodibus Agent
     Loaded: loaded (/etc/systemd/system/procustodibus-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-07-09 12:45:36 PDT; 17s ago
   Main PID: 2770504 (procustodibus-a)
      Tasks: 1 (limit: 38335)
     Memory: 15.0M
     CGroup: /system.slice/procustodibus-agent.service
             └─2770504 /opt/venvs/procustodibus-agent/bin/python3 /opt/venvs/procustodibus-agent/bin/procustodibus-agent --loop=60

Jul 09 12:45:36 localhost systemd[1]: Started Pro Custodibus Agent.

If the agent is running, this command output should include active (running). Otherwise, try starting the agent with this command:

$ sudo systemctl start procustodibus-agent

Substitute restart for start to restart an already running agent.

Linux With OpenRC

Run the following command in a terminal on the host:

$ rc-service procustodibus-agent status
procustodibus-agent started

If the agent is running, this command output should end with started. Otherwise, try starting the agent with this command:

$ sudo rc-service procustodibus-agent start

Substitute restart for start to restart an already running agent.

Linux With Sysvinit

Run the following command in a terminal on the host:

$ /etc/init.d/procustodibus-agent status
started

If the agent is running, this command should output started. Otherwise, try starting the agent with this command:

$ sudo /etc/init.d/procustodibus-agent start
starting Pro Custodibus Agent ...

Substitute restart for start to restart an already running agent.

FreeBSD

Run the following command in a terminal on the host as root:

# service procustodibus-agent status
procustodibus_agent is running as pid 1234.

If the agent is running, this command output should contain is running. Otherwise, try starting the agent with this command as root:

# service procustodibus-agent start
Starting procustodibus_agent.

Substitute restart for start to restart an already running agent.

Agent Logs

To check the log files of the agent on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ journalctl -u procustodibus-agent

Linux With OpenRC

Run the following command in a terminal on the host:

$ less /var/log/procustodibus-agent.log

Also check the error log if present:

$ less /var/log/procustodibus-agent.err

Linux With Sysvinit

Run the following command in a terminal on the host:

$ less /var/log/procustodibus-agent.log

FreeBSD

Run the following command in a terminal on the host, then search for procustodibus-agent:

$ less /var/log/messages
/procustodibus-agent

Log Messages

Unauthorized for url

If the agent log ends with the following error message:

401 Client Error: Unauthorized for url: https://api.custodib.us/users/ABC123def45/credentials/signature

Try re-downloading the procustodibus.conf and procustodibus-setup.conf files as directed by the Set Up Host instructions, and then restart the agent as directed by the Agent Status instructions.

Also, check the host’s clock drift — the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate.

No module named procustodibus_agent

If after upgrading the OS on a host you see the following error message in the agent logs:

ModuleNotFoundError: No module named 'procustodibus_agent'

This probably means the OS is now using a newer version of Python than the agent was originally installed under.

To fix, download the latest version of the agent, and reinstall it with the --install flag:

$ cd procustodibus-agent-*/
$ sudo ./install.sh --install
running as root
install 1.3.1
agent configuration found at /etc/wireguard/procustodibus.conf
agent credentials found at /etc/wireguard/procustodibus-credentials.conf
/etc/wireguard/procustodibus.conf mode ok (-rw-rw-r--)
/etc/wireguard/procustodibus.conf owner is root
/etc/wireguard/procustodibus.conf group is root
/etc/wireguard/procustodibus-credentials.conf mode ok (-rw-r-----)
/etc/wireguard/procustodibus-credentials.conf owner is root
/etc/wireguard/procustodibus-credentials.conf group is root
libsodium found at /lib/aarch64-linux-gnu/libsodium.so.23
python 3.10.0 found at /usr/bin/python3
python includes all packages needed for venv
WARNING python virtualenv broken at /opt/venvs/procustodibus-agent
recreate virtualenv? ([y]es, [q]uit): y
OK will delete virtualenv
deleted virtualenv
OK will create virtualenv
created virtualenv
WARNING agent package not installed
install agent from /root/procustodibus-agent-1.3.1? ([y]es, [q]uit): y
OK will install agent
...
installed agent
systemd daemon found at /etc/systemd/system/procustodibus-agent.service
WARNING daemon failed
start daemon? ([y]es, [q]uit): y
OK will start daemon
started daemon
install SUCCESS

SELinux

If you’re using SELinux, try running the ls -lZ command to check the context labels for the files in the /etc/wireguard directory. If the files are labeled correctly, you should see output similar to the following:

$ sudo ls -lZ /etc/wireguard
total 12
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0 181 May 27 22:52 procustodibus.conf
-rw-r-----. 1 root root system_u:object_r:etc_t:s0 275 May 27 22:52 procustodibus-credentials.conf
-rw-r-----. 1 root root system_u:object_r:etc_t:s0 554 May 25 21:43 wg0.conf

If not, try running the restorecon command to relabel them:

$ sudo restorecon -Fv /etc/wireguard/*
Relabeled /etc/wireguard/procustodibus-credentials.conf from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:etc_t:s0
Relabeled /etc/wireguard/procustodibus.conf from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:etc_t:s0