Troubleshooting

Agent Test

To check whether or not the agent is able to send WireGuard data to the Pro Custodibus servers, run the following command on the host:

$ sudo /opt/venvs/procustodibus-agent/bin/procustodibus-agent --test
... 1 wireguard interfaces found ...
... 192.0.2.0 is pro custodibus ip address ...
... healthy pro custodibus api ...
... can access host record on api for My Test Host ...
All systems go :)

Make sure you run this command as root. If the output ends All systems go, then it should be good. Try re-starting the agent as directed by the Agent Status instructions.

Missing conf

If the output of the test is Missing conf for Agent ID and Host ID, then either:

  1. You did not run the test as root; try re-running the test as root.

  2. The agent configuration file is missing required settings (or missing altogether); try re-downloading the configuration file as directed by the Set Up Host instructions.

Missing credentials

If the output of the test ends with No such file or directory: '/etc/wireguard/procustodibus-credentials.conf', then try re-downloading the setup file as directed by the Set Up Host instructions.

No wg executable

If the output of the command includes no wg executable found, you may not have installed WireGuard on the host yet. Try installing WireGuard and adding a WireGuard interface.

No wireguard interfaces

If the output of the command includes no wireguard interfaces found, you may not have configured any WireGuard interfaces on the host yet. Try adding one.

Cannot lookup ip address

If the output of the command includes cannot lookup ip address, then the host may not be able to resolve the Pro Custodibus servers' DNS entries. Make sure that:

  1. The host’s DNS resolver is working

  2. The host’s DNS resolver can resolve public DNS entries

Try running the following command in a terminal on the host:

$ dig +short api.custodib.us
192.0.2.0

If the dig command outputs nothing, likely either a) or b) above is not true.

Server unavailable

If the output of the command includes server unavailable, check whether or not the output also includes cannot lookup ip address. If it does, see the Cannot lookup ip address issue above.

If the output does not include cannot lookup ip address, the output will include the IP address of your active Pro Custodibus API endpoint in a phrase like 192.0.2.0 is pro custodibus ip address (the IP address will be something other than 192.0.2.0, however). Try running the following command in a terminal on the host (replacing 192.0.2.0 with the actual IP address from the output):

$ ping -c1 192.0.2.0
PING 192.0.2.0 (192.0.2.0): 56 data bytes
64 bytes from 192.0.2.0: icmp_seq=0 ttl=37 time=10.138 ms
--- 192.0.2.0 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 10.138/10.138/10.138/0.000 ms

If the ping command hangs or the output of the ping command ends with 100% packet loss, network access from the host to the Pro Custodibus servers may be blocked. You may need to update your firewall or other networking settings to allow the host to access the Pro Custodibus servers.

If the output of the ping command includes 0% packet loss, your connection to the Pro Custodibus servers is good. The Pro Custodibus servers may be down; email support@procustodibus.com to check.

Unhealthy pro custodibus api

If the output of the command includes unhealthy pro custodibus api, there is a problem with the Pro Custodibus servers; email support@procustodibus.com for details.

Cannot access host record

If the output of the command includes cannot access host record, the agent does not has permission to access the host’s record on Pro Custodibus. Either:

  1. The host record on Pro Custodibus has been deleted

  2. The agent record on Pro Custodibus has been deleted

  3. The agent’s credentials for Pro Custodibus have been revoked

  4. The agent’s access to the host on Pro Custodibus has been revoked

Try re-downloading the procustodibus.conf and procustodibus-setup.conf files as directed by the Set Up Host instructions, and then re-start the agent as directed by the Agent Status instructions.

Agent Status

To check the status of the agent on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ systemctl status procustodibus-agent
● procustodibus-agent.service - Pro Custodibus Agent
     Loaded: loaded (/etc/systemd/system/procustodibus-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-07-09 12:45:36 PDT; 17s ago
   Main PID: 2770504 (procustodibus-a)
      Tasks: 1 (limit: 38335)
     Memory: 15.0M
     CGroup: /system.slice/procustodibus-agent.service
             └─2770504 /opt/venvs/procustodibus-agent/bin/python3 /opt/venvs/procustodibus-agent/bin/procustodibus-agent --loop=60

Jul 09 12:45:36 localhost systemd[1]: Started Pro Custodibus Agent.

If the agent is running, this command output should include active (running). Otherwise, try restarting the agent with this command:

$ sudo systemctl start procustodibus-agent

Substitute restart for start to re-start an already running agent.

Linux With OpenRC

Run the following command in a terminal on the host:

$ rc-service procustodibus-agent status
procustodibus-agent started

If the agent is running, this command output should end with started. Otherwise, try restarting the agent with this command:

$ sudo rc-service procustodibus-agent start

Substitute restart for start to re-start an already running agent.

Linux With Sysvinit

Run the following command in a terminal on the host:

$ /etc/init.d/procustodibus-agent status
started

If the agent is running, this command should output started. Otherwise, try restarting the agent with this command:

$ sudo /etc/init.d/procustodibus-agent start
starting Pro Custodibus Agent ...

Substitute restart for start to re-start an already running agent.

FreeBSD

Run the following command in a terminal on the host as root:

# service procustodibus-agent status
procustodibus_agent is running as pid 1234.

If the agent is running, this command output should contain is running. Otherwise, try restarting the agent with this command as root:

# service procustodibus-agent start
Starting procustodibus_agent.

Substitute restart for start to re-start an already running agent.

Agent Logs

To check the log files of the agent on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ journalctl -u procustodibus-agent

Linux With OpenRC

Run the following command in a terminal on the host:

$ less /var/log/procustodibus-agent.log

Also check the error log if present:

$ less /var/log/procustodibus-agent.err

Linux With Sysvinit

Run the following command in a terminal on the host:

$ less /var/log/procustodibus-agent.log

FreeBSD

Run the following command in a terminal on the host, then search for procustodibus-agent:

$ less /var/log/messages
/procustodibus-agent

Log Messages

Unauthorized for url

If the agent log ends with the following error message:

401 Client Error: Unauthorized for url: https://api.custodib.us/users/abcdef12345/credentials/signature

Try re-downloading the procustodibus.conf and procustodibus-setup.conf files as directed by the Set Up Host instructions, and then re-start the agent as directed by the Agent Status instructions.

Also, check the host’s clock drift — the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate.

No module named procustodibus_agent

If after upgrading the OS on a host you see the following error message in the agent logs:

ModuleNotFoundError: No module named 'procustodibus_agent'

This probably means the OS is now using a newer version of Python than the agent was originally installed under.

To fix, download the latest version of the agent, and reinstall it with the --install flag:

$ cd procustodibus-agent-*/
$ sudo ./install.sh --install
running as root
install 1.3.1
agent configuration found at /etc/wireguard/procustodibus.conf
agent credentials found at /etc/wireguard/procustodibus-credentials.conf
/etc/wireguard/procustodibus.conf mode ok (-rw-rw-r--)
/etc/wireguard/procustodibus.conf owner is root
/etc/wireguard/procustodibus.conf group is root
/etc/wireguard/procustodibus-credentials.conf mode ok (-rw-r-----)
/etc/wireguard/procustodibus-credentials.conf owner is root
/etc/wireguard/procustodibus-credentials.conf group is root
libsodium found at /lib/aarch64-linux-gnu/libsodium.so.23
python 3.10.0 found at /usr/bin/python3
python includes all packages needed for venv
WARNING python virtualenv broken at /opt/venvs/procustodibus-agent
recreate virtualenv? ([y]es, [q]uit): y
OK will delete virtualenv
deleted virtualenv
OK will create virtualenv
created virtualenv
WARNING agent package not installed
install agent from /root/procustodibus-agent-1.3.1? ([y]es, [q]uit): y
OK will install agent
...
installed agent
systemd daemon found at /etc/systemd/system/procustodibus-agent.service
WARNING daemon failed
start daemon? ([y]es, [q]uit): y
OK will start daemon
started daemon
install SUCCESS