Troubleshooting
Agent Test
To check whether or not the agent is able to send WireGuard data to the Pro Custodibus servers, run the following command on the host:
$ sudo /opt/venvs/procustodibus-agent/bin/procustodibus-agent --test ... 1 wireguard interfaces found ... ... 192.0.2.0 is pro custodibus ip address ... ... healthy pro custodibus api ... ... can access host record on api for My Test Host ... All systems go :)
Make sure you run this command as root. If the output ends All systems go
, then it should be good. Try re-starting the agent as directed by the Agent Status instructions.
Missing conf
If the output of the test is Missing conf for Agent ID and Host ID
, then either:
-
You did not run the test as root; try re-running the test as root.
-
The agent configuration file is missing required settings (or missing altogether); try re-downloading the configuration file as directed by the Set Up Host instructions.
Missing credentials
If the output of the test ends with No such file or directory: '/etc/wireguard/procustodibus-credentials.conf'
, then try re-downloading the setup file as directed by the Set Up Host instructions.
No wg executable
If the output of the command includes no wg executable found
, you may not have installed WireGuard on the host yet. Try installing WireGuard and adding a WireGuard interface.
No wireguard interfaces
If the output of the command includes no wireguard interfaces found
, you may not have configured any WireGuard interfaces on the host yet. Try adding one.
Cannot lookup ip address
If the output of the command includes cannot lookup ip address
, then the host may not be able to resolve the Pro Custodibus servers' DNS entries. Make sure that:
-
The host’s DNS resolver is working
-
The host’s DNS resolver can resolve public DNS entries
Try running the following command in a terminal on the host:
$ dig +short api.custodib.us 192.0.2.0
If the dig
command outputs nothing, likely either a) or b) above is not true.
Server unavailable
If the output of the command includes server unavailable
, check whether or not the output also includes cannot lookup ip address
. If it does, see the Cannot lookup ip address issue above.
If the output does not include cannot lookup ip address
, the output will include the IP address of your active Pro Custodibus API endpoint in a phrase like 192.0.2.0 is pro custodibus ip address
(the IP address will be something other than 192.0.2.0
, however). Try running the following command in a terminal on the host (replacing 192.0.2.0
with the actual IP address from the output):
$ ping -c1 192.0.2.0 PING 192.0.2.0 (192.0.2.0): 56 data bytes 64 bytes from 192.0.2.0: icmp_seq=0 ttl=37 time=10.138 ms --- 192.0.2.0 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/stddev = 10.138/10.138/10.138/0.000 ms
If the ping
command hangs or the output of the ping
command ends with 100% packet loss
, network access from the host to the Pro Custodibus servers may be blocked. You may need to update your firewall or other networking settings to allow the host to access the Pro Custodibus servers.
If the output of the ping
command includes 0% packet loss
, your connection to the Pro Custodibus servers is good. The Pro Custodibus servers may be down; email support@procustodibus.com to check.
Unhealthy pro custodibus api
If the output of the command includes unhealthy pro custodibus api
, there is a problem with the Pro Custodibus servers; email support@procustodibus.com for details.
Cannot access host record
If the output of the command includes cannot access host record
, the agent does not has permission to access the host’s record on Pro Custodibus. Either:
-
The host record on Pro Custodibus has been deleted
-
The agent record on Pro Custodibus has been deleted
-
The agent’s credentials for Pro Custodibus have been revoked
-
The agent’s access to the host on Pro Custodibus has been revoked
Try re-downloading the procustodibus.conf
and procustodibus-setup.conf
files as directed by the Set Up Host instructions, and then re-start the agent as directed by the Agent Status instructions.
Agent Status
To check the status of the agent on the host, run one the following commands (depending on the host’s operating system):
Linux With Systemd
Run the following command in a terminal on the host:
$ systemctl status procustodibus-agent ● procustodibus-agent.service - Pro Custodibus Agent Loaded: loaded (/etc/systemd/system/procustodibus-agent.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-07-09 12:45:36 PDT; 17s ago Main PID: 2770504 (procustodibus-a) Tasks: 1 (limit: 38335) Memory: 15.0M CGroup: /system.slice/procustodibus-agent.service └─2770504 /opt/venvs/procustodibus-agent/bin/python3 /opt/venvs/procustodibus-agent/bin/procustodibus-agent --loop=60 Jul 09 12:45:36 localhost systemd[1]: Started Pro Custodibus Agent.
If the agent is running, this command output should include active (running)
. Otherwise, try restarting the agent with this command:
$ sudo systemctl start procustodibus-agent
Substitute restart
for start
to re-start an already running agent.
Linux With OpenRC
Run the following command in a terminal on the host:
$ rc-service procustodibus-agent status procustodibus-agent started
If the agent is running, this command output should end with started
. Otherwise, try restarting the agent with this command:
$ sudo rc-service procustodibus-agent start
Substitute restart
for start
to re-start an already running agent.
Linux With Sysvinit
Run the following command in a terminal on the host:
$ /etc/init.d/procustodibus-agent status started
If the agent is running, this command should output started
. Otherwise, try restarting the agent with this command:
$ sudo /etc/init.d/procustodibus-agent start starting Pro Custodibus Agent ...
Substitute restart
for start
to re-start an already running agent.
FreeBSD
Run the following command in a terminal on the host as root:
# service procustodibus-agent status procustodibus_agent is running as pid 1234.
If the agent is running, this command output should contain is running
. Otherwise, try restarting the agent with this command as root:
# service procustodibus-agent start Starting procustodibus_agent.
Substitute restart
for start
to re-start an already running agent.
Agent Logs
To check the log files of the agent on the host, run one the following commands (depending on the host’s operating system):
Linux With Systemd
Run the following command in a terminal on the host:
$ journalctl -u procustodibus-agent
Linux With OpenRC
Run the following command in a terminal on the host:
$ less /var/log/procustodibus-agent.log
Also check the error log if present:
$ less /var/log/procustodibus-agent.err
Log Messages
Unauthorized for url
If the agent log ends with the following error message:
401 Client Error: Unauthorized for url: https://api.custodib.us/users/abcdef12345/credentials/signature
Try re-downloading the procustodibus.conf
and procustodibus-setup.conf
files as directed by the Set Up Host instructions, and then re-start the agent as directed by the Agent Status instructions.
Also, check the host’s clock drift — the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate.
No module named procustodibus_agent
If after upgrading the OS on a host you see the following error message in the agent logs:
ModuleNotFoundError: No module named 'procustodibus_agent'
This probably means the OS is now using a newer version of Python than the agent was originally installed under.
To fix, download the latest version of the agent, and reinstall it with the --install
flag:
$ cd procustodibus-agent-*/ $ sudo ./install.sh --install running as root install 1.3.1 agent configuration found at /etc/wireguard/procustodibus.conf agent credentials found at /etc/wireguard/procustodibus-credentials.conf /etc/wireguard/procustodibus.conf mode ok (-rw-rw-r--) /etc/wireguard/procustodibus.conf owner is root /etc/wireguard/procustodibus.conf group is root /etc/wireguard/procustodibus-credentials.conf mode ok (-rw-r-----) /etc/wireguard/procustodibus-credentials.conf owner is root /etc/wireguard/procustodibus-credentials.conf group is root libsodium found at /lib/aarch64-linux-gnu/libsodium.so.23 python 3.10.0 found at /usr/bin/python3 python includes all packages needed for venv WARNING python virtualenv broken at /opt/venvs/procustodibus-agent recreate virtualenv? ([y]es, [q]uit): y OK will delete virtualenv deleted virtualenv OK will create virtualenv created virtualenv WARNING agent package not installed install agent from /root/procustodibus-agent-1.3.1? ([y]es, [q]uit): y OK will install agent ... installed agent systemd daemon found at /etc/systemd/system/procustodibus-agent.service WARNING daemon failed start daemon? ([y]es, [q]uit): y OK will start daemon started daemon install SUCCESS