Troubleshooting
Agent Test
To check whether or not the agent is able to send WireGuard data to the Pro Custodibus servers, run the following command on the host:
$ sudo /opt/venvs/procustodibus-agent/bin/procustodibus-agent --test ... 1 wireguard interfaces found ... ... 192.0.2.3 is pro custodibus ip address ... ... healthy pro custodibus api ... ... can access host record on api for My Test Host ... All systems go :)
Make sure you run this command as root. If the output ends All systems go
, then it should be good. Try restarting the agent as directed by the Agent Status instructions.
Missing conf
If the output of the test is Missing conf for Agent ID and Host ID
, then either:
-
You did not run the test as root; try re-running the test as root.
-
The agent configuration file is missing required settings (or missing altogether); try re-downloading the configuration file as directed by the Set Up Host instructions.
-
The agent configuration file has the wrong SELinux context label.
If the agent configuration file is present at /etc/wireguard/procustodibus.conf
, and you examine it, it should look similar to the following:
$ sudo cat /etc/wireguard/procustodibus.conf # procustodibus.conf generated 1/1/2020, 12:34:56 PM PDT [Procustodibus] # Alice's Laptop Agent Agent = ABC123def45 # Alice's Laptop Host = DEF456ghi78
Missing code
If the output of the test includes Missing code in setup file
, then the agent setup file is missing required settings; try re-downloading the setup file as directed by the Set Up Host instructions.
If you examine the agent setup file, it should look similar to the following:
$ sudo cat /etc/wireguard/procustodibus-setup.conf # procustodibus-setup.conf generated 1/1/2020, 12:34:56 PM PDT [Procustodibus.Setup] # Alice's Laptop Agent Agent = ABC123def45 Code = GHI789JKL01 Expires = 2020-01-03T20:34:56Z
Setup code has expired
If the output of the test includes Setup code has expired
, then the code in the agent setup file has expired; download a new setup file as directed by the Set Up Host instructions.
Missing credentials
If the output of the test ends with No such file or directory: '/etc/wireguard/procustodibus-credentials.conf'
, then the agent credentials file is missing; download a new setup file as directed by the Set Up Host instructions (a new setup file will allow the agent to generate a new credentials file).
Missing private key
If the output of the test includes Missing private key in credentials file
, then the agent credentials file is missing required settings; try downloading a new setup file as directed by the Set Up Host instructions (a new setup file will allow the agent to generate a new credentials file).
If you examine the agent credentials file, it should look similar to the following:
$ sudo cat /etc/wireguard/procustodibus-credentials.conf # procustodibus-credentials.conf generated 2020-01-01T20:34:56Z # for agent ABC123def45 on a123.corp.example.com [Procustodibus.Credentials] PublicKey = O2onvM62pC1io6jQKm8Nc2UyFXcd4kOmOsBIoYtZ2ik= PrivateKey = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
No wg executable
If the output of the command includes no wg executable found
, then you may not have installed WireGuard on the host yet. Try installing WireGuard and adding a WireGuard interface.
No wireguard interfaces
If the output of the command includes no wireguard interfaces found
, then you may not have configured any WireGuard interfaces on the host yet. Try adding one.
Cannot lookup ip address
If the output of the command includes cannot lookup ip address
, then the host may not be able to resolve the Pro Custodibus servers' DNS entries. Make sure that:
-
The host’s DNS resolver is working
-
The host’s DNS resolver can resolve public DNS entries
Try running the following command in a terminal on the host:
$ dig +short api.custodib.us 192.0.2.3
If the dig
command outputs nothing, likely either a) or b) above is not true.
Server unavailable
If the output of the command includes server unavailable
, check whether or not the output also includes cannot lookup ip address
. If it does, see the Cannot lookup ip address issue above.
If the output does not include cannot lookup ip address
, the output will include the IP address of your active Pro Custodibus API endpoint in a phrase like 192.0.2.3 is pro custodibus ip address
(the IP address will be something other than 192.0.2.3
, however). Try running the following command in a terminal on the host (replacing 192.0.2.3
with the actual IP address from the output):
$ ping -c1 192.0.2.3 PING 192.0.2.3 (192.0.2.3): 56 data bytes 64 bytes from 192.0.2.3: icmp_seq=0 ttl=37 time=10.138 ms --- 192.0.2.3 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/stddev = 10.138/10.138/10.138/0.000 ms
If the ping
command hangs or the output of the ping
command ends with 100% packet loss
, network access from the host to the Pro Custodibus servers may be blocked. You may need to update your firewall or other networking settings to allow the host to access the Pro Custodibus servers.
If the output of the ping
command includes 0% packet loss
, your connection to the Pro Custodibus servers is good. The Pro Custodibus servers may be down; email support@procustodibus.com to check.
Unhealthy pro custodibus api
If the output of the command includes unhealthy pro custodibus api
, then there is a problem with the Pro Custodibus servers; email support@procustodibus.com for details.
Cannot access host record
If the output of the command includes cannot access host record
, then the agent does not have permission to access the host’s record on Pro Custodibus. Either:
-
The host record on Pro Custodibus has been deleted
-
The agent record on Pro Custodibus has been deleted
-
The agent’s credentials for Pro Custodibus have been revoked
-
The agent’s access to the host on Pro Custodibus has been revoked
-
The host’s clock is not accurate (the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate)
If the host’s clock is off by more than a few minutes, fix the clock and restart the agent. Otherwise, try re-downloading the procustodibus.conf
and procustodibus-setup.conf
files as directed by the Set Up Host instructions, and then restart the agent as directed by the Agent Status instructions.
Agent Status
To check the status of the agent on the host, run one the following commands (depending on the host’s operating system):
Linux With Systemd
Run the following command in a terminal on the host:
$ systemctl status procustodibus-agent ● procustodibus-agent.service - Pro Custodibus Agent Loaded: loaded (/etc/systemd/system/procustodibus-agent.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-07-09 12:45:36 PDT; 17s ago Main PID: 2770504 (procustodibus-a) Tasks: 1 (limit: 38335) Memory: 15.0M CGroup: /system.slice/procustodibus-agent.service └─2770504 /opt/venvs/procustodibus-agent/bin/python3 /opt/venvs/procustodibus-agent/bin/procustodibus-agent --loop=60 Jul 09 12:45:36 localhost systemd[1]: Started Pro Custodibus Agent.
If the agent is running, this command output should include active (running)
. Otherwise, try starting the agent with this command:
$ sudo systemctl start procustodibus-agent
Substitute restart
for start
to restart an already running agent.
Linux With OpenRC
Run the following command in a terminal on the host:
$ rc-service procustodibus-agent status procustodibus-agent started
If the agent is running, this command output should end with started
. Otherwise, try starting the agent with this command:
$ sudo rc-service procustodibus-agent start
Substitute restart
for start
to restart an already running agent.
Linux With Sysvinit
Run the following command in a terminal on the host:
$ /etc/init.d/procustodibus-agent status started
If the agent is running, this command should output started
. Otherwise, try starting the agent with this command:
$ sudo /etc/init.d/procustodibus-agent start starting Pro Custodibus Agent ...
Substitute restart
for start
to restart an already running agent.
FreeBSD
Run the following command in a terminal on the host as root:
# service procustodibus-agent status procustodibus_agent is running as pid 1234.
If the agent is running, this command output should contain is running
. Otherwise, try starting the agent with this command as root:
# service procustodibus-agent start Starting procustodibus_agent.
Substitute restart
for start
to restart an already running agent.
Agent Logs
To check the log files of the agent on the host, run one the following commands (depending on the host’s operating system):
Linux With Systemd
Run the following command in a terminal on the host:
$ journalctl -u procustodibus-agent
Linux With OpenRC
Run the following command in a terminal on the host:
$ less /var/log/procustodibus-agent.log
Also check the error log if present:
$ less /var/log/procustodibus-agent.err
Log Messages
Unauthorized for url
If the agent log ends with the following error message:
401 Client Error: Unauthorized for url: https://api.custodib.us/users/ABC123def45/credentials/signature
Try re-downloading the procustodibus.conf
and procustodibus-setup.conf
files as directed by the Set Up Host instructions, and then restart the agent as directed by the Agent Status instructions.
Also, check the host’s clock drift — the host’s clock must be within a few minutes of coordinated universal time in order for the agent to authenticate.
No module named procustodibus_agent
If after upgrading the OS on a host you see the following error message in the agent logs:
ModuleNotFoundError: No module named 'procustodibus_agent'
This probably means the OS is now using a newer version of Python than the agent was originally installed under.
To fix, download the latest version of the agent, and reinstall it with the --install
flag:
$ cd procustodibus-agent-*/ $ sudo ./install.sh --install running as root install 1.3.1 agent configuration found at /etc/wireguard/procustodibus.conf agent credentials found at /etc/wireguard/procustodibus-credentials.conf /etc/wireguard/procustodibus.conf mode ok (-rw-rw-r--) /etc/wireguard/procustodibus.conf owner is root /etc/wireguard/procustodibus.conf group is root /etc/wireguard/procustodibus-credentials.conf mode ok (-rw-r-----) /etc/wireguard/procustodibus-credentials.conf owner is root /etc/wireguard/procustodibus-credentials.conf group is root libsodium found at /lib/aarch64-linux-gnu/libsodium.so.23 python 3.10.0 found at /usr/bin/python3 python includes all packages needed for venv WARNING python virtualenv broken at /opt/venvs/procustodibus-agent recreate virtualenv? ([y]es, [q]uit): y OK will delete virtualenv deleted virtualenv OK will create virtualenv created virtualenv WARNING agent package not installed install agent from /root/procustodibus-agent-1.3.1? ([y]es, [q]uit): y OK will install agent ... installed agent systemd daemon found at /etc/systemd/system/procustodibus-agent.service WARNING daemon failed start daemon? ([y]es, [q]uit): y OK will start daemon started daemon install SUCCESS
SELinux
If you’re using SELinux, try running the ls -lZ
command to check the context labels for the files in the /etc/wireguard
directory. If the files are labeled correctly, you should see output similar to the following:
$ sudo ls -lZ /etc/wireguard total 12 -rw-r--r--. 1 root root system_u:object_r:etc_t:s0 181 May 27 22:52 procustodibus.conf -rw-r-----. 1 root root system_u:object_r:etc_t:s0 275 May 27 22:52 procustodibus-credentials.conf -rw-r-----. 1 root root system_u:object_r:etc_t:s0 554 May 25 21:43 wg0.conf
If not, try running the restorecon
command to relabel them:
$ sudo restorecon -Fv /etc/wireguard/* Relabeled /etc/wireguard/procustodibus-credentials.conf from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:etc_t:s0 Relabeled /etc/wireguard/procustodibus.conf from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:etc_t:s0