Testing

When developing Pro Custodibus, in addition to running the unit tests for each component, you will usually want to test the components together manually in an integrated fashion.

TLS Cert

It is not necessary to use HTTPS when developing Pro Custodibus. However, some features of the app UI may not work correctly without HTTPS (see MDN’s Secure Contexts documentation for details). Therefore, if you intend to do much serious development with Pro Custodibus, we recommend you generate and use a local TLS cert for HTTPS.

The mkcert program makes this fairly easy.

The mkcert program will add a new CA (Certificate Authority) to the trust store of your web browsers (and other system components).

If an attacker steals the private key to this CA (which will be stored unencrypted in your home folder, usually at ~/.local/share/mkcert/rootCA-key.pem), it would enable her to potentially read and alter all of your HTTPS traffic to all other web sites — not just to the local dev servers for which you’ve generated local TLS certs.

New CA Cert

If you have already installed mkcert and generated a CA cert previously for some other project, reuse this existing CA (and skip to the New TLS Cert section).

Otherwise, follow the installation instructions on the mkcert page to install the mkcert program. Then run the mkcert -install command once on your development machine to generate a new CA cert and install it into the machine’s trust stores.

If you have other development boxes or virtual machines from which you want to test the Pro Custodibus components, copy the CA cert (but not the private key) from the ~/.local/share/mkcert/rootCA.pem directory on the machine where you generated the CA cert to those other dev boxes or VMs; and move it to the ~/.local/share/mkcert/rootCA.pem path on those other machines. Then run the mkcert -install command on each machine.

New TLS Cert

Run the following command to generate a new TLS cert (signed by the New CA Cert you generated above):

$ mkcert *.local.custodib.us

Created a new certificate valid for the following names 📜
 - "*.local.custodib.us"

Reminder: X.509 wildcards only go one level deep, so this won't match a.b.local.custodib.us ℹ️

The certificate is at "./_wildcard.local.custodib.us.pem" and the key at "./_wildcard.local.custodib.us-key.pem" ✅

It will expire on 1 November 2025 🗓

Move the _wildcard.local.custodib.us.pem file to your /etc/ssl/certs/ directory, and the _wildcard.local.custodib.us-key.pem file to your /etc/ssl/private/ directory:

$ sudo mv _wildcard.local.custodib.us.pem /etc/ssl/certs/.
$ sudo mv _wildcard.local.custodib.us-key.pem /etc/ssl/private/.

You don’t have to use the local.custodib.us domain name (and you don’t have to generate a wildcard cert) — you just need to generate a cert with two domain names: one for the API, and one for the app.

For example, you could instead use a cert like the following if you so choose:

$ mkcert procustodibus-api.dev.example.net procustodibus-app.dev.example.net

If you use a different domain names, make sure to use them consistently in place of api.local.custodib.us and app.local.custodib.us everywhere.

Hostnames

In the /etc/hosts file on your dev box, add an entry to map the api.local.custodib.us and app.local.custodib.us domain names to 127.0.0.1:

$ cat <<EOF | sudo tee --append /etc/hosts
127.0.0.1 api.local.custodib.us
127.0.0.1 app.local.custodib.us
EOF

Do the same on any dev boxes or VMs on which you installed the New CA Cert, except use the IP address that the dev box or VM needs to use to access the machine on which you are developing Pro Custodibus. For other dev boxes on your local network, this would usually be the LAN IP address of the machine running Pro Custodibus; or for a VM running on that machine, the VM’s gateway IP address.

Reverse Proxy

With the API and app Docker containers running (ie, after you have run make dev from both your local procustodibus-api/ and procustodibus-app/ dev directories), run the following command in a different terminal from your procustodibus-app/ directory:

$ bin/nginx-dev.sh
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/08/13 20:51:21 [notice] 1#1: using the "epoll" event method
2023/08/13 20:51:21 [notice] 1#1: nginx/1.25.1
2023/08/13 20:51:21 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r4)
2023/08/13 20:51:21 [notice] 1#1: OS: Linux 5.19.0-50-generic
2023/08/13 20:51:21 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/08/13 20:51:21 [notice] 1#1: start worker processes
2023/08/13 20:51:21 [notice] 1#1: start worker process 21
2023/08/13 20:51:21 [notice] 1#1: start worker process 22
2023/08/13 20:51:21 [notice] 1#1: start worker process 23
2023/08/13 20:51:21 [notice] 1#1: start worker process 24

This will run a generic NGINX Docker container, listening on ports 80 and 443; and reverse-proxy requests to https://api.local.custodib.us to the running API container, and reverse-proxy requests to https://app.local.custodib.us to the running app container.

If you already have a webserver listening on ports 80 or 443, you can run it with a custom HTTPS port, such as 8443:

$ bin/nginx-dev.sh 8443

This will reverse-proxy requests to https://api.local.custodib.us:8443 to the running API container, and reverse-proxy requests to https://app.local.custodib.us:8443 to the running app container.

When using a custom HTTPS port, you must specify both the port (eg 8443) and the scheme (eg https) when attempting to access Pro Custodibus. Your browser will not be able to connect to the reverse proxy if you simply enter a naked domain name like app.local.custodib.us in your browser’s address bar.

When you run the nginx-dev.sh script, other hosts on your LAN will be able to access your development Pro Custodibus server. If you do not want this, either create a custom version of the script that doesn’t expose NGINX’s listen ports externally, or adjust the firewall on your dev box to protect them.

Alternatively, if you have already have NGINX running on your dev box, you can use it as the reverse proxy for Pro Custodibus directly. Simply copy the bin/nginx-dev.d/custos.conf file to NGINX’s config directory, and reload its configuration:

$ sudo cp bin/nginx-dev.d/custos.conf /etc/nginx/conf.d/.
$ sudo systemctl reload nginx

Component Connections

Being on different Docker networks, the various Pro Custodibus components (API, app, & agent) in your dev environment won’t be able to talk to each other without adjusting a few configuration settings.

API URLs

So that redirects and agent authentication will work, the API needs to know its own canonical URL; and so that it renders notifications correctly, it needs to know the app’s canonical URL.

If you haven’t set up a local TLS Cert, and the API is running in its Docker container at 10.242.4.242 on port 4000, and the app at 10.242.5.12 on port 5173, set the following environment variables in the .env file in your procustodibus-api/ directory:

# .env
API_CANONICAL_HOST=10.242.4.242
#API_CANONICAL_PORT=4000
#API_CANONICAL_SCHEME=http
#API_FORWARDED=false
APP_URL=http://10.242.5.12:5173

If you have set up a local TLS Cert, with the Reverse Proxy listening on port 443, set the following environment variables in the .env file in your procustodibus-api/ directory:

# .env
API_CANONICAL_HOST=api.local.custodib.us
API_CANONICAL_PORT=443
API_CANONICAL_SCHEME=https
API_FORWARDED=true
APP_URL=https://app.local.custodib.us

If the Reverse Proxy is listening on a custom port, like say 8443, use that port in the API_CANONICAL_PORT and APP_URL variables:

# .env
API_CANONICAL_HOST=api.local.custodib.us
API_CANONICAL_PORT=8443
API_CANONICAL_SCHEME=https
API_FORWARDED=true
APP_URL=https://app.local.custodib.us:8443

Kill the make dev command if it is currently running, and re-run it.

App URLs

So that the app can connect to the API, it needs to know the API’s canonical URL.

If you haven’t set up a local TLS Cert, and the API is running in its Docker container at 10.242.4.242 on port 4000, set the following environment variable in the .env.local file in your procustodibus-app/ directory:

# .env.local
VITE_APP_API_URL=http://10.242.4.242:4000

If you have set up a local TLS Cert, with the Reverse Proxy listening on port 443, set the following environment variables in the .env file in your procustodibus-api/ directory:

# .env.local
VITE_APP_API_URL=https://api.local.custodib.us

If the Reverse Proxy is listening on a custom port, like say 8443, include that port in the variable value:

# .env.local
VITE_APP_API_URL=https://api.local.custodib.us:8443

Kill the make dev command if it is currently running, and re-run it.

Agent URLs

So that the agent can connect to the API, it needs to know the API’s canonical URL. When you download a procustodibus.conf agent config file (following the steps in the Agent Account section), this file will automatically include an Api setting with the app’s current value for the API’s canonical URL:

# procustodibus.conf generated 8/1/2023, 12:34:56 PM PDT
[Procustodibus]
# My Test Host Agent
Agent = ABC123def45
# My Test Host
Host = DEF456ghi78
Api = https://api.local.custodib.us:8443

If you change your API’s canonical URL in the future, you’ll need to update this file to match (and restart the agent).

User Account

To set up an initial user with which to log into your dev API server, first set the following environment variable in the .env file of your procustodibus-api/ directory to any value:

# .env
SIGNUP_KEY=abcd1234

Kill the make dev command if it is currently running, and re-run it.

Then navigate to the canonical app URL in a web browser, with /signup appended to the URL. If you haven’t set up a local TLS Cert, use this URL:

http://10.242.5.12:5173/signup

Or if you have set up a local TLS Cert, use this URL:

https://app.local.custodib.us/signup

Enter whatever values you want for the Organization Name, Your Name, and Email Address fields (no emails will be sent from the API running in dev mode). Enter a 16-character or longer password into the Password field. Enter the value of the SIGNUP_KEY variable into the Signup Key field. Then press the Sign Up button.

This will create a new organization and user account with which to test. Write down the Login ID displayed on the resulting page (as well as the password you entered into the sign-up form) if you want to be able to use them to log in again.

Agent Account

Once logged in, navigate to the Hosts page, and click the Add (plus) icon. Enter some name into the Name field, the click the Add button.

This will create a new host record. Click the procustodibus.conf link to download the agent configuration file; then click the procustodibus-setup.conf link to download the agent setup file.

You can use these two files to run a test with the cli tox environment in your procustodibus-agent/ directory:

$ tox -e cli -- --config ~/Downloads/procustodibus.conf --test
... !!! no wireguard interfaces found !!! ...
... 127.0.0.1 is pro custodibus ip address ...
... healthy pro custodibus api ...
... can access host record on api for My Test Host ...
All systems go :)

If you’ve set up a local TLS Cert, you’ll need to specify the path to the cert bundle to avoid an CERTIFICATE_VERIFY_FAILED error:

$ CURL_CA_BUNDLE=~/.local/share/mkcert/rootCA.pem \
  tox -e cli -- --config ~/Downloads/procustodibus.conf --test

Whenever an agent successfully connects to the API using a procustodibus-setup.conf file, it will automatically replace the set-up file with a new procustodibus-credential.conf file, containing newly-set-up credentials.

Agent Docker

To test out the agent locally in a controlled environment, build and run a Docker dev image for it.

Run the following commands from the procustodibus-agent/ directory to build a local image named procustodibus-agent:dev:

$ tox -e build
$ docker/dev.sh dev

Create a directory to hold your test WireGuard and Pro Custodibus config files in a convenient location, and place them in that directory:

$ mkdir -p ~/test-agent/wireguard
$ cp ~/Downloads/procustodibus*.conf ~/test-agent/wireguard/.

Then, if you haven’t set up a local TLS Cert, run the image in a temporary container like the following:

$ docker run \
    --cap-add NET_ADMIN \
    --publish 51820:51820/udp \
    --name test-agent \
    --network custos \
    --rm \
    --volume $HOME/test-agent/wireguard:/etc/wireguard \
    procustodibus-agent:dev

 * /proc is already mounted
 * /run/lock: creating directory
 * /run/lock: correcting owner
   OpenRC 0.45.2 is starting up Linux 6.4.0-17-generic (x86_64) [DOCKER]

 * Caching service dependencies ... [ ok ]
 * Starting WireGuard interface wg0 ...Warning: `/etc/wireguard/wg0.conf' is world accessible
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.0.0.1/32 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip -4 route add 10.0.0.2/32 dev wg0
 [ ok ]
 * Starting procustodibus-agent ... [ ok ]
 * Starting dev-down-up ... [ ok ]
 * Starting dev-receive-traffic ... [ ok ]
 * Starting dev-send-traffic ... [ ok ]
2023-08-01 12:34:56,789 procustodibus_agent.agent INFO: Starting agent 1.2.3
... 1 wireguard interfaces found ...
... 10.242.4.242 is pro custodibus ip address ...
... healthy pro custodibus api ...
... can access host record on api for My Test Host ...
All systems go :)

However, if you have set up a local TLS Cert, first create a new file in a separate directory to hold the hostname mappings to your Reverse Proxy from within the custos network (where the network gateway is 10.242.4.1):

$ mkdir ~/test-agent/hostnames
$ cat <<EOF >~/test-agent/hostnames/custos
10.242.4.1 api.local.custodib.us
10.242.4.1 app.local.custodib.us
EOF

Then run the image in a temporary container with additional volume mappings for your custom TLS Cert CA and hostnames:

$ docker run \
    --cap-add NET_ADMIN \
    --publish 51820:51820/udp \
    --name test-agent \
    --network custos \
    --rm \
    --volume $HOME/test-agent/wireguard:/etc/wireguard \
    --volume $HOME/.local/share/mkcert:/usr/local/share/ca-certificates \
    --volume $HOME/test-agent/hostnames:/usr/local/share/hostnames \
    procustodibus-agent:dev

 * /proc is already mounted
 * /run/lock: creating directory
 * /run/lock: correcting owner
   OpenRC 0.45.2 is starting up Linux 6.4.0-17-generic (x86_64) [DOCKER]

 * Caching service dependencies ... [ ok ]
 * Applying dev custom cacerts /usr/local/share/ca-certificates/rootCA-key.pem ...
 * Applying dev custom cacerts /usr/local/share/ca-certificates/rootCA.pem ... [ ok ]
 * Applying dev custom hosts /usr/local/share/hostnames/custos ... [ ok ]
 * Starting WireGuard interface wg0 ...Warning: `/etc/wireguard/wg0.conf' is world accessible
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.0.0.1/32 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip -4 route add 10.0.0.2/32 dev wg0
 [ ok ]
 * Starting procustodibus-agent ... [ ok ]
 * Starting dev-down-up ... [ ok ]
 * Starting dev-receive-traffic ... [ ok ]
 * Starting dev-send-traffic ... [ ok ]
2023-08-01 12:34:56,789 procustodibus_agent.agent INFO: Starting agent 1.2.3
... 1 wireguard interfaces found ...
... 10.242.4.1 is pro custodibus ip address ...
... healthy pro custodibus api ...
... can access host record on api for My Test Host ...
All systems go :)

Or alternatively, set up a docker-compose.yml file like the following, to run the image with Docker Compose:

# ~/test-agent/docker-compose.yml
version: '3'

networks:
  custos:
    external: true
    name: custos

services:
  agent:
    image: procustodibus-agent:dev
    cap_add:
    - NET_ADMIN
    ports:
    - 51820:51820
    networks:
    - custos
    volumes:
    - ./wireguard:/etc/wireguard
    - /home/me/.local/share/mkcert:/usr/local/share/ca-certificates
    - ./hostnames:/usr/local/share/hostnames