Brokers

Pro Custodibus can push several different types of events to your own systems through a local lightweight daemon that you run on one of your own hosts. We call this daemon the Pro Custodibus broker.

The broker can push several types of events:

To several types of sinks:

File
Syslog

Add

To add a broker, navigate to the Add Broker page:

Click the Admin link in the app header.
Click the Brokers link in the Administration panel.
Click the “plus” icon on the right side of the Brokers panel.

Name

Enter a descriptive name for the host, like “SIEM Alerts”, in the Name field. This name will be used as the display name for the broker in the UI. There are no character restrictions, and you can change it later.

Form Submit

Click the Add button to submit the form and create the broker.

Setup

After adding a broker, download the broker.conf and broker-setup.conf configuration files from the broker’s setup page. This page will be displayed after adding a broker; you can return to this page later by following these steps:

Click the Admin link in the app header.
Click the Brokers link in the Administration panel.
Find the broker in the list, and click its name to view its main page.
Click the “gear” icon on the right side of the Credentials panel to view the broker’s setup page.

Configuration File

On the broker’s setup page, click the broker.conf link to download the broker configuration file. Copy this file to the following location on the host on which you will run the broker:

/etc/procustodibus/broker/broker.conf

The configuration file contains the configuration settings the broker will use. You will need to customize this file; see the Configure section.

Setup File

On the broker’s setup page, click the broker-setup.conf link to download the broker setup file. Copy this file to the following location on the broker’s host:

/etc/procustodibus/broker/broker-setup.conf

The setup file contains a one-time setup code that the broker will use to authenticate with the Pro Custodibus servers the first time it connects. The broker will delete the setup file after it uses it.

The one-time code in the setup file is good for three days. If the code already has been used or has expired, you can download a replacement setup file with a new one-time code by clicking the same broker-setup.conf link again.

Credentials File

As part of the setup process, the broker will generate a unique Ed25519 keypair that it will use to authenticate with on subsequent connections. The broker will save this keypair to the following location on the host:

/etc/procustodibus/broker/broker-credentials.conf

The configuration file and either the credentials file or the setup file must be present on the host in order for the broker to run.

Configure

Each [Pipe.XYZ] section in the /etc/procustodibus/broker/broker.conf file represents a separate sink configuration to which events from Pro Custodibus can be piped. The XYZ portion of the section name is the pipe’s name, an arbitrary alphanumeric value (A-Z, a-z, 0-9, _, and - characters allowed). Each pipe’s name must be unique. You can have as many pipes as you like.

The broker.conf file you download will contain several example pipes, commented out, like this:

# broker.conf generated 1/1/2022, 12:34:56 PM PST
[Broker]
# My Example Broker
Broker = abcdef12345

# [Pipe.ExampleSyslog]
# From = alerts, endpoint_stats, log_authn, log_changes
# Max = 100
# To = syslog
# Socket = localhost:514
# Facility = local0
# Priority = info
# Tag = procustodibus

# [Pipe.ExampleFile]
# From = alerts, endpoint_stats, log_authn, log_changes
# Max = 100
# To = file
# File = /var/log/procustodibus/broker/all.jsonl

Events

The From property of each pipe lists the (comma-separated) type of events the broker will pull from Pro Custodibus for the pipe. The following types are available:

alerts: The same alert events displayed on the Alerts page in the UI.
endpoint_stats: The raw feed of WireGuard handshake events, with endpoint IPs and bytes sent/received (the same data used by various charts and the endpoint Activity panel).
log_authn: Events related to Pro Custodibus authentication (similar to the Recent Sessions panel on user pages, but not fully exposed anywhere in the UI).
log_changes: The WireGuard configuration changes listed in the Changes panels on the dashboard and other pages (WireGuard endpoint IP address changes, keypair and preshared-key changes, endpoints added and removed, etc).

The Max property indicates the maximum number of events to pull at a time (defaults to 100).

Sinks

The To property of each pipe indicates the sink to which the broker will push events. The following types are available:

file: Local log file on the host (eg /var/log/procustodibus/broker/all.jsonl).
syslog: Syslog on the local host, or any remote host to which the local host can connect.

The other properties of a pipe are specific to the type of sink used by the pipe, described below:

File

The file type pipes to a local JSON Lines log file on the host. This log file is specified via the File property. For example, the following pipes to the /var/log/procustodibus/broker/stats.jsonl file:

[Pipe.MyStats]
From = endpoint_stats
To = file
File = /var/log/procustodibus/broker/stats.jsonl

To log to a file outside of the /var/log/procustodibus/broker directory on a host that uses systemd, you must edit the /etc/systemd/system/procustodibus-broker.service file, and add a ReadWritePaths entry for the log file path (or for a directory path containing the log file).

For example, to log to a file called /var/log/custom/example.log, you’d have to add an entry like the following to the [Service] section of /etc/systemd/system/procustodibus-broker.service:

ReadWritePaths=/var/log/custom/example.log

After changing the procustodibus-broker.service file, run the following commands to reload your changes and restart the broker:

sudo systemctl daemon-reload
sudo systemctl restart procustodibus-broker

When run with systemd, log files created by the broker will have an access mode of 640 by default (readable and writeable by the procustodibus-broker user, readable by members of the procustodibus-broker group, and not readable by others). To allow others to read log files created by the broker, either change the log files' access mode after they have been created, or change the UMask setting in the /etc/systemd/system/procustodibus-broker.service file to 0022:

UMask=0022

After changing the procustodibus-broker.service file, run the following commands to reload your changes and restart the broker:

sudo systemctl daemon-reload
sudo systemctl restart procustodibus-broker

Syslog

The syslog type pipes JSON messages to the syslog server running on the local host, or a remote syslog server. The following properties may be specified:

Socket: Hostname and UDP port of syslog server (eg 10.10.10.10:1234) or path to local socket (eg /dev/log). Defaults to localhost:514.
Facility: Name of syslog facility. Defaults to local0.
Priority: Name of syslog priority. Defaults to info.
Tag: Syslog tag. Defaults to procustodibus.

Download

After setting up the broker’s configuration files, download the broker tarball below. After you have downloaded the tarball and verified it, copy it to the host on which you will run the broker, and run the broker installer.

Version	Tarball	PGP Signature	SHA256 Checksum
latest	procustodibus-broker-latest.tar.gz	.sig	.sha256
1.1.1	procustodibus-broker-1.1.1.tar.gz	.sig	.sha256
1.1.0	procustodibus-broker-1.1.0.tar.gz	.sig	.sha256
1.0.0	procustodibus-broker-1.0.0.tar.gz	.sig	.sha256

Version

Tarball

PGP Signature

SHA256 Checksum

latest

procustodibus-broker-latest.tar.gz

.sig

.sha256

1.1.1

procustodibus-broker-1.1.1.tar.gz

.sig

.sha256

1.1.0

procustodibus-broker-1.1.0.tar.gz

.sig

.sha256

1.0.0

procustodibus-broker-1.0.0.tar.gz

.sig

.sha256

Signing Keys

All tarballs will be signed by one of the following PGP keys:

https://www.arcemtene.com/keys/procustodibus-build.asc
primary key ID: EFC1AE969DD8159F
primary key fingerprint: 98EE 78FA B506 BEAB C0DA DE15 EFC1 AE96 9DD8 159F

Verify Signature

To check the integrity of a downloaded tarball, verify it via PGP signature. Do it the same way for the broker as for the agent; in brief:

$ gpg --keyserver keys.openpgp.org --recv-keys EFC1AE969DD8159F
gpg: key EFC1AE969DD8159F: public key "Pro Custodibus Build 1 <build@custodib.us>" imported
gpg: Total number processed: 1
gpg:               imported: 1

$ wget https://ad.custodib.us/brokers/procustodibus-broker-latest.tar.gz.sig -O - | gpg --verify - procustodibus-broker-latest.tar.gz
gpg: Signature made Fri Oct  9 19:53:00 2020 UTC
gpg:                using RSA key 98EE78FAB506BEABC0DADE15EFC1AE969DD8159F
gpg: Good signature from "Pro Custodibus Build 1 <build@custodib.us>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 98EE 78FA B506 BEAB C0DA  DE15 EFC1 AE96 9DD8 159F

Source Code

The Pro Custodibus broker is open source. You can check out the source code at https://git.sr.ht/~arx10/procustodibus-broker. Note, however, that we only support versions of the broker listed in the table above.

Install

To install the Pro Custodibus broker, copy the broker tarball to the host on which you want to install it, and extract the tarball. Run the following command in a terminal on the host:

$ tar xf procustodibus-broker-latest.tar.gz

This will extract the latest version of the broker to a directory called procustodibus-broker-1.0.0 (or whatever the actual latest version number is instead of 1.0.0).

Then run the installer. The installer must be run as root. Run the following commands in the terminal on the host to run the installer:

$ cd procustodibus-broker-*/
$ sudo ./install.sh --install
running as root
install 1.0.0
broker configuration found at /etc/procustodibus/broker/broker.conf
broker setup found at /etc/procustodibus/broker/broker-setup.conf
WARNING daemon user procustodibus-broker not found
create user? ([y]es, [q]uit): y
OK will create user
created user
...
...
...
WARNING daemon dead
start daemon? ([y]es, [q]uit): y
OK will start daemon
Created symlink /etc/systemd/system/default.target.wants/procustodibus-broker.service → /etc/systemd/system/procustodibus-broker.service.
started daemon
install SUCCESS

Run the installer with the --help flag to see the other installer options.

Upgrade

To upgrade the broker, download the latest version of the broker, and run its installer with the --install flag. The installer will detect that you are upgrading an existing install, and prompt you accordingly.

Troubleshooting

Broker Test

To check whether or not the broker is pull data from the Pro Custodibus servers, run the following command on the broker’s host:

$ sudo -u procustodibus-broker /opt/venvs/procustodibus-broker/bin/procustodibus-broker --test
... 192.0.2.0 is pro custodibus ip address ...
... healthy pro custodibus api ...
... can access queues on api for My Test Broker ...
All systems go :)

Make sure you run this command as the broker’s daemon user. If the output ends All systems go, then it should be good. Try re-starting the broker as directed by the Broker Status instructions.

Pipe Test

To check whether or not the broker is able to push data to one its configured pipes, run the following command on the broker’s host, substituting the configured name of the pipe for “MyPipe”:

$ sudo -u procustodibus-broker /opt/venvs/procustodibus-broker/bin/procustodibus-broker --test-pipe MyPipe

You should see a test record pushed to the sink configured for the pipe.

Broker Status

To check the status of the broker on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ systemctl status procustodibus-broker
● procustodibus-broker.service - Pro Custodibus Broker
     Loaded: loaded (/etc/systemd/system/procustodibus-broker.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-07-09 12:45:36 PDT; 17s ago
   Main PID: 2770504 (procustodibus-b)
      Tasks: 1 (limit: 38335)
     Memory: 15.0M
     CGroup: /system.slice/procustodibus-broker.service
             └─2770504 /opt/venvs/procustodibus-broker/bin/python3 /opt/venvs/procustodibus-broker/bin/procustodibus-broker --loop=120

Jul 09 12:45:36 localhost systemd[1]: Started Pro Custodibus Broker.

If the broker is running, this command output should include active (running). Otherwise, try restarting the broker with this command:

$ sudo systemctl start procustodibus-broker

Substitute restart for start to re-start an already running broker.

Linux With OpenRC

Run the following command in a terminal on the host:

$ rc-service procustodibus-broker status
procustodibus-broker started

If the broker is running, this command output should end with started. Otherwise, try restarting the broker with this command:

$ sudo rc-service procustodibus-broker start

Substitute restart for start to re-start an already running broker.

Linux With Sysvinit

Run the following command in a terminal on the host:

$ /etc/init.d/procustodibus-broker status
started

If the broker is running, this command should output started. Otherwise, try restarting the broker with this command:

$ sudo /etc/init.d/procustodibus-broker start
starting Pro Custodibus Agent ...

Substitute restart for start to re-start an already running broker.

FreeBSD

Run the following command in a terminal on the host as root:

# service procustodibus-broker status
procustodibus_broker is running as pid 1234.

If the broker is running, this command output should contain is running. Otherwise, try restarting the broker with this command as root:

# service procustodibus-broker start
Starting procustodibus_broker.

Substitute restart for start to re-start an already running broker.

Broker Logs

To check the log files of the broker on the host, run one the following commands (depending on the host’s operating system):

Linux With Systemd

Run the following command in a terminal on the host:

$ journalctl -u procustodibus-broker
-- Logs begin at Thu 2020-07-09 12:34:36 UTC, end at Thu 2020-07-09 12:34:36 UTC. --
Jul 09 12:45:36 localhost systemd[1]: Started Pro Custodibus Broker.

Linux With OpenRC

Run the following command in a terminal on the host:

$ less /var/log/procustodibus/broker/daemon.log

Also check the error log if present:

$ less /var/log/procustodibus/broker/daemon.err

Linux With Sysvinit

Run the following command in a terminal on the host:

$ less /var/log/procustodibus-broker.log

FreeBSD

Run the following command in a terminal on the host, then search for procustodibus-broker:

$ less /var/log/messages
/procustodibus-broker

Changelog

1.1.1 - 2024-01-15

Fixed

Fixed DNS connectivity check in IPv6-only environments.

1.1.0 - 2023-09-22

Requires Pro Custodibus API version 1.1 or newer.

Changed

Use server challenge for signature authentication (insteading of signing an URL with a timestamp).
Added timeout to Pro Custodibus API requests.
Added additional hardening constraints to systemd service unit. This will not be upgraded automatically — see the below “Manually Upgrade Systemd Hardening” section to apply.
Require recent versions of package dependencies.
Installer detect and recreate broken virtualenv.
Installer upgrade pip and setuptools before install.

Fixed

Installer now detects and fixes conf files with wrong SELinux context labels.
Installer now ensures libffi headers and other necessary OS packages are installed on platforms where they are needed to build PyNaCl.

Manually Upgrade Systemd Hardening

The additional hardening constraints for the broker’s systemd service unit will only be installed automatically on fresh installs. It will not be upgraded automatically. To apply the additional hardening manually, copy the etc/systemd.service file from the broker tarball to the /etc/systemd/system/procustodibus-broker.service file on the host, reload the configuration, and restart the broker:

$ sudo cp etc/systemd.service /etc/systemd/system/procustodibus-broker.service
$ sudo systemctl daemon-reload
$ sudo systemctl restart procustodibus-broker

1.0.0 - 2022-02-13

Initial release.