Geolocation Data

You can use Pro Custodibus just fine without geolocation data. However, adding geolocation data to your database can help make the UI more relevant, plus enable some additional alerts.

The API source code includes scripts to download and deploy the GeoLite2 geolocation data from MaxMind. You can get a free license key for the GeoLite2 data from here:

https://dev.maxmind.com/geoip/geolite2-free-geolocation-data

Docker

If you’re running Pro Custodibus via Docker, add your GeoLite2 license key to your API environment variables file, as the GEOIP_LICENSE_KEY variable:

# /srv/containers/procustodibus/api.env
GEOIP_LICENSE_KEY=abcd1234

Then, while your Pro Custodibus database is up and running, run the procustodibus/update-geoip-data Docker image, connected to the same network that your Pro Custodibus database is on (and using your API environment variables file to access the database credentials):

$ sudo docker run \
    --env-file /srv/containers/procustodibus/api.env \
    --network procustodibus_default \
    --rm \
    procustodibus/update-geoip-data

This container will automatically download the latest GeoLite2 data from MaxMind, process it, and deploy it to your Pro Custodibus database. You can run this container (with the same settings) every time you want to update your database with the latest geolocation data. We recommend you do this on a monthly basis.

Download

If you’re not running Pro Custodibus via Docker, you can download, process, and deploy the GeoLite2 data via a series of scripts.

To start, clone the API Git repository, and run the download.sh script from its ops/geolite2/ directory, passing the script your GeoLite2 license key as the GEOIP_LICENSE_KEY environment variable:

$ git clone https://git.sr.ht/~arx10/procustodibus-api
$ GEOIP_LICENSE_KEY=abcd1234 procustodibus-api/ops/geolite2/download.sh
--2023-09-01 09:37:43--  https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN-CSV&license_key=abcd1234&suffix=zip.sha256
Resolving download.maxmind.com (download.maxmind.com)... 2606:4700::6810:262f, 2606:4700::6810:252f, 104.16.37.47, ...
Connecting to download.maxmind.com (download.maxmind.com)|2606:4700::6810:262f|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96 [text/plain]
Saving to: ‘STDOUT’

-                  100%[==========================>]      96  --.-KB/s    in 0s

2023-09-01 09:37:44 (15.8 MB/s) - written to stdout [96/96]

--2023-09-01 09:37:44--  https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN-CSV&license_key=abcd1234&suffix=zip
Resolving download.maxmind.com (download.maxmind.com)... 2606:4700::6810:262f, 2606:4700::6810:252f, 104.16.38.47, ...
Connecting to download.maxmind.com (download.maxmind.com)|2606:4700::6810:262f|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5674597 (5.4M) [application/zip]
Saving to: ‘asn.zip’

asn.zip            100%[==========================>]   5.41M  27.7MB/s    in 0.2s

2023-09-01 09:37:44 (27.7 MB/s) - ‘asn.zip’ saved [5674597/5674597]

--2023-09-01 09:37:44--  https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City-CSV&license_key=abcd1234&suffix=zip.sha256
Resolving download.maxmind.com (download.maxmind.com)... 2606:4700::6810:262f, 2606:4700::6810:252f, 104.16.37.47, ...
Connecting to download.maxmind.com (download.maxmind.com)|2606:4700::6810:262f|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 97 [text/plain]
Saving to: ‘STDOUT’

-                  100%[==========================>]      97  --.-KB/s    in 0s

2023-09-01 09:37:45 (18.5 MB/s) - written to stdout [97/97]

--2023-09-01 09:37:45--  https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City-CSV&license_key=abcd1234&suffix=zip
Resolving download.maxmind.com (download.maxmind.com)... 2606:4700::6810:252f, 2606:4700::6810:262f, 104.16.37.47, ...
Connecting to download.maxmind.com (download.maxmind.com)|2606:4700::6810:252f|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 52453859 (50M) [application/zip]
Saving to: ‘city.zip’

city.zip           100%[==========================>]  50.02M  29.7MB/s    in 1.7s

2023-09-01 09:37:47 (29.7 MB/s) - ‘city.zip’ saved [52453859/52453859]

asn.zip: OK
city.zip: OK

This script will download the GeoLite2 data as a couple of .zip files to your /tmp/geolite2-downloaded/ directory.

You can specify a different directory into which to download the data via the first argument of the download.sh script.

This script requires you to have the wget utility installed.

Process

Once you download the GeoLite2 data, run the process.sh script from the same ops/geolite2/ directory to unzip it and process the CSV files it contains:

$ procustodibus-api/ops/geolite2/process.sh

This script will format the geolocation data files so that they can be loaded directly into the Pro Custodibus database. It will save them into your /tmp/geolite2-processed/ directory.

You can specify a different directory into which to save the processed files via the second argument of the process.sh script. Use the first argument of the script to specify the directory from which to read the unprocessed .zip files (/tmp/geolite2-downloaded/ by default).

This script requires you to have the unzip and gawk (GNU Awk) utilities installed.

Deploy

Once you’ve downloaded and processed the GeoLite2 data, copy the data files from the /tmp/geolite2-processed/ directory, plus the deploy.sql script from the ops/geolite2/ directory, to a machine that can access your Pro Custodibus database. Then run the following command to load the processed GeoLite2 data into the database (substituting your actual database connection information as appropriate):

$ psql \
    --host db.example.com \
    --dbname procustodibus_db \
    --username procustodibus_user \
    --password \
    --set data_dir=/tmp/geolite2-processed \
    --file ops/geolite2/deploy.sql
Password:
TRUNCATE TABLE
psql:ops/geolite2/deploy.sql:3: NOTICE:  truncate cascades to table "geoip_city_blocks"
TRUNCATE TABLE
TRUNCATE TABLE
COPY 129277
COPY 498442
COPY 114731
COPY 1305237
COPY 3787917

Note that this script may take a number of minutes to complete, as the GeoLite2 data files contain several million records.