Troubleshooting

Backing up the database

docker-compose.yaml (under services > dhdb > volumes) configures Docker to create the mariadb-backups directory in the host root, bound to /mariadb-backups/ in the database container. This is intended to be a convenient way of moving backups in and out of the container.

Backup and restore scripts are bundled in the database container for convenience. These generate and execute - respectively - a series of SQL INSERT statements that creates and populates a duplicate of every table in the database. With the dhdb container running, may backup the database as follows: $ docker exec <db container name> /etc/dhdb/backup-to-sql.sh

In case you need to perform a backup operation manually, you may find more information at https://mariadb.com/kb/en/backup-and-restore-overview.

Caution: The MariaDB documentation contains procedures for backing up the entire database directory. This is not recommended, as restoring a backup from a previous version may introduce a conflict with the database version migration system. Instead, whenever possible, we recommend backing up the the database as SQL statements. To save on disk space, use gzip’s ‘-9' argument to compress the data more aggressively.

Restoring the database

First, navigate to the Datahosting project root directory.

To prevent database resource conflicts, stop the other rbr containers before restoring the database.

$ docker container stop rbr-dhweb-1
$ docker container stop rbr-dhdaemon-1
$ docker exec -i rbr-dhdb-1 /etc/dhdb/restore-from-sql.sh /mariadb-backups/<backup-name>.sql
$ docker container start rbr-dhweb-1
$ docker container start rbr-dhdaemon-1

Accessing the database manually

1. $ docker exec -it rbr-dhdb-1 mariadb -p

2. Enter the root password.

3. To access the Datahosting database, enter > using datahosting

Accessing the shell

$ docker <container name> exec -it bash

Fixing port conflicts

If another application (e.g. a web server) on the host is already using a port used by the Datahosting application (e.g. port 80), the containers will fail to run with an error such as “address already is in use“.

The port mappings are defined in docker-compose.yaml as services > <service> > ports > <host port>:<container port>. The container ports should remain unchanged, but the host port should be changed to a different port (e.g. port 8080). Note that any external network configurations, RBRcervello configurations, or service configurations referencing the original port number will have to be updated as well.

Fixing restarting containers

If a container fails to start, view and save a copy of the logs, and include them in any emails to RBR.

If dhdaemon is continually restarting, the most likely problem is a misconfiguration causing a JSON file to fail validation. Check secrets/web-user-config.json and daemon/database-config/customer-config.json files against their respective secrets/web-user-config-schema.json and daemon/database-config/customer-config-schema.json schemas using https://www.jsonschemavalidator.net/.

Fixing communication with instruments

If an instrument cannot communicate with the Datahosting service or isn’t appearing in the web interface, please verify that:

• The instrument is assigned the correct serial number in customer-configuration.json.

• The RBRcervello’s user.json configuration has the correct destination address.

• Port 9140 on the container host (or the port configured to map to dhdaemon’s port 9140 in docker-compose.yaml) is exposed to the internet and not blocked by a firewall.

Viewing the logs

You may view a container’s terminal output with docker logs <container name> or view within it files individually with docker <container name> exec <command> <options>, where <command> is any of the commonly used file access programs (e.g. ls, cat, tail, head, less). To back up these files, you may redirect the output of docker exec to a file on the container host.

Daemon

The daemon logs are stored in a Docker volume that is only accessible within the containers. Logs are stored at /var/log/dhdaemon.

Web host

The web host does not output anything to the terminal by default. Logs can be found at /var/log/nginx/access.log, /var/log/nginx/error.log, and /var/log/php7.2-fpm.log.

Helper/example scripts

Some more scripts are included in the tools directory. Usage information can be obtained with the standard -h option.

Running a specific version of a container

1. Execute $ aws ecr list-images --region ca-central-1 --repository-name "<repository name>" for each of the “systems/dhdb”, “systems/dhdaemon”, and “systems/dhweb” repositories to view the available versions.

2. Note the desired version for each container.

3. Modify the tag (the version number) after the colon following the URI at services > <service> > image in docker-compose.yaml to match the desired version.

Setting the tag to ‘latest’ will cause docker compose to automatically download latest version of the container uploaded to the registry.

Uninstall

1. First, navigate to the Datahosting project root directory.

2. Remove the volumes with tools/removeVolumes.sh.

3. Remove the images with tools/removeImages.sh.

4. Uninstall Docker from the container host.

5. Delete the directory into which you copied the supporting files on the container host.