Troubleshooting
Backing up the database
docker-compose.yaml
(under services > dhdb > volumes
) configures Docker to create the mariadb-backups
directory in the host root, bound to /mariadb-backups/
in the database container. This is intended to be a convenient way of moving backups in and out of the container.
Backup and restore scripts are bundled in the database container for convenience. These generate and execute - respectively - a series of SQL INSERT statements that creates and populates a duplicate of every table in the database. With the dhdb container running, may backup the database as follows: $ docker exec <db container name> /etc/dhdb/backup-to-sql.sh
In case you need to perform a backup operation manually, you may find more information at https://mariadb.com/kb/en/backup-and-restore-overview.
Caution: The MariaDB documentation contains procedures for backing up the entire database directory. This is not recommended, as restoring a backup from a previous version may introduce a conflict with the database version migration system. Instead, whenever possible, we recommend backing up the the database as SQL statements. To save on disk space, use gzip’s ‘-9' argument to compress the data more aggressively.
Restoring the database
First, navigate to the Datahosting project root directory.
To prevent database resource conflicts, stop the other rbr containers before restoring the database.
$ docker container stop rbr-dhweb-1
$ docker container stop rbr-dhdaemon-1
$ docker exec -i rbr-dhdb-1 /etc/dhdb/restore-from-sql.sh /mariadb-backups/<backup-name>.sql
$ docker container start rbr-dhweb-1
$ docker container start rbr-dhdaemon-1
Accessing the database manually
$ docker exec -it rbr-dhdb-1 mariadb -p
Enter the root password.
To access the Datahosting database, enter
> using datahosting
Accessing the shell
$ docker <container name> exec -it bash
Fixing port conflicts
If another application (e.g. a web server) on the host is already using a port used by the Datahosting application (e.g. port 80), the containers will fail to run with an error such as “address already is in use“.
The port mappings are defined in docker-compose.yaml
as services > <service> > ports > <host port>:<container port>
. The container ports should remain unchanged, but the host port should be changed to a different port (e.g. port 8080). Note that any external network configurations, RBRcervello configurations, or service configurations referencing the original port number will have to be updated as well.
Fixing restarting containers
If a container fails to start, view and save a copy of the logs, and include them in any emails to RBR.
If dhdaemon is continually restarting, the most likely problem is a misconfiguration causing a JSON file to fail validation. Check secrets/web-user-config.json
and daemon/database-config/customer-config.json
files against their respective secrets/web-user-config-schema.json
and daemon/database-config/customer-config-schema.json
schemas using https://www.jsonschemavalidator.net/.
Fixing communication with instruments
If an instrument cannot communicate with the Datahosting service or isn’t appearing in the web interface, please verify that:
The instrument is assigned the correct serial number in
customer-configuration.json
.The RBRcervello’s
user.json
configuration has the correct destination address.Port 9140 on the container host (or the port configured to map to dhdaemon’s port 9140 in
docker-compose.yaml
) is exposed to the internet and not blocked by a firewall.
Viewing the logs
You may view a container’s terminal output with docker logs <container name>
or view within it files individually with docker <container name> exec <command> <options>
, where <command>
is any of the commonly used file access programs (e.g. ls
, cat
, tail
, head
, less
). To back up these files, you may redirect the output of docker exec
to a file on the container host.
Daemon
The daemon logs are stored in a Docker volume that is only accessible within the containers. Logs are stored at /var/log/dhdaemon
.
Web host
The web host does not output anything to the terminal by default. Logs can be found at /var/log/nginx/access.log
, /var/log/nginx/error.log
, and /var/log/php7.2-fpm.log
.
Helper/example scripts
Some more scripts are included in the tools
directory. Usage information can be obtained with the standard -h
option.
Running a specific version of a container
Execute
$ aws ecr list-images --region ca-central-1 --repository-name "<repository name>"
for each of the “systems/dhdb”, “systems/dhdaemon”, and “systems/dhweb” repositories to view the available versions.Note the desired version for each container.
Modify the tag (the version number) after the colon following the URI at
services > <service> > image
indocker-compose.yaml
to match the desired version.
Setting the tag to ‘latest’ will cause docker compose to automatically download latest version of the container uploaded to the registry.
Uninstall
First, navigate to the Datahosting project root directory.
Remove the volumes with
tools/removeVolumes.sh
.Remove the images with
tools/removeImages.sh
.Uninstall Docker from the container host.
Delete the directory into which you copied the supporting files on the container host.