This vignette describes how to run roreviewapi on a
Digital Ocean droplet, including the editor volunteer search feature
introduced in v0.2. Endpoints for that feature are described in the editor
search vignette.
It covers two cases:
- Starting from scratch: setting up a new droplet, or a droplet that has been rebuilt.
- General maintenance: keeping a running droplet up, up to date, and in the right hands.
How the pieces fit together
A few terms used throughout this vignette:
-
Droplet: a virtual server rented from Digital
Ocean. It runs Linux, and you work on it by connecting from your own
computer with SSH (
ssh <user>@<droplet-ip>) or by using the web console button from Digital Ocean interface. -
Docker runs programs in isolated
containers. Each container is started from an
image, which bundles the program with everything it
needs. Docker Compose starts several containers
together, as described in
docker-compose.yml. -
DNS turns a name such as
reviewbot.ropensci.orginto the droplet’s IP address. rOpenSci’s DNS is managed in Cloudflare, which also sits between visitors and the droplet. - TLS certificate: the file that lets a server prove its identity, so that connections can use HTTPS. See TLS certificate.
- Cron runs commands on a schedule.
docker-compose.yml defines two containers:
-
plumberis the API itself. It is built from the repository’sDockerfile, and listens on port 8000, which is only reachable from the other container. -
nginxanswers on ports 80 (HTTP) and 443 (HTTPS), handles HTTPS with the certificate, redirects HTTP to HTTPS, and passes requests on toplumber.
So a request travels: visitor → Cloudflare → nginx (port
443) → plumber (port 8000).
Starting from scratch
Follow this section to set up a new droplet. It also applies when a droplet is rebuilt: Digital Ocean’s “Rebuild” reinstalls the operating system and erases the whole disk. Everything that is not in the Git repository is then lost, in particular:
- the
.envfile of secrets (see Environment variables), - the data directory
/srv/roreviewapi/data/, which holds the database and the Gmail credential files, - the TLS certificate and key in
/etc/ssl/cloudflare/, - the cron job that redeploys the service every week.
Prerequisites
You need:
- SSH access to the Digital Ocean droplet, with
sudorights. - Access to the AirTable base containing the editor-in-chief rotation table.
- Access to the Cloudflare account that manages the
ropensci.orgdomain, or someone who has it. - Access to rOpenSci’s 1Password, where the certificate’s private key is stored.
All commands below are run on the droplet, unless stated otherwise.
Install Git and Docker
Git downloads the code from GitHub. Install it with:
Install Docker Engine and its Compose plugin by following Docker’s instructions for Ubuntu. Then check that both work:
Docker must start automatically when the droplet boots, otherwise the service stays down after every reboot:
Set the Git user
Git records a name and an email address with every commit. Set them to the bot account, for root (which runs the weekly redeploy, see Cron job) and for your own user:
Get the code
The project lives in /home/shared/roreviewapi. Download
it there with:
sudo mkdir -p /home/shared
cd /home/shared
git clone https://github.com/ropensci-review-tools/roreviewapi.git
cd roreviewapiThe remaining commands that refer to files such as .env
or docker-compose.yml are run from this directory.
Environment variables
Secrets are not stored in the
Dockerfile or anywhere else in the Git repository. They
live in a .env file that exists only on the droplet. It is
listed in .gitignore and .dockerignore, and is
passed to the plumber container when it starts, via
env_file: .env in docker-compose.yml. Because
the file is not tracked by Git, git pull never overwrites
it, and the values never end up in an image.
Create .env in the project directory, with one
KEY=value entry per line (no quotes and no
export), and make it readable only by its owner:
| Variable | Description |
|---|---|
GITHUB_PAT |
GitHub personal access token (also used as
GITHUB_TOKEN) |
PKGCHECK_TOKEN |
pkgcheck authentication token: a secret password shared by the Digital Ocean droplet and buffy |
GMAIL_SENDER |
Sender address used for outgoing emails |
GMAIL_OAUTH_CLIENT_SECRET |
Path to the Google OAuth client secret JSON file,
e.g. /data/email/<client_secret>.json
|
GMAIL_TOKEN_CACHE |
Path to the cached Gmail token,
e.g. /data/email/gmailr_token
|
GMAIL_AUTH_EMAIL |
Google account used to authenticate with Gmail |
AIRTABLE_API_KEY |
AirTable personal access token |
AIRTABLE_BASE_ID |
ID of the AirTable base containing the EiC rotation table |
ROREVIEWAPI_BASE_URL |
Public HTTPS base URL of the deployed API
(https://reviewbot.ropensci.org) |
When the container starts, docker-entrypoint.sh writes
these values to ~/.Renviron (setting
GITHUB_TOKEN from GITHUB_PAT, and
ROREV_CONTAINER=true), so that R can read them, and then
starts the API.
The two GMAIL_* file paths must be inside
/data/email/. That is the container’s view of the data
directory (see Data directory). Place the
corresponding files in /srv/roreviewapi/data/ on the
droplet.
AirTable
- Generate a personal access token at https://airtable.com/create/tokens with at least the
data.records:readscope on the relevant base. Set this asAIRTABLE_API_KEY. - Open the AirTable base and copy the base ID from the URL
(
https://airtable.com/appXXXXXXXX/...). Set this asAIRTABLE_BASE_ID.
The editor-in-chief-rotation table within that base must
have period_start, period_end, and
acting_eic_email fields.
Data directory
The data directory holds the SQLite database, the notify-email cache file, and the Gmail OAuth client secret and token cache files. It lives on the droplet, outside the containers, so that it survives when the containers are rebuilt every week. Create it with:
docker-compose.yml makes it available inside the
plumber container as /data/email. The
container must be able to write to it, because the Gmail token cache is
updated during authentication.
After a rebuild, restore its contents, and .env, from a
backup (see Backing up the
data and secrets). Without a backup, the database starts empty, and
the Gmail authorization must be redone, following the editor
search vignette.
DNS
The domain’s DNS is managed in Cloudflare. It needs an A
record pointing the subdomain to the droplet’s IP address, for
example:
reviewbot.ropensci.org. IN A <droplet-ip>
The record must be proxied (orange cloud in the Cloudflare dashboard), not “DNS only”. The TLS certificate below is a Cloudflare Origin Certificate, which only Cloudflare trusts, so visitors must reach the droplet through Cloudflare.
A rebuilt droplet keeps its IP address, so the record does not change. A new droplet has a new IP address, so the record must be updated.
TLS certificate
TLS uses a Cloudflare Origin Certificate. It secures the connection between Cloudflare and the droplet, is valid for up to 15 years, and needs no renewal on the droplet.
In the Cloudflare dashboard for the domain, go to SSL/TLS → Origin Server → Create Certificate. Keep the default key type, list the hostname (
reviewbot.ropensci.org), and choose a validity period.-
Cloudflare shows the certificate and the private key. Cloudflare does not store the private key, so it is only shown once: save it straight away. A copy of the current key is stored in rOpenSci’s 1Password. On the droplet, save them as
/etc/ssl/cloudflare/origin.pemand/etc/ssl/cloudflare/origin.key.Create the directory and the certificate file.
teewaits for input: paste the certificate, press Enter, then Ctrl+D to save:Do the same for the private key, then make it readable only by root:
sudo teeis needed because insudo echo ... > filethe redirection runs as the current user, not as root. Sending the output to/dev/nullkeeps the pasted key from being printed back to the terminal.Check that both files are valid:
sudo openssl x509 -in /etc/ssl/cloudflare/origin.pem -noout -subject -enddate sudo openssl pkey -in /etc/ssl/cloudflare/origin.key -noout && echo "key OK"The first command shows the hostname and the expiry date. The second prints “key OK” if the key can be read.
In SSL/TLS → Overview, set the encryption mode to Full (strict).
After a rebuild, there is no need to create a new certificate. Copy
the key back from 1Password, and download the certificate again from the
Cloudflare dashboard (SSL/TLS → Origin Server). Save
both with the tee commands above.
Start the service
From the project directory, build the images and start the containers in the background:
The first build takes a while. nginx only starts once
plumber reports itself as healthy, which can take a few
minutes.
The compose file:
- Mounts
/srv/roreviewapi/datainto theplumbercontainer at/data/email, and setsROREVIEWAPI_EMAIL_DB=/data/email/searches.sqlite, so that the SQLite database persists across rebuilds. - Reads secrets from
.envviaenv_file. - Mounts
/etc/ssl/cloudflareread-only into thenginxcontainer, so that the certificate is available. - Sets the domain name (
NGINX_SERVER_NAME), which is substituted intonginx.confwhen the container starts. - Exposes ports 80 (HTTP → HTTPS redirect) and 443 (HTTPS).
Cron job
restart.sh downloads new commits and rebuilds the
service (see Weekly redeploy). A cron job
runs it every week. One authorized admin of the droplet needs to set it
up.
The job must run as a user that can run docker compose
(either root or a member of the docker group). The
following uses root’s crontab, so that the log file below can be written
to /var/log.
Check whether a job already exists (see Viewing, changing and checking the cron job), to avoid creating a duplicate.
-
Open the crontab for editing. If prompted, choose an editor (
nanois the easiest): -
Add the following line:
This runs the script at 00:00 every Tuesday. The five leading fields are minute, hour, day of month, month, and day of week (0–7, where both 0 and 7 are Sunday). crontab.guru is useful for checking a schedule. The trailing redirect appends all output, including errors, to a log file.
Save and exit. Cron picks up the change automatically; no restart of the cron service is needed.
An absolute path is required because cron does not run in the project
directory. restart.sh finds the project directory (and thus
.env) from its own location, so it needs no other
configuration.
Check that it works
Both containers should be running, with plumber marked
healthy, and curl should print
200. Otherwise, see Troubleshooting.
Finally, set up the uptime alert if it does not already exist.
General maintenance
Uptime alert
The Software Peer Review Lead should set up a regular alert to check that the service is up. This bash script prints a message and fails if anything is wrong:
#!/usr/bin/env bash
URL="https://reviewbot.ropensci.org"
status=$(curl -s -o /dev/null -w "%{http_code}" "$URL")
if [[ "$status" != "404" ]]; then
echo -e "reviewbot is down"
exit 1
fiIt should then be run regularly, for example from a
~/.bashrc file, or from a cron job.
Weekly redeploy
The cron job runs restart.sh
every Tuesday. The script:
- downloads new commits with
git pull --ff-only, - stops the containers,
- deletes old images,
- rebuilds the images from scratch,
- starts the containers again.
So changes pushed to GitHub reach the droplet within a week, and the images pick up updated R packages. The service is down while the images rebuild.
To redeploy straight away, for example after pushing a fix, run the script by hand:
Viewing, changing and checking the cron job
List the jobs in root’s crontab with:
Omit sudo to list the current user’s own crontab
instead. Each user has a separate crontab, so if nothing appears, check
the other user (or root) that may have been used to set the job up.
To change the job, re-open the crontab with
sudo crontab -e, edit the line (for example, change the
schedule fields), and save. To disable the job without deleting it,
comment the line out by prefixing it with #. To remove it,
delete the line. Confirm the result with
sudo crontab -l.
To check that the job ran, read the end of its log file:
and confirm that the containers are up with
docker compose ps.
Restarting the service
To restart the containers without rebuilding them, for example after
changing .env or installing a new certificate, run from the
project directory:
To change a secret, edit .env on the droplet and
restart. No change to the Dockerfile is needed.
Backing up the data and secrets
The data directory (/srv/roreviewapi/data/) and the
.env file are the only things on the droplet that cannot be
recreated from GitHub or Cloudflare. Keep a copy of them off the
droplet, and refresh it before risky changes such as an operating system update, and
after changing .env or redoing the Gmail authorization.
The backup contains secrets, so store it somewhere safe, such as rOpenSci’s 1Password, rather than in email, chat or a shared drive.
Downloading a backup
-
On the droplet, stop the API briefly, so that the database is not being written to while it is copied:
-
Pack the data directory and
.envinto one archive in your home directory, make it yours so that it can be downloaded, and start the API again:sudo tar -czf ~/roreviewapi-backup.tar.gz -C / srv/roreviewapi/data home/shared/roreviewapi/.env sudo chown $USER ~/roreviewapi-backup.tar.gz docker compose start plumber-C /makes the paths in the archive start from the root of the disk, so that they can be restored to the same places. -
On your own computer, download the archive with
scp, which copies files over SSH.<user>and<droplet-ip>are the same as forssh: -
Back on the droplet, delete the archive, so that no extra copy of the secrets is left behind:
To check what an archive contains, without unpacking it, run
tar -tzf roreviewapi-backup.tar.gz.
Restoring a backup
On a rebuilt droplet, restore the backup after getting the code, and before starting the service.
-
On your own computer, upload the archive to your home directory on the droplet:
-
On the droplet, unpack it, which puts the files back in
/srv/roreviewapi/data/and/home/shared/roreviewapi/.env, then delete the archive:Unpacking with
sudokeeps the original owners and permissions, including the restricted permissions of.envand of the data directory. If the service is already running, restart it so that it picks up the restored files.
Updating the operating system
Update the droplet’s operating system every 2 years. Ubuntu publishes a new long-term support (LTS) release every 2 years, and older releases eventually stop receiving security fixes. Check the current release with:
-
Download a backup of
the data directory and
.envfirst, because rebuilding erases the whole disk. - Use the rebuild feature in Settings of the droplet.
- Set the droplet up again by following Starting from scratch, and restore the backup.
Recreating the certificate
Create a new certificate when the current one is about to expire,
when the key is lost, or when it may have been exposed. Nothing renews
the certificate automatically, so check its expiry date with the
openssl x509 command in TLS
certificate. The current certificate expires on 16 October 2041.
Create a new certificate and install it on the droplet, following steps 1 and 2 of TLS certificate. Replace the key in 1Password with the new one.
-
Restart
nginxso that it loads the new files, and check that the site responds: Revoke the old certificate in SSL/TLS → Origin Server, so that the old key can no longer be used.
Cloudflare’s Origin CA documentation describes creating, installing and revoking origin certificates in more detail.
Troubleshooting
Cloudflare reports problems with the droplet as status codes:
| Status | Likely cause |
|---|---|
| 521 | Nothing answers on port 443: nginx or the whole stack is down, or nginx cannot find the certificate files and exits on start |
| 522 | The droplet is unreachable, for example a firewall blocks port 443 |
| 526 | The certificate is invalid, expired, or does not match the hostname |
| 502 | nginx is up but plumber is not |
For 521 or 502, check the containers and their logs on the droplet:
docker compose ps
docker compose logs --tail=50 nginx
docker compose logs --tail=50 plumber
sudo ls /etc/ssl/cloudflare/For 522, there are two independent firewalls to check.
Droplet firewall (ufw):
If ufw is active and port 443 is not listed, open it:
Digital Ocean cloud firewall: in the Digital Ocean console under Networking → Firewalls, confirm there is an inbound rule allowing TCP port 443. The cloud firewall sits in front of the droplet and blocks traffic before it reaches ufw, so both layers must allow it.
Optionally, Authenticated Origin Pulls make nginx accept only connections that come through Cloudflare.
Transferring maintenance
Maintenance falls into two parts with very different transfer costs:
-
Operations (deploying, debugging, restarting):
needs access only. Nothing in
.envchanges. - Credentials tied to an individual: the Gmail authorization, and the GitHub and AirTable tokens, belong to whoever created them. These must be re-created by the new maintainer if the outgoing person’s accounts will be closed.
Operations only
When the new person only deploys and debugs, and the existing authorizing account stays active:
- Add their SSH key and
sudoaccess on the droplet. - Add them to the
roreviewapiGitHub repository. - Add them to the
ro-auto-emailGoogle Cloud project’s IAM. - Hand over this document and the contents of
.envsecurely, for example via a password manager rather than email or chat.
Transferring credentials
This is required if the outgoing maintainer’s accounts will become
unusable, for example on leaving the organization. The values in
.env that are tied to a person are:
| Variable | Tied to | Action on transfer |
|---|---|---|
GITHUB_PAT |
GitHub account that created it | Replace with a token created by the new maintainer |
AIRTABLE_API_KEY |
AirTable account that created it | Replace with a token created by the new maintainer (see AirTable for the required scope) |
GMAIL_AUTH_EMAIL, GMAIL_TOKEN_CACHE
|
Google account that authorized gmailr
|
Replace, following the Gmail transfer steps in the editor search vignette |
GMAIL_OAUTH_CLIENT_SECRET,
GMAIL_SENDER
|
Google Cloud project and the reviewbot@ropensci.org
alias |
Unchanged if the same Cloud project and alias are kept |
AIRTABLE_BASE_ID, ROREVIEWAPI_BASE_URL,
PKGCHECK_TOKEN
|
Not tied to a person | Unchanged unless the base, URL or pkgcheck service changes |
The steps are then:
The new maintainer creates their own GitHub and AirTable personal access tokens, and completes the Gmail steps in the editor search vignette.
Update
.envon the droplet with the new values.Restart the service so that the container picks up the new
.envand credentials.Verify that the editor search workflow still sends and receives emails (see the editor search vignette).
Once the new setup works, the outgoing maintainer should revoke their old GitHub and AirTable tokens, and the old Google grant.
Other items to update
- Cron job. If the cron job was set up under the outgoing maintainer’s account, move it to root’s crontab (see Cron job), or recreate it under the new account.
-
Git identity. The
git configlines in theDockerfilehold theuser.nameanduser.emailused for commits made inside the container. - Uptime alert. Whoever receives the uptime alert should be updated.
- 1Password. The new maintainer needs access to the certificate’s private key.
