Updated load_production_data script to default download hyperkitty data, related docs and minor tweaks.

Author: daveoconnor

README.md

@@ -33,6 +33,7 @@ There are two options for development setups, nix and native. The native setup d
 
 
 1. Development System setup
+   1. Give your ssh key to Sam so he can add it to the boost.cpp.al server.
    1. Basic Setup Options:
       1. [Nix-based](docs/development_setup_notes_nix.md)
       1. [Native](docs/development_setup_notes_native.md)
@@ -137,6 +138,9 @@ currently generated by `docker compose build` and is included in the Docker imag
 ---
 
 ## Setting up Mailman/Hyperkitty locally
+
+Note: This is optional - not required for the website to function, here for reference in case a need arises.
+
 ``shell
 sudo apt-get install sassc
 git clone [email protected]:mailman/hyperkitty.git
@@ -147,18 +151,10 @@ pip install psycopg2-binary
 ``
 change settings.py to use postgres database:
         'ENGINE': 'django.db.backends.postgresql_psycopg2',
-Update database values in settings to use the same host, user, password, and the database name as in the .env file value for `HYPERKITTY_DATABASE_NAME`.
+Update database values in settings to use the same host, user, password, and the database name as in the .env file value for `HYPERKITTY_DATABASE_NAME` (`lists_production_web` by default).
 
 run `django-admin migrate --pythonpath example_project --settings settings`
 
-Give your ssh key to Sam so he can add it to the boost.cpp.al server, and then download the mailman db archive and cp the sql to the docker container
-```shell
-scp {user}@staging-db1.boost.cpp.al:/tmp/lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql.gz .
-docker cp lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql website-v2-web-1:/lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql
-docker exec -it website-v2-web-1 /bin/bash
-psql -U postgres -W hyperkitty_db < /lists_stage_web.staging-db1-2.2025-02-06-08-00-01.sql
-```
-
 ## Syncing EmailData Locally
 
 To work with mailinglist data locally, the django application expects to be

docs/development_setup_notes_nix.md

@@ -36,7 +36,7 @@ For a basic rundown on Nix, this video could be useful https://www.youtube.com/w
    * STATIC_CONTENT_AWS_ACCESS_KEY_ID - ask for this
    * STATIC_CONTENT_AWS_SECRET_ACCESS_KEY - ask for this
 5. Run `just setup` to build services, and build the JS and CSS assets. If docker fails with permissions errors, reboot your machine.
-6. Run `just load_production_data` to download live data from the backup server.
+6. Run `just load_production_data` to download live data from the backup server for the website and mailman/hyperkitty. That script can be run regularly to update data from the live servers. You will lose all local data.
 7. Run `docker compose up` to start the server.
 
 

docs/first_time_data_import.md

@@ -8,10 +8,13 @@ This document contains information about importing Boost Versions (also called R
 
 ## Development Environments
 
+Download the production data for the website and mailman/hyperkitty.
 ```bash
 just load_production_data
 ````
 
+That script can be run regularly to update data from the live servers. You will lose all local data.
+
 ## Deployed Environments
 
 There are several steps to populating the database with historical Boost data, because we retrieve Boost data from multiple sources.

env.template

@@ -33,7 +33,7 @@ PROD_MEDIA_CONTENT_REGION=$STATIC_CONTENT_REGION
 PROD_MEDIA_CONTENT_AWS_S3_ENDPOINT_URL=$STATIC_CONTENT_AWS_S3_ENDPOINT_URL
 
 # Mailman database settings
-HYPERKITTY_DATABASE_NAME="hyperkitty_db"
+HYPERKITTY_DATABASE_NAME="lists_production_web"
 DATABASE_URL="postgresql://postgres@db:5432/postgres"
 DATABASE_TYPE="postgres"
 DATABASE_CLASS="mailman.database.postgresql.PostgreSQLDatabase"

scripts/load_production_data.sh

@@ -11,33 +11,33 @@ set -eu
 #
 
 # READ IN COMMAND-LINE OPTIONS
-TEMP=$(getopt -o h:: --long help::,lists::,only-lists:: -- "$@")
+TEMP=$(getopt -o h:: --long help::,no-web::,no-lists:: -- "$@")
 eval set -- "$TEMP"
 
 # extract options and their arguments into variables.
 while true ; do
     case "$1" in
         -h|--help)
             helpmessage="""
-usage: load_production_data.sh [-h] [--lists] [--only-lists]
+usage: load_production_data.sh [-h] [--no-web] [--no-lists]
 
-Load production data. By default this will import the main website database.
+Load production data. By default this will import both the main website database and mailing list databases.
 
 optional arguments:
   -h, --help            Show this help message and exit
-  --lists               Import mailing list dbs also.
-  --only-lists		Import mailing list database and not the main web database.
+  --no-web              Skip importing the main website database.
+  --no-lists            Skip importing mailing list databases.
 """
 
             echo ""
             echo "$helpmessage" ;
             echo ""
             exit 0
             ;;
-        --lists)
-            lists_option="yes" ; shift 2 ;;
-        --only-lists)
-            lists_option="yes" ; skip_web_option="yes" ; shift 2 ;;
+        --no-web)
+            skip_web_option="yes" ; shift 2 ;;
+        --no-lists)
+            skip_lists_option="yes" ; shift 2 ;;
         --) shift ; break ;;
         *) echo "Internal error!" ; exit 1 ;;
     esac
@@ -201,7 +201,7 @@ if [ "${skip_web_option:-}" != "yes" ]; then
     }
 fi
 
-if [ "${lists_option:-}" = "yes" ]; then
+if [ "${skip_lists_option:-}" != "yes" ]; then
     download_latest_db_dump lists_web_db || {
         echo "Failed to download and restore latest lists_web_db dump";
         exit 1;