Skip to content

seqr deployment fails; timeouts. Centos 7.8 #1409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jbsysadmin opened this issue Aug 4, 2020 · 25 comments
Closed

seqr deployment fails; timeouts. Centos 7.8 #1409

jbsysadmin opened this issue Aug 4, 2020 · 25 comments
Labels

Comments

@jbsysadmin
Copy link

jbsysadmin commented Aug 4, 2020

Greetings. With a file docker-compose.yml in place, and python 3.8.1 / docker compose in path, I follow these steps:
sudo docker-compose up -d seqr
sudo /usr/local/docker-compose/docker-compose up -d seqr

However, reviewing those logs confirms that the first step never actually finishes, despite the promising output below.

Creating network "app_datasets_default" with the default driver
Creating app_datasets_redis_1         ... done
Creating app_datasets_elasticsearch_1 ... done
Creating app_datasets_postgres_1      ... done
Creating app_datasets_seqr_1          ... done

The logs show that all net-related download actions within the docker are timing out. Example:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f0ab3876450>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/django/

Adding proxy info to a config ( /root/.docker/config.json, reference: https://medium.com/@airman604/getting-docker-to-work-with-a-proxy-server-fadec841194e) will not avail. Everything halts with thingvs like"Container "4e19a6c5f903" is unhealthy."

In addition to the timeouts, I see mention of a non-existent host postgres. I assume that was to have been created within the docker environment:
psycopg2.OperationalError: could not translate host name "postgres" to address: Name or service not known

The environment is an oVirt VM running Centos 7.8.
Do you have any leads on the conditions needed to get seqr running? We there special firewall exceptions needed outside of docker?

@jbsysadmin jbsysadmin added the bug label Aug 4, 2020
@jbsysadmin
Copy link
Author

I have some findings to add:

  • The setting COMPOSE_HTTP_TIMEOUT had no effect at value "750"

  • This is the point in logs at which the timeouts start:

+ git pull
seqr_1             | fatal: unable to access 'https://github.com/macarthur-lab/seqr/': Failed to connect to github.com port 443: Connection timed out
seqr_1             | + git checkout dev
  • This is the point after which all future statements time out:
 + pip install --upgrade -r requirements.txt
seqr_1             | WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5b5c3b0410>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/django/
  • The commands docker-compose stop and docker-compose start result in:
Starting postgres        ... done
Starting redis           ... done
Starting elasticsearch   ... done
Starting kibana          ... failed
Starting seqr            ... error
Starting pipeline-runner ... failed

@jbsysadmin
Copy link
Author

Does anyone have a lead on seqr is supposed to run from behind a department proxy and port? That may be the issue here.

@bw2
Copy link
Contributor

bw2 commented Aug 11, 2020

Thanks for troubleshooting this further, as I wasn't able to reproduce the issue here.
There does need to be network access from the machine that's running docker to urls like https://github.com/macarthur-lab/seqr.

Also, we just updated the docker-compose config to work with the latest postgres image.
If you are able to work around the proxy issues and retry, please update your local copy of docker-compose.yml

@jbsysadmin
Copy link
Author

In docker-compose.yml, I was able to progress, by adding proxy statements "https_proxy" and "http.proxy" (for git, as we need to do that for outbound git requests.) Furthermore, I was able to get past the first postgres error, per this reference: docker-library/postgres#681

However, now the step "docker-compose exec seqr python manage.py createsuperuser" fails, with:

psycopg2.OperationalError: could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 5432?

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

Hmm.. I'm not seeing this issue with the latest docker-compose.yml. I get:

~/code/seqr 1946 0 $ docker-compose exec seqr python manage.py createsuperuser
Username (leave blank to use 'root'): bw
Email address: [email protected]
Password:
Password (again):
Superuser created successfully.
~/code/seqr 1947 0 $

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

When I run docker-compose logs postgres, it shows:

~/code/seqr 1951 0 $ docker-compose logs postgres
Attaching to seqr_postgres_1
postgres_1         |
postgres_1         | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1         |
postgres_1         | 2020-08-11 15:38:53.136 UTC [1] LOG:  starting PostgreSQL 12.3 (Debian 12.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1         | 2020-08-11 15:38:53.137 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1         | 2020-08-11 15:38:53.137 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1         | 2020-08-11 15:38:53.141 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1         | 2020-08-11 15:38:53.159 UTC [25] LOG:  database system was shut down at 2020-08-11 15:38:52 UTC
postgres_1         | 2020-08-11 15:38:53.165 UTC [1] LOG:  database system is ready to accept connections

and for docker-compose logs seqr:

~/code/seqr 1952 0 $ docker-compose logs seqr
Attaching to seqr_seqr_1
seqr_1             | + REFERENCE_DATA_DB_INIT_URL=https://storage.googleapis.com/seqr-reference-data/gene_reference_data_backup.gz
seqr_1             | + env
seqr_1             | SEQR_SERVICE_PORT=8000
seqr_1             | POSTGRES_SERVICE_HOSTNAME=postgres
seqr_1             | HOSTNAME=2032c57ee375
seqr_1             | SEQR_GIT_BRANCH=dev
seqr_1             | PWD=/seqr
seqr_1             | HOME=/root
seqr_1             | REDIS_SERVICE_HOSTNAME=redis
seqr_1             | STATIC_MEDIA_DIR=/seqr_static_files
seqr_1             | PGHOST=postgres
seqr_1             | TERM=xterm
seqr_1             | PGUSER=postgres
seqr_1             | POSTGRES_PASSWORD=docker-compose-postgress-password
seqr_1             | SHLVL=1
seqr_1             | PYTHONPATH=/seqr
seqr_1             | GUNICORN_WORKER_THREADS=4
seqr_1             | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
seqr_1             | KIBANA_SERVICE_HOSTNAME=kibana
seqr_1             | ELASTICSEARCH_SERVICE_HOSTNAME=elasticsearch
seqr_1             | _=/usr/bin/env
seqr_1             | SHELL: /bin/bash
seqr_1             | PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
seqr_1             | PYTHONPATH: /seqr
seqr_1             | + echo SHELL: /bin/bash
seqr_1             | + echo PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
seqr_1             | + echo PYTHONPATH: /seqr
seqr_1             | + '[' ']'
seqr_1             | + '[' ']'
seqr_1             | + '[' -e /.config/service-account-key.json ']'
seqr_1             | + mkdir -p /seqr_static_files/generated_files
seqr_1             | + cd /seqr
seqr_1             | + '[' dev ']'
seqr_1             | + git pull
seqr_1             | From https://github.com/macarthur-lab/seqr
seqr_1             |    28938aa49..bd9b316ea  master                -> origin/master
seqr_1             |  * [new branch]          anvil-update          -> origin/anvil-update
seqr_1             |    bc8067733..c39bb84fd  dev                   -> origin/dev
seqr_1             |  * [new branch]          dev-mirror            -> origin/dev-mirror
seqr_1             |  * [new branch]          fix-cron              -> origin/fix-cron
seqr_1             |  * [new branch]          fix-individual-update -> origin/fix-individual-update
seqr_1             |  * [new branch]          fix-migrations        -> origin/fix-migrations
seqr_1             |  * [new branch]          fix-pedigree-image0update -> origin/fix-pedigree-image0update
seqr_1             |  * [new branch]          fix-recessive-pagination -> origin/fix-recessive-pagination
seqr_1             |  * [new branch]          fix-reference-data-update -> origin/fix-reference-data-update
seqr_1             |  * [new branch]          fix-typo              -> origin/fix-typo
seqr_1             |  * [new branch]          google-auth           -> origin/google-auth
seqr_1             |  * [new branch]          mme-updates           -> origin/mme-updates
seqr_1             |  * [new branch]          postgres-upgrade      -> origin/postgres-upgrade
seqr_1             |  * [new branch]          redis-upgrade         -> origin/redis-upgrade
seqr_1             |  * [new branch]          remove-dependency     -> origin/remove-dependency
seqr_1             |  * [new branch]          remove-pthon2         -> origin/remove-pthon2
seqr_1             |  * [new branch]          seqr-matchmaker-link  -> origin/seqr-matchmaker-link
seqr_1             |  * [new branch]          sv-tag-types          -> origin/sv-tag-types
seqr_1             |  * [new branch]          upgrade-es            -> origin/upgrade-es
seqr_1             | Updating 28938aa49..bd9b316ea
seqr_1             | Fast-forward
seqr_1             |  .travis.yml                                        |  12 +-
seqr_1             |  collect_static.sh                                  |   5 -
seqr_1             |  deploy/MIGRATE.md                                  |  10 +-
seqr_1             |  deploy/docker/postgres/Dockerfile                  |   2 +-
seqr_1             |  deploy/docker/redis/Dockerfile                     |   2 +-
seqr_1             |  deploy/docker/seqr/Dockerfile                      |   4 +-
seqr_1             |  deploy/docker/seqr/entrypoint.sh                   |   8 +-
seqr_1             |  deploy/kubernetes/postgres/postgres.gcloud.yaml    |   6 +-
seqr_1             |  deploy/servctl_utils/deploy_command_utils.py       |  94 +++---
seqr_1             |  deploy/servctl_utils/kubectl_utils.py              | 167 +++++++++
seqr_1             |  deploy/servctl_utils/other_command_utils.py        |   3 +-
seqr_1             |  matchmaker/apps.py                                 |   2 -
seqr_1             |  matchmaker/matchmaker_utils.py                     |   6 +-
seqr_1             |  matchmaker/migrations/0001_initial.py              |   6 +-
seqr_1             |  ...0002_matchmakerresult_originating_submission.py |  35 ++
seqr_1             |  matchmaker/models.py                               |   3 +-
seqr_1             |  matchmaker/views/external_api.py                   |   9 +-
seqr_1             |  ...rnal_api_2_3_tests.py => external_api_tests.py} |   2 -
seqr_1             |  matchmaker/views/matchmaker_api.py                 |  31 +-
seqr_1             |  ...er_api_2_3_tests.py => matchmaker_api_tests.py} |  68 +++-
seqr_1             |  reference_data/apps.py                             |   2 -
seqr_1             |  .../management/commands/utils/download_utils.py    |  11 +-
seqr_1             |  ..._utils_2_3_tests.py => download_utils_tests.py} |  18 +-
seqr_1             |  ...=> run_reference_data_database_backup_tests.py} |   0
seqr_1             |  ...tests.py => update_all_reference_data_tests.py} |   0
seqr_1             |  ...ne_2_3_tests.py => update_dbnsfp_gene_tests.py} |  38 +--
seqr_1             |  ...encode_2_3_tests.py => update_gencode_tests.py} |  41 ++-
seqr_1             |  ..._3_tests.py => update_gene_constraint_tests.py} |  39 +--
seqr_1             |  ...update_hpo_2_3_tests.py => update_hpo_tests.py} |  43 +--
seqr_1             |  ...update_mgi_2_3_tests.py => update_mgi_tests.py} |  39 +--
seqr_1             |  ...date_omim_2_3_tests.py => update_omim_tests.py} |  62 ++--
seqr_1             |  ..._ai_2_3_tests.py => update_primate_ai_tests.py} |  37 +-
seqr_1             |  reference_data/migrations/0001_initial.py          |   4 +-
seqr_1             |  .../migrations/0002_auto_20170319_0332.py          |   2 -
seqr_1             |  reference_data/migrations/0003_clinvar.py          |   2 -
seqr_1             |  .../migrations/0004_auto_20170626_1401.py          |   2 -
seqr_1             |  .../migrations/0005_auto_20170627_0318.py          |   2 -
seqr_1             |  .../migrations/0006_auto_20170630_1754.py          |   2 -
seqr_1             |  .../migrations/0007_auto_20180809_2053.py          |   2 -
seqr_1             |  reference_data/migrations/0008_geneexpression.py   |   2 -
seqr_1             |  .../migrations/0009_auto_20180917_2015.py          |   2 -
seqr_1             |  .../migrations/0010_auto_20190319_1518.py          |   4 +-
seqr_1             |  reference_data/migrations/0011_primateai.py        |   2 -
seqr_1             |  .../migrations/0012_auto_20190321_1656.py          |   2 -
seqr_1             |  .../migrations/0013_auto_20190404_1953.py          |   2 -
seqr_1             |  .../migrations/0014_auto_20190510_2150.py          |   2 -
seqr_1             |  .../migrations/0015_auto_20190625_1813.py          |   2 -
seqr_1             |  .../migrations/0016_dbnsfpgene_gene_names.py       |   2 -
seqr_1             |  .../migrations/0017_auto_20190909_2102.py          |   2 -
seqr_1             |  requirements.txt                                   |   4 +-
seqr_1             |  seqr/README.md                                     |   2 +-
seqr_1             |  seqr/admin.py                                      |   2 -
seqr_1             |  seqr/apps.py                                       |   2 -
seqr_1             |  seqr/fixtures/1kg_project.json                     |   2 +
seqr_1             |  seqr/fixtures/new_variant_tag_types.json           |  33 +-
seqr_1             |  seqr/fixtures/variant_tag_types.json               |  47 +++
seqr_1             |  seqr/management/commands/add_project_tag.py        |   2 -
seqr_1             |  seqr/management/commands/check_bam_cram_paths.py   |   3 -
seqr_1             |  seqr/management/commands/copy_project_tags.py      |   2 -
seqr_1             |  seqr/management/commands/lift_project_to_hg38.py   |   2 -
seqr_1             |  seqr/management/commands/lift_variant_to_hg38.py   |   2 -
seqr_1             |  .../commands/reload_saved_variant_json.py          |   2 -
seqr_1             |  seqr/management/commands/resend_welcome_email.py   |   2 -
seqr_1             |  .../commands/run_postgres_database_backup.py       |   2 -
seqr_1             |  seqr/management/commands/run_settings_backup.py    |   2 -
seqr_1             |  .../transfer_families_to_different_project.py      |   2 -
seqr_1             |  ...t_tag_2_3_tests.py => add_project_tag_tests.py} |   0
seqr_1             |  ..._2_3_tests.py => check_bam_cram_paths_tests.py} |   2 -
seqr_1             |  ...ags_2_3_tests.py => copy_project_tags_tests.py} |   3 -
seqr_1             |  ..._2_3_tests.py => lift_project_to_hg38_tests.py} |   3 -
seqr_1             |  ..._2_3_tests.py => lift_variant_to_hg38_tests.py} |   3 -
seqr_1             |  ...tests.py => reload_saved_variant_json_tests.py} |   2 -
seqr_1             |  ..._2_3_tests.py => resend_welcome_email_tests.py} |   3 -
seqr_1             |  .../tests/reset_cached_search_results_tests.py     |   1 -
seqr_1             |  ...ts.py => run_postgres_database_backup_tests.py} |   2 -
seqr_1             |  ...p_2_3_tests.py => run_settings_backup_tests.py} |   2 -
seqr_1             |  ...ransfer_families_to_different_project_tests.py} |   2 -
seqr_1             |  ..._remove_project_custom_reference_populations.py | 178 +++++-----
seqr_1             |  seqr/migrations/0002_auto_20191203_1913.py         |   2 -
seqr_1             |  seqr/migrations/0003_auto_20191203_1130.py         |   2 -
seqr_1             |  seqr/migrations/0004_auto_20200124_1912.py         |   2 -
seqr_1             |  seqr/migrations/0005_locuslist_projects.py         |   1 -
seqr_1             |  seqr/migrations/0006_family_mme_notes.py           |   2 -
seqr_1             |  seqr/migrations/0007_auto_20200303_1842.py         |   4 +-
seqr_1             |  seqr/migrations/0008_auto_20200317_1716.py         |   5 +-
seqr_1             |  seqr/migrations/0009_auto_20200402_2219.py         |   3 +-
seqr_1             |  seqr/migrations/0010_auto_20200413_2159.py         |  25 +-
seqr_1             |  .../0011_individual_proband_relationship.py        |  13 +-
seqr_1             |  seqr/migrations/0012_auto_20200603_1924.py         |   2 -
seqr_1             |  seqr/migrations/0013_individual_sv_flags.py        |   2 -
seqr_1             |  seqr/models.py                                     |   2 -
seqr_1             |  seqr/urls.py                                       |   2 -
seqr_1             |  seqr/utils/communication_utils.py                  |   2 -
seqr_1             |  seqr/utils/elasticsearch/constants.py              |   3 -
seqr_1             |  seqr/utils/elasticsearch/es_gene_agg_search.py     |   2 -
seqr_1             |  seqr/utils/elasticsearch/es_search.py              |  14 +-
seqr_1             |  .../{es_utils_2_3_tests.py => es_utils_tests.py}   |   3 -
seqr_1             |  seqr/utils/elasticsearch/utils.py                  |   2 -
seqr_1             |  seqr/utils/file_utils.py                           |   2 -
seqr_1             |  seqr/utils/gene_utils.py                           |   2 -
seqr_1             |  seqr/utils/middleware.py                           |   5 +-
seqr_1             |  seqr/utils/redis_utils.py                          |   2 -
seqr_1             |  ...dis_utils_2_3_tests.py => redis_utils_tests.py} |   2 -
seqr_1             |  seqr/utils/xpos_utils.py                           |   2 -
seqr_1             |  ...xpos_utils_2_3_tests.py => xpos_utils_tests.py} |   2 -
seqr_1             |  seqr/views/apis/analysis_group_api.py              |   2 -
seqr_1             |  ...pi_2_3_tests.py => analysis_group_api_tests.py} |   2 -
seqr_1             |  seqr/views/apis/auth_api.py                        |   2 -
seqr_1             |  .../{auth_api_2_3_tests.py => auth_api_tests.py}   |   2 -
seqr_1             |  seqr/views/apis/awesomebar_api.py                  |   2 -
seqr_1             |  ...ar_api_2_3_tests.py => awesomebar_api_tests.py} |   2 -
seqr_1             |  seqr/views/apis/case_review_api.py                 |   2 -
seqr_1             |  ...w_api_2_3_tests.py => case_review_api_tests.py} |   1 -
seqr_1             |  seqr/views/apis/dashboard_api_tests.py             |  22 +-
seqr_1             |  seqr/views/apis/dataset_api.py                     |   3 -
seqr_1             |  ...taset_api_2_3_tests.py => dataset_api_tests.py} |   8 +-
seqr_1             |  seqr/views/apis/family_api.py                      |   3 -
seqr_1             |  ...family_api_2_3_tests.py => family_api_tests.py} |   2 -
seqr_1             |  seqr/views/apis/gene_api.py                        |   2 -
seqr_1             |  .../{gene_api_2_3_tests.py => gene_api_tests.py}   |   2 -
seqr_1             |  seqr/views/apis/igv_api.py                         |   2 -
seqr_1             |  .../{igv_api_2_3_tests.py => igv_api_tests.py}     |   2 -
seqr_1             |  seqr/views/apis/individual_api.py                  |  25 +-
seqr_1             |  ...al_api_2_3_tests.py => individual_api_tests.py} |   9 +-
seqr_1             |  seqr/views/apis/locus_list_api.py                  |   2 -
seqr_1             |  ...st_api_2_3_tests.py => locus_list_api_tests.py} |   2 -
seqr_1             |  seqr/views/apis/project_api.py                     |   8 +-
seqr_1             |  ...oject_api_2_3_tests.py => project_api_tests.py} |   6 +-
seqr_1             |  seqr/views/apis/project_categories_api.py          |   3 -
seqr_1             |  ..._3_tests.py => project_categories_api_tests.py} |   2 -
seqr_1             |  seqr/views/apis/saved_variant_api.py               |   2 -
seqr_1             |  ...api_2_3_tests.py => saved_variant_api_tests.py} |   2 -
seqr_1             |  seqr/views/apis/staff_api.py                       | 372 ++++++++++++---------
seqr_1             |  .../{staff_api_2_3_tests.py => staff_api_tests.py} |  38 ++-
seqr_1             |  seqr/views/apis/users_api.py                       |   2 -
seqr_1             |  .../{users_api_2_3_tests.py => users_api_tests.py} |   2 -
seqr_1             |  seqr/views/apis/variant_search_api.py              |   2 -
seqr_1             |  ...pi_2_3_tests.py => variant_search_api_tests.py} |   2 -
seqr_1             |  seqr/views/react_app.py                            |   2 -
seqr_1             |  .../{react_app_2_3_tests.py => react_app_tests.py} |   2 -
seqr_1             |  seqr/views/utils/dataset_utils.py                  |   2 -
seqr_1             |  seqr/views/utils/export_utils.py                   |   7 +-
seqr_1             |  ...rt_utils_2_3_tests.py => export_utils_tests.py} |   3 -
seqr_1             |  seqr/views/utils/file_utils.py                     |   3 -
seqr_1             |  ...file_utils_2_3_tests.py => file_utils_tests.py} |   2 -
seqr_1             |  seqr/views/utils/individual_utils.py               |   2 -
seqr_1             |  seqr/views/utils/json_to_orm_utils.py              |   2 -
seqr_1             |  seqr/views/utils/json_utils.py                     |   2 -
seqr_1             |  seqr/views/utils/orm_to_json_utils.py              |   2 -
seqr_1             |  ...ils_2_3_tests.py => orm_to_json_utils_tests.py} |   2 -
seqr_1             |  seqr/views/utils/pedigree_image_utils.py           |  22 +-
seqr_1             |  seqr/views/utils/pedigree_image_utils_tests.py     |  14 +-
seqr_1             |  seqr/views/utils/pedigree_info_utils.py            |   6 +-
seqr_1             |  ...s_2_3_tests.py => pedigree_info_utils_tests.py} |   2 -
seqr_1             |  seqr/views/utils/permissions_utils.py              |   2 -
seqr_1             |  seqr/views/utils/test_utils.py                     |   2 -
seqr_1             |  seqr/views/utils/variant_utils.py                  |   2 -
seqr_1             |  servctl                                            |   7 +-
seqr_1             |  ui/dist/{app-3d8d6a9d.js => app-f1ded2ce.js}       |   4 +-
seqr_1             |  .../{app-3d8d6a9d.js.map => app-f1ded2ce.js.map}   |   2 +-
seqr_1             |  ui/dist/app.html                                   |   2 +-
seqr_1             |  ui/dist/asset-manifest.json                        |   4 +-
seqr_1             |  ui/pages/Project/components/Matchmaker.jsx         |  12 +-
seqr_1             |  ui/pages/Project/components/ProjectOverview.jsx    |   4 +-
seqr_1             |  ui/pages/Project/fixtures.js                       |  61 +++-
seqr_1             |  ui/pages/Project/selectors.js                      |  45 ++-
seqr_1             |  ui/pages/Project/selectors.test.js                 |   4 +-
seqr_1             |  ui/pages/Staff/constants.js                        |   4 +-
seqr_1             |  ui/redux/selectors.test.js                         |  21 +-
seqr_1             |  169 files changed, 1121 insertions(+), 937 deletions(-)
seqr_1             |  delete mode 100755 collect_static.sh
seqr_1             |  create mode 100644 deploy/servctl_utils/kubectl_utils.py
seqr_1             |  create mode 100644 matchmaker/migrations/0002_matchmakerresult_originating_submission.py
seqr_1             |  rename matchmaker/views/{external_api_2_3_tests.py => external_api_tests.py} (99%)
seqr_1             |  rename matchmaker/views/{matchmaker_api_2_3_tests.py => matchmaker_api_tests.py} (93%)
seqr_1             |  rename reference_data/management/commands/utils/{download_utils_2_3_tests.py => download_utils_tests.py} (75%)
seqr_1             |  rename reference_data/management/tests/{run_reference_data_database_backup_2_3_tests.py => run_reference_data_database_backup_tests.py} (100%)
seqr_1             |  rename reference_data/management/tests/{update_all_reference_data_2_3_tests.py => update_all_reference_data_tests.py} (100%)
seqr_1             |  rename reference_data/management/tests/{update_dbnsfp_gene_2_3_tests.py => update_dbnsfp_gene_tests.py} (84%)
seqr_1             |  rename reference_data/management/tests/{update_gencode_2_3_tests.py => update_gencode_tests.py} (87%)
seqr_1             |  rename reference_data/management/tests/{update_gene_constraint_2_3_tests.py => update_gene_constraint_tests.py} (80%)
seqr_1             |  rename reference_data/management/tests/{update_hpo_2_3_tests.py => update_hpo_tests.py} (86%)
seqr_1             |  rename reference_data/management/tests/{update_mgi_2_3_tests.py => update_mgi_tests.py} (68%)
seqr_1             |  rename reference_data/management/tests/{update_omim_2_3_tests.py => update_omim_tests.py} (81%)
seqr_1             |  rename reference_data/management/tests/{update_primate_ai_2_3_tests.py => update_primate_ai_tests.py} (65%)
seqr_1             |  rename seqr/management/tests/{add_project_tag_2_3_tests.py => add_project_tag_tests.py} (100%)
seqr_1             |  rename seqr/management/tests/{check_bam_cram_paths_2_3_tests.py => check_bam_cram_paths_tests.py} (97%)
seqr_1             |  rename seqr/management/tests/{copy_project_tags_2_3_tests.py => copy_project_tags_tests.py} (94%)
seqr_1             |  rename seqr/management/tests/{lift_project_to_hg38_2_3_tests.py => lift_project_to_hg38_tests.py} (99%)
seqr_1             |  rename seqr/management/tests/{lift_variant_to_hg38_2_3_tests.py => lift_variant_to_hg38_tests.py} (96%)
seqr_1             |  rename seqr/management/tests/{reload_saved_variant_json_2_3_tests.py => reload_saved_variant_json_tests.py} (98%)
seqr_1             |  rename seqr/management/tests/{resend_welcome_email_2_3_tests.py => resend_welcome_email_tests.py} (95%)
seqr_1             |  rename seqr/management/tests/{run_postgres_database_backup_2_3_tests.py => run_postgres_database_backup_tests.py} (98%)
seqr_1             |  rename seqr/management/tests/{run_settings_backup_2_3_tests.py => run_settings_backup_tests.py} (98%)
seqr_1             |  rename seqr/management/tests/{transfer_families_to_different_project_2_3_tests.py => transfer_families_to_different_project_tests.py} (97%)
seqr_1             |  rename seqr/utils/elasticsearch/{es_utils_2_3_tests.py => es_utils_tests.py} (99%)
seqr_1             |  rename seqr/utils/{redis_utils_2_3_tests.py => redis_utils_tests.py} (98%)
seqr_1             |  rename seqr/utils/{xpos_utils_2_3_tests.py => xpos_utils_tests.py} (97%)
seqr_1             |  rename seqr/views/apis/{analysis_group_api_2_3_tests.py => analysis_group_api_tests.py} (99%)
seqr_1             |  rename seqr/views/apis/{auth_api_2_3_tests.py => auth_api_tests.py} (98%)
seqr_1             |  rename seqr/views/apis/{awesomebar_api_2_3_tests.py => awesomebar_api_tests.py} (97%)
seqr_1             |  rename seqr/views/apis/{case_review_api_2_3_tests.py => case_review_api_tests.py} (98%)
seqr_1             |  rename seqr/views/apis/{dataset_api_2_3_tests.py => dataset_api_tests.py} (98%)
seqr_1             |  rename seqr/views/apis/{family_api_2_3_tests.py => family_api_tests.py} (99%)
seqr_1             |  rename seqr/views/apis/{gene_api_2_3_tests.py => gene_api_tests.py} (98%)
seqr_1             |  rename seqr/views/apis/{igv_api_2_3_tests.py => igv_api_tests.py} (98%)
seqr_1             |  rename seqr/views/apis/{individual_api_2_3_tests.py => individual_api_tests.py} (97%)
seqr_1             |  rename seqr/views/apis/{locus_list_api_2_3_tests.py => locus_list_api_tests.py} (99%)
seqr_1             |  rename seqr/views/apis/{project_api_2_3_tests.py => project_api_tests.py} (98%)
seqr_1             |  rename seqr/views/apis/{project_categories_api_2_3_tests.py => project_categories_api_tests.py} (98%)
seqr_1             |  rename seqr/views/apis/{saved_variant_api_2_3_tests.py => saved_variant_api_tests.py} (99%)
seqr_1             |  rename seqr/views/apis/{staff_api_2_3_tests.py => staff_api_tests.py} (96%)
seqr_1             |  rename seqr/views/apis/{users_api_2_3_tests.py => users_api_tests.py} (99%)
seqr_1             |  rename seqr/views/apis/{variant_search_api_2_3_tests.py => variant_search_api_tests.py} (99%)
seqr_1             |  rename seqr/views/{react_app_2_3_tests.py => react_app_tests.py} (98%)
seqr_1             |  rename seqr/views/utils/{export_utils_2_3_tests.py => export_utils_tests.py} (98%)
seqr_1             |  rename seqr/views/utils/{file_utils_2_3_tests.py => file_utils_tests.py} (99%)
seqr_1             |  rename seqr/views/utils/{orm_to_json_utils_2_3_tests.py => orm_to_json_utils_tests.py} (99%)
seqr_1             |  rename seqr/views/utils/{pedigree_info_utils_2_3_tests.py => pedigree_info_utils_tests.py} (99%)
seqr_1             |  rename ui/dist/{app-3d8d6a9d.js => app-f1ded2ce.js} (88%)
seqr_1             |  rename ui/dist/{app-3d8d6a9d.js.map => app-f1ded2ce.js.map} (59%)
seqr_1             | + git checkout dev
seqr_1             | Switched to a new branch 'dev'
seqr_1             | Branch dev set up to track remote branch dev from origin.
seqr_1             | + pip install --upgrade -r requirements.txt
seqr_1             | Requirement already up-to-date: Django==1.11.29 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (1.11.29)
seqr_1             | Requirement already up-to-date: django-anymail==7.0.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (7.0.0)
seqr_1             | Requirement already up-to-date: django-guardian==1.5.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (1.5.1)
seqr_1             | Requirement already up-to-date: django-hijack==2.1.10 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (2.1.10)
seqr_1             | Requirement already up-to-date: django-cors-headers==3.0.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (3.0.2)
seqr_1             | Requirement already up-to-date: elasticsearch==6.4.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (6.4.0)
seqr_1             | Requirement already up-to-date: elasticsearch-dsl==6.4.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.4.0)
seqr_1             | Requirement already up-to-date: gunicorn==19.10.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (19.10.0)
seqr_1             | Requirement already up-to-date: jmespath==0.9.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (0.9.4)
seqr_1             | Requirement already up-to-date: openpyxl==2.6.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (2.6.4)
seqr_1             | Collecting pillow==7.1.0
seqr_1             |   Downloading Pillow-7.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
seqr_1             | Requirement already up-to-date: psycopg2==2.8.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (2.8.4)
seqr_1             | Requirement already up-to-date: pyliftover==0.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (0.4)
seqr_1             | Requirement already up-to-date: requests==2.22.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (2.22.0)
seqr_1             | Requirement already up-to-date: redis==3.3.11 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (3.3.11)
seqr_1             | Requirement already up-to-date: requests-toolbelt==0.9.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (0.9.1)
seqr_1             | Requirement already up-to-date: slacker==0.13.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 18)) (0.13.0)
seqr_1             | Requirement already up-to-date: slugify==0.0.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 19)) (0.0.1)
seqr_1             | Requirement already up-to-date: tqdm==4.40.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 20)) (4.40.2)
seqr_1             | Requirement already up-to-date: whitenoise==3.3.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 21)) (3.3.0)
seqr_1             | Requirement already satisfied, skipping upgrade: pytz in /usr/local/lib/python3.7/site-packages (from Django==1.11.29->-r requirements.txt (line 1)) (2020.1)
seqr_1             | Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.7/site-packages (from django-anymail==7.0.0->-r requirements.txt (line 2)) (1.13.0)
seqr_1             | Requirement already satisfied, skipping upgrade: django-compat>=1.0.14 in /usr/local/lib/python3.7/site-packages (from django-hijack==2.1.10->-r requirements.txt (line 4)) (1.0.15)
seqr_1             | Requirement already satisfied, skipping upgrade: urllib3>=1.21.1 in /usr/local/lib/python3.7/site-packages (from elasticsearch==6.4.0->-r requirements.txt (line 7)) (1.25.10)
seqr_1             | Requirement already satisfied, skipping upgrade: python-dateutil in /usr/local/lib/python3.7/site-packages (from elasticsearch-dsl==6.4.0->-r requirements.txt (line 8)) (2.8.1)
seqr_1             | Requirement already satisfied, skipping upgrade: jdcal in /usr/local/lib/python3.7/site-packages (from openpyxl==2.6.4->-r requirements.txt (line 11)) (1.4.1)
seqr_1             | Requirement already satisfied, skipping upgrade: et_xmlfile in /usr/local/lib/python3.7/site-packages (from openpyxl==2.6.4->-r requirements.txt (line 11)) (1.0.1)
seqr_1             | Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests==2.22.0->-r requirements.txt (line 15)) (2.8)
seqr_1             | Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests==2.22.0->-r requirements.txt (line 15)) (3.0.4)
seqr_1             | Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests==2.22.0->-r requirements.txt (line 15)) (2020.6.20)
seqr_1             | Installing collected packages: pillow
seqr_1             |   Attempting uninstall: pillow
seqr_1             |     Found existing installation: Pillow 6.2.2
seqr_1             |     Uninstalling Pillow-6.2.2:
seqr_1             |       Successfully uninstalled Pillow-6.2.2
seqr_1             | Successfully installed pillow-7.1.0
seqr_1             | WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
seqr_1             | You should consider upgrading via the '/usr/local/bin/python3.7 -m pip install --upgrade pip' command.
seqr_1             | + echo '*:*:*:*:docker-compose-postgress-password'
seqr_1             | + chmod 600 /root/.pgpass
seqr_1             | + cat /root/.pgpass
seqr_1             | *:*:*:*:docker-compose-postgress-password
seqr_1             | + psql --host postgres -U postgres -l
seqr_1             | + grep seqrdb
seqr_1             |  seqrdb            | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
seqr_1             | + /usr/local/bin/start_server.sh
seqr_1             | + '[' ']'
seqr_1             | + sleep 1000000000000
seqr_1             | [2020-08-11 15:39:14 +0000] [42] [INFO] Starting gunicorn 19.10.0
seqr_1             | [2020-08-11 15:39:14 +0000] [42] [INFO] Listening at: http://0.0.0.0:8000 (42)
seqr_1             | [2020-08-11 15:39:14 +0000] [42] [INFO] Using worker: sync
seqr_1             | [2020-08-11 15:39:14 +0000] [48] [INFO] Booting worker with pid: 48
seqr_1             | [2020-08-11 15:39:14 +0000] [49] [INFO] Booting worker with pid: 49
seqr_1             | [2020-08-11 15:39:14 +0000] [50] [INFO] Booting worker with pid: 50
seqr_1             | [2020-08-11 15:39:14 +0000] [51] [INFO] Booting worker with pid: 51

Do you see the same thing?

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

You can now also see local deployment logs at the bottom of our travis-ci builds - for example
https://travis-ci.org/github/macarthur-lab/seqr/builds/717251157#L3360

(this is building a branch where the postgres port has been changed to 5433 since 5432 conflicts with the travis-ci environment - the branch code is @
https://github.com/macarthur-lab/seqr/compare/travis-test-local-deployment#diff-4e5e90c6228fd48698d074241c2ba760L7 )

@jbsysadmin
Copy link
Author

Total error output on "docker-compose exec seqr python manage.py createsuperuser" is below.

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.7/site-packages/psycopg2/init.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Network is unreachable
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

If you run
docker-compose logs postgres
and
docker-compose logs seqr
I would guess you will see errors there.
You may also want to try running
docker-compose ps to check that all containers are running as expected.

If there's no obvious fix, it may be worth restarting from scratch by running

rm -rf data/   # assuming you currently have a blank instance
docker-compose down

and then running through the install steps again.

@jbsysadmin
Copy link
Author

Hello again.

  • My output from docker-compose logs -f seqr ends in:
seqr_1             | [2020-08-12 14:18:07 +0000] [41] [INFO] Starting gunicorn 19.10.0
seqr_1             | [2020-08-12 14:18:07 +0000] [41] [INFO] Listening at: http://0.0.0.0:8000 (41)
seqr_1             | [2020-08-12 14:18:07 +0000] [41] [INFO] Using worker: sync
seqr_1             | [2020-08-12 14:18:07 +0000] [47] [INFO] Booting worker with pid: 47
seqr_1             | [2020-08-12 14:18:07 +0000] [48] [INFO] Booting worker with pid: 48
seqr_1             | [2020-08-12 14:18:07 +0000] [49] [INFO] Booting worker with pid: 49
seqr_1             | [2020-08-12 14:18:07 +0000] [50] [INFO] Booting worker with pid: 50
  • My output from docker-compose logs postgres:
Attaching to app_datasets_postgres_1
postgres_1         |
postgres_1         | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1         |
postgres_1         | 2020-08-12 14:17:49.241 UTC [1] LOG:  starting PostgreSQL 12.3 (Debian 12.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1         | 2020-08-12 14:17:49.243 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1         | 2020-08-12 14:17:49.243 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1         | 2020-08-12 14:17:49.271 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1         | 2020-08-12 14:17:49.463 UTC [25] LOG:  database system was shut down at 2020-08-11 21:29:08 UTC
postgres_1         | 2020-08-12 14:17:49.528 UTC [1] LOG:  database system is ready to accept connections

@jbsysadmin
Copy link
Author

docker-compose ps shows:

app_datasets_elasticsearch_1   /entrypoint.sh                   Up (healthy)     0.0.0.0:9200->9200/tcp
app_datasets_postgres_1        docker-entrypoint.sh postgres    Up (healthy)     0.0.0.0:5432->5432/tcp
app_datasets_redis_1           docker-entrypoint.sh redis ...   Up (healthy)     0.0.0.0:6379->6379/tcp
app_datasets_seqr_1            /entrypoint.sh                   Up (unhealthy)   0.0.0.0:80->8000/tcp

@jbsysadmin
Copy link
Author

One thing: I never ran rm -rf data/ prior to any restart attempt. Will try next.

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

Great. Yeah, don't worry about branches - they aren't relevant.
The only issue I see is docker-compose ps shows the seqr container as unhealthy so I'd look at more of the seqr logs.

@jbsysadmin
Copy link
Author

Total output from: docker-compose logs -f seqr is enclosed.

seqr_error_output_8-12-2020.txt

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

Thanks. I see the issue. The seqr error message

Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?

tells me it's trying to connect to postgres on localhost when it should be connecting to hostname postgres.

The postgres hostname in seqr is determined by the POSTGRES_SERVICE_HOSTNAME env var, and defaults to localhost if the env. var is not set.

Within docker-compose, it's being set to postgres here:
https://github.com/macarthur-lab/seqr/blob/master/docker-compose.yml#L67

Is there some reason this may be behaving differently in your environment?

@jbsysadmin
Copy link
Author

jbsysadmin commented Aug 12, 2020 via email

@jbsysadmin
Copy link
Author

So at this point, I noticed this log excerpt:

qr_1             | CREATE DATABASE
seqr_1             | + python -u manage.py makemigrations
seqr_1             | Traceback (most recent call last):
seqr_1             |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
seqr_1             |     self.connect()
seqr_1             |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 189, in connect
seqr_1             |     self.connection = self.get_new_connection(conn_params)
seqr_1             |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
seqr_1             |     connection = Database.connect(**conn_params)
seqr_1             |   File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect
seqr_1             |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
seqr_1             | psycopg2.OperationalError: FATAL:  database "reference_data_db" does not exist

Out of curiosity, I ran the docker-compose exec seqr python manage.py createsuperuser
I got the credentials output at last.

In the event that this is working:

# authenticate to your gcloud account so you can download public reference data
gcloud auth application-default login

What is on the other end of that?

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

@hanars can confirm, but I believe that reference_data_db error can be ignored.
To your other questions:

  • the non-root username created by createsuperuser becomes the 1st seqr user that can login (with the username & password you typed in to the prompt) using the seqr web UI that should show up when you open http://localhost:80 in your browser (or remotely, using the hostname of the machine running docker-compose instead of localhost).
    Once you've logged in to the web UI as this admin user, you should be able to add more admin users by clicking on Staff Pages at the top and using the Admin web UI:
    image
    Non-staff users can be added by creating a "project" and adding collaborators on the Project page:
    image

@bw2
Copy link
Contributor

bw2 commented Aug 12, 2020

@hanars
Copy link
Collaborator

hanars commented Aug 13, 2020

I can confirm that there is no issue with that error

@jbsysadmin
Copy link
Author

Thank you for the guidance. I forgot to mention: There is an error on the documentation link: "TemplateDoesNotExist at /admin/doc/" Any thoughts on that?

Template-loader postmortem
Django tried loading these templates, in this order:
Using engine django:

    django.template.loaders.filesystem.Loader: /seqr/ui/dist/admin_doc/missing_docutils.html (Source does not exist)
    django.template.loaders.app_directories.Loader: /usr/local/lib/python3.7/site-packages/django/contrib/admin/templates/admin_doc/missing_docutils.html (Source does not exist)
    django.template.loaders.app_directories.Loader: /usr/local/lib/python3.7/site-packages/django/contrib/auth/templates/admin_doc/missing_docutils.html (Source does not exist)
    django.template.loaders.app_directories.Loader: /usr/local/lib/python3.7/site-packages/hijack/templates/admin_doc/missing_docutils.html (Source does not exist)
    django.template.loaders.app_directories.Loader: /usr/local/lib/python3.7/site-packages/guardian/templates/admin_doc/missing_docutils.html (Source does not exist)

@bw2
Copy link
Contributor

bw2 commented Aug 13, 2020

I'm not sure. What command are you running that produces this error?

@jbsysadmin
Copy link
Author

No command; broken link when I try to web browse http://<seqr_presence_IP>/admin/doc/.

@hanars
Copy link
Collaborator

hanars commented Aug 13, 2020

thats an error on our seqr too, we just ignore it. http://<seqr_presence_IP>/admin should work fine

@jbsysadmin
Copy link
Author

This may be out-of-scope...

In the Google Cloud Platform, "Always" Free language:

    You're eligible for the Always Free program if you meet the following criteria: 
    You do not have a custom contract or rate card with Google.
    You have an upgraded billing account. An upgraded account is ready to be charged for use, as opposed to a free trial only account. 

After a 12-month free trial:

    All resources you created during the trial are stopped.
    Any data you stored in Compute Engine is lost.

I was going to try with my own email address. But you are obliged to leave a card number, so I declined. Do you know how one actually can proceed without credit card?

@hanars hanars closed this as completed Oct 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants