pipeline-runner copy_reference_data_to_gs.sh

[xshah]

How much time does this command take?
docker-compose exec pipeline-runner copy_reference_data_to_gs.sh $BUILD_VERSION $GS_BUCKET

BUILD_VERSION = 38

In our case it has been running since 24 hours, is that expected?

[xshah]

Is there a better alternative or any parameters we can pass along to speed this process?

[hanars]

you can manually run the commands the script generates and add a -m option which might work? So instead of

gsutil rsync -r "gs://seqr-reference-data/GRCh${BUILD_VERSION}/all_reference_data/combined_reference_data_grch${BUILD_VERSION}.ht" "${GS_BUCKET}/reference_data/GRCh${BUILD_VERSION}/combined_reference_data_grch${BUILD_VERSION}.ht"

it would be

gsutil -m rsync -r "gs://seqr-reference-data/GRCh${BUILD_VERSION}/all_reference_data/combined_reference_data_grch${BUILD_VERSION}.ht" "${GS_BUCKET}/reference_data/GRCh${BUILD_VERSION}/combined_reference_data_grch${BUILD_VERSION}.ht"