Catalogue update #496
Assignees
Milestone
Comments
|
Hi @Maite-Machipi Have you seen the documentation and videos in the wiki? Please try to follow the procedures there then come back to me if anything doesn't work or if the documentation isn't clear or correct. There are wiki pages and YouTube videos covering ingestion. |
|
Hi @Maite-Machipi the ingestion procedure is documented in https://www.youtube.com/watch?v=zZpwRSym6Ls&t=8s and https://github.com/kartoza/catalogue/wiki/Ingest-the-metadata. I've spent some time browsing around /EOSHARE/Karabo/CATALOGUE, where we expect to find new metadata and I cannot find any for any sensor even for this year. Are your systems still writing metadata with each new image? If they are, which path are they writing to? Once you have identified the new metadata to ingest, you still need to follow the ingestion procedures by copying it and running the ingestion scripts. Once you have sorted out the metadata generation process and located and ingested the missing metadata, I agree with @lavhengwa that we should be automating the ingestion process. Can we go ahead and create tickets for that to plan work on automation? |
|
Good day Gavin, |
|
Hi Tendani, I have started a discussion on the gitter. Thanks. |
|
Hi @boney-bun , Please post the discussion here as well. Thanks. |
|
Hi @Maite-Machipi @lavhengwa below is from gitter: i think I found two problems for the ingestion. docker doesn't follow symbolic link on a mounted volume. the solutions to the problem is to directly mount the secondly, the it is tricky when using different ip addresses on docker. |
|
Hi @boney-bun
|
|
Good @boney-bun |
|
Looking at the image provided, it is not an error, but warning messages instead. The problem is the app can't find any new files to be ingested. this is why i suspect that there is an inconsistencies between the volume on the host and the volume on the docker container. |
|
I've updated the wiki doc to overcome the inconsistencies (https://github.com/kartoza/catalogue/wiki/Ingesting-metadata). Below is the example of ingesting NAMIBIA folder as described in the wiki link above: Now the ingestion can find the metadata. But there exist another error with respect to the P10 instrument type. But, I need to make sure whether the current metadata is valid or not. Sorry for the log message on the GIF above. |
|
Good day @boney-bun with regards to the ingestion, can we also have an option to only ingest a single strip, and after the ingestion is done will it reflect immediately on the catalogue or it will reflect after some time. Regarding the naming convention the metadata naming should hold. |
|
Good evening @Maite-Machipi @kmithi The ingestion of I have recorded these action into a GIF file, but can't attach the file here due to size restriction. Please note that in order to be ingested successfully, a modification to the database was made for satisfying the metadata. I've added satellite name of 'CBERS-04 Below is the commands taken from the shell as a proof: |
|
hi @boney-bun, i noticed that from your ingestion it says 232 products updates. and then this is the screen shot i took today. |
|
Thanks for pointing this out.
I previously have modified the |
|
Hi @boney-bun , Can you please advise why the ingested data doesn't reflect directly after i've done the ingest. |
|
Hi @boney-bun , can you please assist with the ingest for other sensors, as i seem to be getting an error, it might be that my command is incorrect. for example if im to do a spot7 ingest : |
|
Hi @Maite-Machipi, It turns out the path for spot6 or spot7 ingestion is slightly different. The ingestion will try to find xml files on the folders. I've done the ingestion, below is the snapshot of the shell: |
@Maite-Machipi where do you check the result? on |
|
Hi @boney-bun, yes I check the results on data summary. |
|
Hi @boney-bun, Please advise on a way to have the thumbnails visible, as they are currently not visible. |
the data summary is not updated frequently as we don't often run the ingestion. |
the Thanks. |
|
Hi @boney-bun, Please note that all ingested Cbers files are still on the data folder, with the corresponding thumbnails. |
|
Good morning @Maite-Machipi, the thumbnails work now. please let me know if there are any other non working thumbnails. i think it's a good idea to have a zoom session when we have another new metadata that needs to be ingested. what do you think? |
|
Morning @boney-bun , Ill drop you an email then we can organize the zoom session. |
|
From @Maite-Machipi via email: Please note that I created a folder in the P10 folder and named it Completed, that is the folder for all the ingested files,
|
|
Hi @boney-bun , Please note that the data summary tab is not working.
|
|
Hi @Maite-Machipi could you try again? |
|
Good morning @Maite-Machipi
i'll add this on the wiki page as well. Feel free to letting me know if you still found any not working thumbnails. |
|
Morning @boney-bun Please note that now the data summary tab is working, and thanks for assisting with the thumbnails ill let you know if i find any not working thumbnail. |
|
@Maite-Machipi please close this if it has been resolved to your satisfaction |
|
Hi @boney-bun,
Please note below, I’ve added data to ingest Spot but it gives me the same outcome that it gave you before I even added data in the Spot folder.
root@uwsgi:/home/web/django_project# python manage.py spot67_harvest --source=./data/SPOT
{'WORKER_ENV_C_FORCE_ROOT': 'true', 'VIRTUAL_HOST': 'catalogue.kartoza.com', 'CATALOGUE_UWSGI_ENV_VIRTUAL_PORT': '8080', 'DB_ENV_USERNAME': 'docker', 'RABBITMQ_1_PORT_5672_TCP_PORT': '5672', 'RABBITMQ_PORT_4369_TCP_ADDR': '172.17.0.4', 'UWSGI_PORT_8080_TCP': 'tcp://172.17.0.6:8080', 'CATALOGUE_RABBITMQ_1_ENV_USER': 'rabbit_user', 'WORKER_ENV_PYTHON_PIP_VERSION': '8.1.0', 'CATALOGUE_UWSGI_ENV_LANG': 'C.UTF-8', 'CATALOGUE_WORKER_1_NAME': '/catalogue_uwsgi_run_124/catalogue_worker_1', 'WORKER_1_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_ENV_RABBITMQ_LOGS': '-', 'PYTHON_VERSION': '2.7.11', 'CATALOGUE_SMTP_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_ADDR': '172.17.0.4', 'SMTP_ENV_maildomain': 'kartoza.com', 'WORKER_1_PORT_8080_TCP_PORT': '8080', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP': 'tcp://172.17.0.4:5671', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_ADDR': '172.17.0.4', 'DB_PORT_5432_TCP_PROTO': 'tcp', 'WORKER_NAME': '/catalogue_uwsgi_run_124/worker', 'WORKER_ENV_LANG': 'C.UTF-8', 'RABBITMQ_PORT_5672_TCP_PROTO': 'tcp', 'RABBITMQ_1_ENV_USER': 'rabbit_user', 'UWSGI_ENV_DATABASE_USERNAME': 'docker', 'RABBITMQ_PORT': 'tcp://172.17.0.4:4369', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_SASL_LOGS': '-', 'RABBITMQ_HOST': 'rabbitmq', 'RABBITMQ_1_PORT_5671_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_DATABASE_PASSWORD': 'docker', 'CATALOGUE_UWSGI_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_UWSGI_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_RABBITMQ_1_ENV_GOSU_VERSION': '1.7', 'RABBITMQ_1_PORT_25672_TCP': 'tcp://172.17.0.4:25672', 'DB_NAME': '/catalogue_uwsgi_run_124/db', 'CATALOGUE_WORKER_1_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_1_ENV_GOSU_VERSION': '1.7', 'DB_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_UWSGI_NAME': '/catalogue_uwsgi_run_124/catalogue-uwsgi', 'CATALOGUE_RABBITMQ_1_ENV_affinity:container': '=450af8e7f3fa0726bfce0e8d0e5add7370fe02e22ba5da46f8577a4b26308605', 'CATALOGUE_UWSGI_ENV_DATABASE_HOST': 'db', 'WORKER_1_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'RABBITMQ_ENV_GOSU_VERSION': '1.7', 'CATALOGUE_WORKER_1_PORT_8080_TCP_ADDR': '172.17.0.5', 'RABBITMQ_PORT_4369_TCP': 'tcp://172.17.0.4:4369', 'RABBITMQ_1_PORT_4369_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_PYTHON_PIP_VERSION': '8.1.0', 'CATALOGUE_WORKER_1_PORT_8080_TCP_PORT': '8080', 'RABBITMQ_1_ENV_RABBITMQ_SASL_LOGS': '-', 'HOSTNAME': 'uwsgi', 'CATALOGUE_WORKER_1_ENV_VIRTUAL_PORT': '8080', 'RABBITMQ_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'RABBITMQ_1_PORT_25672_TCP_PORT': '25672', 'CATALOGUE_UWSGI_PORT_8080_TCP': 'tcp://172.17.0.6:8080', 'CATALOGUE_UWSGI_PORT_8080_TCP_ADDR': '172.17.0.6', 'DATABASE_HOST': 'db', 'RABBITMQ_NAME': '/catalogue_uwsgi_run_124/rabbitmq', 'CATALOGUE_UWSGI_ENV_DATABASE_PASSWORD': 'docker', 'DB_PORT_5432_TCP_PORT': '5432', 'DB_PORT': 'tcp://172.17.0.3:5432', 'CATALOGUE_DB_PORT_5432_TCP_ADDR': '172.17.0.3', 'UWSGI_PORT_8080_TCP_PORT': '8080', 'WORKER_PORT_8080_TCP': 'tcp://172.17.0.5:8080', 'CATALOGUE_UWSGI_PORT_8080_TCP_PORT': '8080', 'WORKER_1_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_WORKER_1_PORT': 'tcp://172.17.0.5:8080', 'RABBITMQ_1_PORT_4369_TCP_PORT': '4369', 'CATALOGUE_DB_NAME': '/catalogue_uwsgi_run_124/catalogue-db', 'RABBITMQ_1_PORT_5672_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT_4369_TCP_ADDR': '172.17.0.4', 'SHLVL': '1', 'RABBITMQ_PORT_25672_TCP': 'tcp://172.17.0.4:25672', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP': 'tcp://172.17.0.4:25672', 'WORKER_1_ENV_PYTHON_VERSION': '2.7.11', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_ADDR': '172.17.0.4', 'CATALOGUE_DB_PORT': 'tcp://172.17.0.3:5432', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_PROTO': 'tcp', 'UWSGI_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'RABBITMQ_PORT_5672_TCP_PORT': '5672', 'UWSGI_ENV_VIRTUAL_HOST': 'catalogue.kartoza.com', 'RABBITMQ_ENV_affinity:container': '=450af8e7f3fa0726bfce0e8d0e5add7370fe02e22ba5da46f8577a4b26308605', 'UWSGI_PORT_8080_TCP_PROTO': 'tcp', 'SMTP_ENV_smtp_user': 'noreply:docker', 'CATALOGUE_UWSGI_PORT': 'tcp://172.17.0.6:8080', 'WORKER_1_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_PORT_8080_TCP_PROTO': 'tcp', 'VIRTUAL_PORT': '8080', 'RABBITMQ_PORT_5671_TCP_ADDR': '172.17.0.4', 'RABBITMQ_PORT_4369_TCP_PORT': '4369', 'RABBITMQ_1_ENV_RABBITMQ_LOGS': '-', 'WORKER_1_ENV_DATABASE_USERNAME': 'docker', 'WORKER_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', '_': '/usr/local/bin/python', 'WORKER_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_WORKER_1_ENV_C_FORCE_ROOT': 'true', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_VERSION': '3.6.1', 'RABBITMQ_PORT_5672_TCP_ADDR': '172.17.0.4', 'DATABASE_NAME': 'gis', 'WORKER_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_1_ENV_C_FORCE_ROOT': 'true', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_PORT': '25672', 'DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'CATALOGUE_UWSGI_ENV_DEBIAN_FRONTEND': 'noninteractive', 'WORKER_1_PORT_8080_TCP_ADDR': '172.17.0.5', 'CATALOGUE_WORKER_1_ENV_PYTHON_VERSION': '2.7.11', 'WORKER_ENV_DEBIAN_FRONTEND': 'noninteractive', 'UWSGI_ENV_DATABASE_HOST': 'db', 'UWSGI_ENV_DATABASE_PASSWORD': 'docker', 'CATALOGUE_SMTP_ENV_smtp_user': 'noreply:docker', 'UWSGI_PORT': 'tcp://172.17.0.6:8080', 'CATALOGUE_UWSGI_ENV_PYTHON_VERSION': '2.7.11', 'RABBITMQ_PORT_5671_TCP_PORT': '5671', 'RABBITMQ_PORT_5671_TCP': 'tcp://172.17.0.4:5671', 'HOME': '/root', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_DEBIAN_VERSION': '3.6.1-1', 'CATALOGUE_RABBITMQ_1_PORT': 'tcp://172.17.0.4:4369', 'LANG': 'C.UTF-8', 'RABBITMQ_1_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_ADDR': '172.17.0.4', 'UWSGI_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_ENV_RABBITMQ_VERSION': '3.6.1', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_PORT': '4369', 'PYTHON_PIP_VERSION': '8.1.0', 'RABBITMQ_PORT_25672_TCP_PORT': '25672', 'RABBITMQ_1_ENV_affinity:container': '=450af8e7f3fa0726bfce0e8d0e5add7370fe02e22ba5da46f8577a4b26308605', 'CATALOGUE_UWSGI_PORT_8080_TCP_PROTO': 'tcp', 'SMTP_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_PORT_4369_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT_25672_TCP_PROTO': 'tcp', 'CATALOGUE_DB_ENV_USERNAME': 'docker', 'CATALOGUE_WORKER_1_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_1_ENV_RABBITMQ_NODENAME': 'rabbit', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_PROTO': 'tcp', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_LANG': 'C.UTF-8', 'RABBITMQ_1_PORT_5672_TCP': 'tcp://172.17.0.4:5672', 'RABBITMQ_1_PORT_5671_TCP': 'tcp://172.17.0.4:5671', 'DEBIAN_FRONTEND': 'noninteractive', 'WORKER_ENV_PYTHON_VERSION': '2.7.11', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_PROTO': 'tcp', 'WORKER_1_PORT': 'tcp://172.17.0.5:8080', 'CATALOGUE_UWSGI_ENV_PYTHON_PIP_VERSION': '8.1.0', 'CATALOGUE_UWSGI_ENV_RABBITMQ_HOST': 'rabbitmq', 'CATALOGUE_WORKER_1_ENV_DATABASE_USERNAME': 'docker', 'DB_PORT_5432_TCP': 'tcp://172.17.0.3:5432', 'CATALOGUE_WORKER_1_PORT_8080_TCP': 'tcp://172.17.0.5:8080', 'WORKER_PORT_8080_TCP_ADDR': '172.17.0.5', 'SMTP_NAME': '/catalogue_uwsgi_run_124/smtp', 'WORKER_PORT_8080_TCP_PORT': '8080', 'WORKER_1_ENV_LANG': 'C.UTF-8', 'CATALOGUE_RABBITMQ_1_NAME': '/catalogue_uwsgi_run_124/catalogue_rabbitmq_1', 'RABBITMQ_PORT_25672_TCP_PROTO': 'tcp', 'WORKER_1_PORT_8080_TCP': 'tcp://172.17.0.5:8080', 'CATALOGUE_WORKER_1_PORT_8080_TCP_PROTO': 'tcp', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_PORT': '5671', 'UWSGI_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'RABBITMQ_1_PORT_5672_TCP_ADDR': '172.17.0.4', 'CATALOGUE_SMTP_ENV_maildomain': 'kartoza.com', 'RABBITMQ_1_PORT_25672_TCP_ADDR': '172.17.0.4', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_PORT': '5672', 'RABBITMQ_ENV_RABBITMQ_DEBIAN_VERSION': '3.6.1-1', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP': 'tcp://172.17.0.4:5672', 'WORKER_ENV_VIRTUAL_PORT': '8080', 'DB_ENV_PASS': 'docker', 'UWSGI_NAME': '/catalogue_uwsgi_run_124/uwsgi', 'CATALOGUE_DB_ENV_DEBIAN_FRONTEND': 'noninteractive', 'WORKER_ENV_DATABASE_USERNAME': 'docker', 'CATALOGUE_DB_PORT_5432_TCP_PROTO': 'tcp', 'WORKER_PORT': 'tcp://172.17.0.5:8080', 'UWSGI_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'UWSGI_ENV_PYTHON_PIP_VERSION': '8.1.0', 'UWSGI_PORT_8080_TCP_ADDR': '172.17.0.6', 'RABBITMQ_PORT_25672_TCP_ADDR': '172.17.0.4', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'CATALOGUE_WORKER_1_ENV_DATABASE_NAME': 'gis', 'RABBITMQ_PORT_5672_TCP': 'tcp://172.17.0.4:5672', 'CATALOGUE_SMTP_NAME': '/catalogue_uwsgi_run_124/catalogue-smtp', 'CATALOGUE_WORKER_1_ENV_DATABASE_HOST': 'db', 'RABBITMQ_1_ENV_RABBITMQ_DEBIAN_VERSION': '3.6.1-1', 'TERM': 'xterm', 'RABBITMQ_ENV_RABBITMQ_SASL_LOGS': '-', 'RABBITMQ_1_PORT_5671_TCP_ADDR': '172.17.0.4', 'CATALOGUE_UWSGI_ENV_VIRTUAL_HOST': 'catalogue.kartoza.com', 'CATALOGUE_WORKER_1_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_NODENAME': 'rabbit', 'WORKER_1_PORT_8080_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT': 'tcp://172.17.0.4:4369', 'UWSGI_ENV_LANG': 'C.UTF-8', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_LOGS': '-', 'WORKER_1_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_DB_PORT_5432_TCP_PORT': '5432', 'WORKER_ENV_DATABASE_PASSWORD': 'docker', 'RABBITMQ_ENV_USER': 'rabbit_user', 'CATALOGUE_RABBITMQ_1_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'WORKER_1_ENV_VIRTUAL_PORT': '8080', 'WORKER_ENV_DATABASE_HOST': 'db', 'UWSGI_ENV_DATABASE_NAME': 'gis', 'DATABASE_USERNAME': 'docker', 'CATALOGUE_WORKER_1_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_1_ENV_PYTHON_PIP_VERSION': '8.1.0', 'DATABASE_PASSWORD': 'docker', 'DB_PORT_5432_TCP_ADDR': '172.17.0.3', 'CATALOGUE_UWSGI_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_PORT_5671_TCP_PROTO': 'tcp', 'WORKER_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'RABBITMQ_1_PORT_4369_TCP': 'tcp://172.17.0.4:4369', 'UWSGI_ENV_PYTHON_VERSION': '2.7.11', 'CATALOGUE_UWSGI_ENV_DATABASE_USERNAME': 'docker', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP': 'tcp://172.17.0.4:4369', 'RABBITMQ_1_PORT_5671_TCP_PORT': '5671', 'RABBITMQ_ENV_RABBITMQ_NODENAME': 'rabbit', 'WORKER_1_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'WORKER_1_NAME': '/catalogue_uwsgi_run_124/worker_1', 'WORKER_1_ENV_DATABASE_HOST': 'db', 'WORKER_1_ENV_DATABASE_PASSWORD': 'docker', 'PWD': '/home/web/django_project', 'RABBITMQ_1_NAME': '/catalogue_uwsgi_run_124/rabbitmq_1', 'RABBITMQ_1_ENV_RABBITMQ_VERSION': '3.6.1', 'CATALOGUE_WORKER_1_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'UWSGI_ENV_VIRTUAL_PORT': '8080', 'CATALOGUE_DB_ENV_PASS': 'docker', 'CATALOGUE_DB_PORT_5432_TCP': 'tcp://172.17.0.3:5432'}
System check identified some issues:
WARNINGS:
catalogue.OpticalProduct.cloud_cover: (fields.W122) 'max_length' is ignored when used with IntegerField
HINT: Remove 'max_length' from field
orders.OrderNotificationRecipients.classes: (fields.W340) null has no effect on ManyToManyField.
orders.OrderNotificationRecipients.satellite_instrument_group: (fields.W340) null has no effect on ManyToManyField.
search.Search.cloud_max: (fields.W122) 'max_length' is ignored when used with IntegerField
HINT: Remove 'max_length' from field
search.Search.cloud_min: (fields.W122) 'max_length' is ignored when used with IntegerField
HINT: Remove 'max_length' from field
search.Search.collection: (fields.W340) null has no effect on ManyToManyField.
search.Search.instrument_type: (fields.W340) null has no effect on ManyToManyField.
search.Search.license_type: (fields.W340) null has no effect on ManyToManyField.
search.Search.processing_level: (fields.W340) null has no effect on ManyToManyField.
search.Search.satellite: (fields.W340) null has no effect on ManyToManyField.
search.Search.spectral_group: (fields.W340) null has no effect on ManyToManyField.
Scanning folders in /home/web/django_project/data/SPOT/
Trying to update
Imported scene : spot7
Trying to update
Imported scene : spot6
===============================
Products processed : 2
Products updated : 2
Products imported : 0
Products failed to import : 0
===============================
From: Boney Bun [mailto:[email protected]]
Sent: Tuesday, December 26, 2017 9:19 AM
To: kartoza/catalogue <[email protected]>
Cc: Maite Marageni <[email protected]>; Mention <[email protected]>
Subject: Re: [kartoza/catalogue] Catalogue update (#496)
Hi @Maite-Machipi<https://github.com/maite-machipi>,
It turns out the path for spot6 or spot7 ingestion is slightly different.
I've updated the wiki for ingestion according to your feedback.
The ingestion will try to find xml files on the folders.
The path should be: ./data/SPOT, not ./data/SPOT/spot7.
I've done the ingestion, below is the snapshot of the shell:
root@uwsgi:/home/web/django_project# python manage.py spot67_harvest source=./data/SPOT
{'WORKER_ENV_C_FORCE_ROOT': 'true', 'VIRTUAL_HOST': 'catalogue.kartoza.com', 'CATALOGUE_UWSGI_ENV_VIRTUAL_PORT': '8080', 'DB_ENV_USERNAME': 'docker', 'RABBITMQ_1_PORT_5672_TCP_PORT': '5672', 'RABBITMQ_PORT_4369_TCP_ADDR': '172.17.0.4', 'UWSGI_PORT_8080_TCP': 'tcp://172.17.0.6:8080', 'CATALOGUE_RABBITMQ_1_ENV_USER': 'rabbit_user', 'WORKER_ENV_PYTHON_PIP_VERSION': '8.1.0', 'CATALOGUE_UWSGI_ENV_LANG': 'C.UTF-8', 'CATALOGUE_WORKER_1_NAME': '/catalogue_uwsgi_run_78/catalogue_worker_1', 'WORKER_1_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_ENV_RABBITMQ_LOGS': '-', 'PYTHON_VERSION': '2.7.11', 'CATALOGUE_SMTP_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_ADDR': '172.17.0.4', 'SMTP_ENV_maildomain': 'kartoza.com', 'WORKER_1_PORT_8080_TCP_PORT': '8080', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP': 'tcp://172.17.0.4:5671', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_ADDR': '172.17.0.4', 'DB_PORT_5432_TCP_PROTO': 'tcp', 'WORKER_NAME': '/catalogue_uwsgi_run_78/worker', 'WORKER_ENV_LANG': 'C.UTF-8', 'RABBITMQ_PORT_5672_TCP_PROTO': 'tcp', 'RABBITMQ_1_ENV_USER': 'rabbit_user', 'UWSGI_ENV_DATABASE_USERNAME': 'docker', 'RABBITMQ_PORT': 'tcp://172.17.0.4:4369', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_SASL_LOGS': '-', 'RABBITMQ_HOST': 'rabbitmq', 'RABBITMQ_1_PORT_5671_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_DATABASE_PASSWORD': 'docker', 'CATALOGUE_UWSGI_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_UWSGI_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_RABBITMQ_1_ENV_GOSU_VERSION': '1.7', 'RABBITMQ_1_PORT_25672_TCP': 'tcp://172.17.0.4:25672', 'DB_NAME': '/catalogue_uwsgi_run_78/db', 'CATALOGUE_WORKER_1_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_1_ENV_GOSU_VERSION': '1.7', 'DB_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_UWSGI_NAME': '/catalogue_uwsgi_run_78/catalogue-uwsgi', 'CATALOGUE_RABBITMQ_1_ENV_affinity:container': '=450af8e7f3fa0726bfce0e8d0e5add7370fe02e22ba5da46f8577a4b26308605', 'CATALOGUE_UWSGI_ENV_DATABASE_HOST': 'db', 'WORKER_1_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'RABBITMQ_ENV_GOSU_VERSION': '1.7', 'CATALOGUE_WORKER_1_PORT_8080_TCP_ADDR': '172.17.0.5', 'RABBITMQ_PORT_4369_TCP': 'tcp://172.17.0.4:4369', 'RABBITMQ_1_PORT_4369_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_PYTHON_PIP_VERSION': '8.1.0', 'CATALOGUE_WORKER_1_PORT_8080_TCP_PORT': '8080', 'RABBITMQ_1_ENV_RABBITMQ_SASL_LOGS': '-', 'HOSTNAME': 'uwsgi', 'CATALOGUE_WORKER_1_ENV_VIRTUAL_PORT': '8080', 'RABBITMQ_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'RABBITMQ_1_PORT_25672_TCP_PORT': '25672', 'CATALOGUE_UWSGI_PORT_8080_TCP_ADDR': '172.17.0.6', 'DATABASE_HOST': 'db', 'RABBITMQ_NAME': '/catalogue_uwsgi_run_78/rabbitmq', 'CATALOGUE_UWSGI_ENV_DATABASE_PASSWORD': 'docker', 'DB_PORT_5432_TCP_PORT': '5432', 'DB_PORT': 'tcp://172.17.0.3:5432', 'CATALOGUE_DB_PORT_5432_TCP_ADDR': '172.17.0.3', 'UWSGI_PORT_8080_TCP_PORT': '8080', 'WORKER_PORT_8080_TCP': 'tcp://172.17.0.5:8080', 'RABBITMQ_PORT_25672_TCP_ADDR': '172.17.0.4', 'WORKER_1_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_WORKER_1_PORT': 'tcp://172.17.0.5:8080', 'RABBITMQ_1_PORT_4369_TCP_PORT': '4369', 'CATALOGUE_DB_NAME': '/catalogue_uwsgi_run_78/catalogue-db', 'RABBITMQ_1_PORT_5672_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT_4369_TCP_ADDR': '172.17.0.4', 'SHLVL': '1', 'RABBITMQ_PORT_25672_TCP': 'tcp://172.17.0.4:25672', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP': 'tcp://172.17.0.4:25672', 'WORKER_1_ENV_PYTHON_VERSION': '2.7.11', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_ADDR': '172.17.0.4', 'CATALOGUE_DB_PORT': 'tcp://172.17.0.3:5432', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_PROTO': 'tcp', 'UWSGI_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'RABBITMQ_PORT_5672_TCP_PORT': '5672', 'UWSGI_ENV_VIRTUAL_HOST': 'catalogue.kartoza.com', 'RABBITMQ_ENV_affinity:container': '=450af8e7f3fa0726bfce0e8d0e5add7370fe02e22ba5da46f8577a4b26308605', 'UWSGI_PORT_8080_TCP_PROTO': 'tcp', 'SMTP_ENV_smtp_user': 'noreply:docker', 'CATALOGUE_UWSGI_PORT': 'tcp://172.17.0.6:8080', 'WORKER_1_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_PORT_8080_TCP_PROTO': 'tcp', 'VIRTUAL_PORT': '8080', 'RABBITMQ_PORT_5671_TCP_ADDR': '172.17.0.4', 'RABBITMQ_PORT_4369_TCP_PORT': '4369', 'RABBITMQ_1_ENV_RABBITMQ_LOGS': '-', 'WORKER_1_ENV_DATABASE_USERNAME': 'docker', 'WORKER_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', '_': '/usr/local/bin/python', 'WORKER_ENV_DATABASE_NAME': 'gis', 'CATALOGUE_WORKER_1_ENV_C_FORCE_ROOT': 'true', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_VERSION': '3.6.1', 'RABBITMQ_PORT_5672_TCP_ADDR': '172.17.0.4', 'DATABASE_NAME': 'gis', 'WORKER_1_NAME': '/catalogue_uwsgi_run_78/worker_1', 'WORKER_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_1_ENV_C_FORCE_ROOT': 'true', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_PORT': '25672', 'DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'CATALOGUE_UWSGI_ENV_DEBIAN_FRONTEND': 'noninteractive', 'WORKER_1_PORT_8080_TCP_ADDR': '172.17.0.5', 'CATALOGUE_WORKER_1_ENV_PYTHON_VERSION': '2.7.11', 'WORKER_ENV_DEBIAN_FRONTEND': 'noninteractive', 'UWSGI_ENV_DATABASE_HOST': 'db', 'UWSGI_ENV_DATABASE_PASSWORD': 'docker', 'CATALOGUE_SMTP_ENV_smtp_user': 'noreply:docker', 'UWSGI_PORT': 'tcp://172.17.0.6:8080', 'CATALOGUE_UWSGI_ENV_PYTHON_VERSION': '2.7.11', 'RABBITMQ_PORT_5671_TCP_PORT': '5671', 'RABBITMQ_PORT_5671_TCP': 'tcp://172.17.0.4:5671', 'HOME': '/root', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_DEBIAN_VERSION': '3.6.1-1', 'CATALOGUE_RABBITMQ_1_PORT': 'tcp://172.17.0.4:4369', 'LANG': 'C.UTF-8', 'RABBITMQ_1_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_ADDR': '172.17.0.4', 'UWSGI_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_ENV_RABBITMQ_VERSION': '3.6.1', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_PORT': '4369', 'PYTHON_PIP_VERSION': '8.1.0', 'RABBITMQ_PORT_25672_TCP_PORT': '25672', 'RABBITMQ_1_ENV_affinity:container': '=450af8e7f3fa0726bfce0e8d0e5add7370fe02e22ba5da46f8577a4b26308605', 'CATALOGUE_UWSGI_PORT_8080_TCP_PROTO': 'tcp', 'SMTP_ENV_DEBIAN_FRONTEND': 'noninteractive', 'RABBITMQ_PORT_4369_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT_25672_TCP_PROTO': 'tcp', 'CATALOGUE_DB_ENV_USERNAME': 'docker', 'CATALOGUE_WORKER_1_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_1_ENV_RABBITMQ_NODENAME': 'rabbit', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP_PROTO': 'tcp', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_PROTO': 'tcp', 'CATALOGUE_WORKER_1_ENV_LANG': 'C.UTF-8', 'RABBITMQ_1_PORT_5672_TCP': 'tcp://172.17.0.4:5672', 'RABBITMQ_1_PORT_5671_TCP': 'tcp://172.17.0.4:5671', 'DEBIAN_FRONTEND': 'noninteractive', 'WORKER_ENV_PYTHON_VERSION': '2.7.11', 'CATALOGUE_RABBITMQ_1_PORT_25672_TCP_PROTO': 'tcp', 'WORKER_1_PORT': 'tcp://172.17.0.5:8080', 'CATALOGUE_UWSGI_ENV_PYTHON_PIP_VERSION': '8.1.0', 'CATALOGUE_UWSGI_ENV_RABBITMQ_HOST': 'rabbitmq', 'CATALOGUE_WORKER_1_ENV_DATABASE_USERNAME': 'docker', 'DB_PORT_5432_TCP': 'tcp://172.17.0.3:5432', 'CATALOGUE_WORKER_1_PORT_8080_TCP': 'tcp://172.17.0.5:8080', 'WORKER_PORT_8080_TCP_ADDR': '172.17.0.5', 'SMTP_NAME': '/catalogue_uwsgi_run_78/smtp', 'WORKER_PORT_8080_TCP_PORT': '8080', 'WORKER_1_ENV_LANG': 'C.UTF-8', 'CATALOGUE_RABBITMQ_1_NAME': '/catalogue_uwsgi_run_78/catalogue_rabbitmq_1', 'RABBITMQ_PORT_25672_TCP_PROTO': 'tcp', 'WORKER_1_PORT_8080_TCP': 'tcp://172.17.0.5:8080', 'CATALOGUE_WORKER_1_PORT_8080_TCP_PROTO': 'tcp', 'CATALOGUE_RABBITMQ_1_PORT_5671_TCP_PORT': '5671', 'UWSGI_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'RABBITMQ_1_PORT_5672_TCP_ADDR': '172.17.0.4', 'CATALOGUE_SMTP_ENV_maildomain': 'kartoza.com', 'RABBITMQ_1_PORT_25672_TCP_ADDR': '172.17.0.4', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP_PORT': '5672', 'RABBITMQ_ENV_RABBITMQ_DEBIAN_VERSION': '3.6.1-1', 'UWSGI_ENV_VIRTUAL_PORT': '8080', 'WORKER_ENV_VIRTUAL_PORT': '8080', 'DB_ENV_PASS': 'docker', 'UWSGI_NAME': '/catalogue_uwsgi_run_78/uwsgi', 'CATALOGUE_DB_ENV_DEBIAN_FRONTEND': 'noninteractive', 'WORKER_ENV_DATABASE_USERNAME': 'docker', 'CATALOGUE_DB_PORT_5432_TCP_PROTO': 'tcp', 'WORKER_PORT': 'tcp://172.17.0.5:8080', 'UWSGI_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'CATALOGUE_UWSGI_PORT_8080_TCP_PORT': '8080', 'UWSGI_ENV_PYTHON_PIP_VERSION': '8.1.0', 'UWSGI_PORT_8080_TCP_ADDR': '172.17.0.6', 'CATALOGUE_RABBITMQ_1_PORT_5672_TCP': 'tcp://172.17.0.4:5672', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'CATALOGUE_WORKER_1_ENV_DATABASE_NAME': 'gis', 'RABBITMQ_PORT_5672_TCP': 'tcp://172.17.0.4:5672', 'CATALOGUE_SMTP_NAME': '/catalogue_uwsgi_run_78/catalogue-smtp', 'CATALOGUE_WORKER_1_ENV_DATABASE_HOST': 'db', 'RABBITMQ_1_ENV_RABBITMQ_DEBIAN_VERSION': '3.6.1-1', 'TERM': 'xterm', 'RABBITMQ_ENV_RABBITMQ_SASL_LOGS': '-', 'RABBITMQ_1_PORT_5671_TCP_ADDR': '172.17.0.4', 'CATALOGUE_UWSGI_ENV_VIRTUAL_HOST': 'catalogue.kartoza.com', 'CATALOGUE_WORKER_1_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_NODENAME': 'rabbit', 'WORKER_1_PORT_8080_TCP_PROTO': 'tcp', 'RABBITMQ_1_PORT': 'tcp://172.17.0.4:4369', 'UWSGI_ENV_LANG': 'C.UTF-8', 'CATALOGUE_RABBITMQ_1_ENV_RABBITMQ_LOGS': '-', 'WORKER_1_ENV_DEBIAN_FRONTEND': 'noninteractive', 'CATALOGUE_DB_PORT_5432_TCP_PORT': '5432', 'WORKER_ENV_DATABASE_PASSWORD': 'docker', 'RABBITMQ_ENV_USER': 'rabbit_user', 'CATALOGUE_RABBITMQ_1_ENV_RABBIT_PASSWORD': 'rabbit_test_password', 'GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'WORKER_1_ENV_VIRTUAL_PORT': '8080', 'WORKER_ENV_DATABASE_HOST': 'db', 'UWSGI_ENV_DATABASE_NAME': 'gis', 'DATABASE_USERNAME': 'docker', 'CATALOGUE_WORKER_1_ENV_RABBITMQ_HOST': 'rabbitmq', 'WORKER_1_ENV_PYTHON_PIP_VERSION': '8.1.0', 'DB_PORT_5432_TCP_ADDR': '172.17.0.3', 'CATALOGUE_UWSGI_ENV_DJANGO_SETTINGS_MODULE': 'core.settings.prod_docker', 'RABBITMQ_PORT_5671_TCP_PROTO': 'tcp', 'WORKER_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'RABBITMQ_1_PORT_4369_TCP': 'tcp://172.17.0.4:4369', 'UWSGI_ENV_PYTHON_VERSION': '2.7.11', 'CATALOGUE_UWSGI_ENV_DATABASE_USERNAME': 'docker', 'CATALOGUE_RABBITMQ_1_PORT_4369_TCP': 'tcp://172.17.0.4:4369', 'RABBITMQ_1_PORT_5671_TCP_PORT': '5671', 'RABBITMQ_ENV_RABBITMQ_NODENAME': 'rabbit', 'WORKER_1_ENV_GPG_KEY': 'C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF', 'CATALOGUE_UWSGI_PORT_8080_TCP': 'tcp://172.17.0.6:8080', 'WORKER_1_ENV_DATABASE_HOST': 'db', 'WORKER_1_ENV_DATABASE_PASSWORD': 'docker', 'PWD': '/home/web/django_project', 'RABBITMQ_1_NAME': '/catalogue_uwsgi_run_78/rabbitmq_1', 'RABBITMQ_1_ENV_RABBITMQ_VERSION': '3.6.1', 'CATALOGUE_WORKER_1_ENV_VIRTUAL_HOST': 'catalogue.kartoza.org', 'DATABASE_PASSWORD': 'docker', 'CATALOGUE_DB_ENV_PASS': 'docker', 'CATALOGUE_DB_PORT_5432_TCP': 'tcp://172.17.0.3:5432'}
System check identified some issues:
WARNINGS:
catalogue.OpticalProduct.cloud_cover: (fields.W122) 'max_length' is ignored when used with IntegerField
HINT: Remove 'max_length' from field
orders.OrderNotificationRecipients.classes: (fields.W340) null has no effect on ManyToManyField.
orders.OrderNotificationRecipients.satellite_instrument_group: (fields.W340) null has no effect on ManyToManyField.
search.Search.cloud_max: (fields.W122) 'max_length' is ignored when used with IntegerField
HINT: Remove 'max_length' from field
search.Search.cloud_min: (fields.W122) 'max_length' is ignored when used with IntegerField
HINT: Remove 'max_length' from field
search.Search.collection: (fields.W340) null has no effect on ManyToManyField.
search.Search.instrument_type: (fields.W340) null has no effect on ManyToManyField.
search.Search.license_type: (fields.W340) null has no effect on ManyToManyField.
search.Search.processing_level: (fields.W340) null has no effect on ManyToManyField.
search.Search.satellite: (fields.W340) null has no effect on ManyToManyField.
search.Search.spectral_group: (fields.W340) null has no effect on ManyToManyField.
Scanning folders in /home/web/django_project/data/SPOT/
Trying to update
Imported scene : spot7
Trying to update
Imported scene : spot6
===============================
Products processed : 2
Products updated : 2
Products imported : 0
Products failed to import : 0
===============================
|
Open
|
Thanks for your information. Below is the result of ingesting spot folders |
|
Thank you @boney-bun |
|
Thanks @Maite-Machipi Just noticed that the thumbnail functionality has not fully implemented yet. however, i need your advice about the naming convention for thumbnail files. meanwhile, the code will look at the metadata to determine the thumbnails' name, particularly at |
|
Hi @boney-bun yes previously the spot thumbnail name was thumbnail, but Karabo renamed it as it was replacing each other due to the naming. now the metadata & Thumb for e.g spot7 is:
|
|
Thanks for clarifying the naming convention. |
|
hi @boney-bun , the ingest is not picking up the spot metadata, its failing to ingest. |
|
Good morning @Maite-Machipi it failed due to the file extension. Notice one missing file at the end of the ingestion below is because no thumbnail for the metadata. Please don't hesitate to ask me if you found any ingestion error. |
|
thank you @boney-bun |
|
could we close this ticket @Maite-Machipi ? we are now constantly checking the open tickets. |
|
Hi @boney-bun yes ill close this one, as there's a similar one that is open. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @gubuntu can you please provide us with the procedure to update thumbnails and metadata to the catalogue.
The text was updated successfully, but these errors were encountered: