Merge pull request #107 from SyneRBI/prep_README

update instructions for creating a new dataset

.gitignore

@@ -1,5 +1,6 @@
 data/
 orgdata/
+orgdata
 output/
 tmp*/
 err*.txt
@@ -13,3 +14,6 @@ __pycache__/
 *.ahv
 *.hv
 *.v
+*.csv
+# Tensorboard files
+events.*

SIRF_data_preparation/README.md

@@ -25,3 +25,38 @@ for the Siemens mMR NEMA IQ data (on Zenodo):
 - `download_Siemens_mMR_NEMA_IQ.py`: download and extract
 - `prepare_mMR_NEMA_IQ_data.py`: prepare the data (prompts etc)
 - `BSREM_*.py`: functions with specific settings for a particular data-set
+
+# Steps to follow to prepare data
+If starting from Siemens list-mode data and letting SIRF take care of scatter etc, check for instance [Siemens_mMR_ACR/README.md](steps for Siemens mMR ACR). If pre-prepared data are given, check that naming of all files is correct. KT normally puts all data
+in `~/devel/PETRIC/data/<datasetname>` with `datasetname` following convention of `scanner_phantom-name` as others (instructions below and indeed some scripts might assume this location). Change working directory to where data sits and add PETRIC to your python-path, e.g.
+```
+PYTHONPATH=~/devel/PETRIC:$PYTHONPATH`
+```
+
+1. Run initial [data_QC.py](data_QC)
+   ```
+   python -m SIRF_data_preparation.data_QC
+   ```
+
+2. Run [create_initial_images.py](create_initial_images).
+   ```
+   python -m SIRF_data_preparation.create_initial_images --template_image=<some_image>
+   ```
+   where the template image is one of the given VOIs (it does not matter which one, as they should all have the same geometry). (If you need to create VOIs yourself, you can use `None` or the vendor image).
+3. Edit `OSEM_image.hv` to add modality, radionuclide and duration info which got lost (copy from `prompts.hs`)
+4. Edit [dataset_settings.py](dataset_settings.py) for subsets (used by our reference reconstructions only, not by participants).
+5. Edit [../petric.py](petric.py) for slices to use for creating figures (`DATA_SLICES`). Note that `data_QC` outputs centre-of-mass of the VOIs, which can be helpful for this.
+6. Run [data_QC](data_QC) which should now make more plots. Check VOI alignment etc.
+   ```
+   python -m SIRF_data_preparation.data_QC --dataset=<datasetname>
+   ```
+7. `cd ../..`
+8. `python -m SIRF_data_preparation.run_OSEM <datasetname>`
+9. `python -m SIRF_data_preparation.run_BSREM  <datasetname>`
+10. Adapt [plot_BSREM_metrics.py](plot_BSREM_metrics.py) (probably only the `<datasetname>`) and run interactively.
+11. Copy the BSREM ` iter_final` to `data/<datasetname>/PETRIC/reference_image`, e.g.
+    ```
+    stir_math data/<datasetname>/PETRIC/reference_image.hv output/<datasetname>/iter_final.hv
+    ```
+12. `rm output/<datasetname>/*ahv`, check its `README.md` etc
+13. Transfer to web-server

SIRF_data_preparation/Siemens_mMR_ACR/README.md

@@ -4,7 +4,7 @@ ugly and temporary
 Steps to follow:
 1. `python SIRF_data_preparation/Siemens_mMR_ACR/download.py`
 2. `python SIRF_data_preparation/Siemens_mMR_ACR/prepare.py` (As there is no useful mumap, this will **not** do attenuation and scatter estimation)
-3. `python SIRF_data_preparation/create_initial_images.py data/Siemens_mMR_ACR/final --template_image=None` (Reconstruct at full FOV size)
+3. `python SIRF_data_preparation.create_initial_images data/Siemens_mMR_ACR/final --template_image=None` (Reconstruct at full FOV size)
 4. `mv data/Siemens_mMR_ACR/final/OSEM_image.* data/Siemens_mMR_ACR/processing` (this is really an NAC image and ideally would be renamed)
 5. `python SIRF_data_preparation/Siemens_mMR_ACR/register_mumap.py` (output is orgdata/Siemens_mMR_ACR/processing/reg_mumap.hv etc)
 6. However, registration failed, so this needs a manual intervention step:
@@ -24,9 +24,9 @@ Steps to follow:
    stir_math --accumulate  orgdata/Siemens_mMR_ACR/processing/reg_mumap.hv  orgdata/Siemens_mMR_ACR/output/ACR_hardware-to-STIR.nii.gz
    ```
 7. 11. `python SIRF_data_preparation/run_OSEM.py Siemens_mMR_ACR`--end 200` (As there is now a useful mumap, this will now do attenuation and scatter estimation)
-8. `python SIRF_data_preparation/create_initial_images.py data/Siemens_mMR_ACR --template_image=../../orgdata/Siemens_mMR_ACR/output/sampling_masks/acr-all-sampling-0-2mm_dipy.nii`
-9. `python SIRF_data_preparation/data_QC.py --srcdir='data/Siemens_mMR_ACR' --transverse_slice=99`
-10. edit `SIRF_data_prepatation/dataset_settings.py` for subsets etc. edit `OSEM_image.hv` to add modality, radionuclide and duration info which got lost.
+8. `python -m SIRF_data_preparation.create_initial_images data/Siemens_mMR_ACR --template_image=../../orgdata/Siemens_mMR_ACR/output/sampling_masks/acr-all-sampling-0-2mm_dipy.nii`
+9. `python -m SIRF_data_preparation.data_QC --srcdir='data/Siemens_mMR_ACR' --transverse_slice=99`
+10. edit `petric.py` and `SIRF_data_preparation/dataset_settings.py` for subsets etc. edit `OSEM_image.hv` to add modality, radionuclide and duration info which got lost.
 11. `python SIRF_data_preparation/Siemens_mMR_ACR/VOI_prep.py`
 12. `python SIRF_data_preparation/run_OSEM.py Siemens_mMR_ACR`
 13. `python SIRF_data_preparation/run_BSREM.py Siemens_mMR_ACR`

SIRF_data_preparation/Siemens_mMR_ACR/prepare.py

@@ -2,7 +2,7 @@
 import logging
 import os
 
-from SIRF_data_preparation.data_utilities import prepare_challenge_Siemens_data, the_data_path, the_orgdata_path
+from ..data_utilities import prepare_challenge_Siemens_data, the_data_path, the_orgdata_path
 
 scanID = 'Siemens_mMR_ACR'
 

SIRF_data_preparation/Siemens_mMR_NEMA_IQ/prepare_mMR_NEMA_IQ_data.py

@@ -2,7 +2,7 @@
 import logging
 import os
 
-from .data_utilities import prepare_challenge_Siemens_data, the_data_path
+from ..data_utilities import prepare_challenge_Siemens_data, the_data_path, the_orgdata_path
 
 this_directory = os.path.dirname(__file__)
 repo_directory = os.path.dirname(this_directory)

SIRF_data_preparation/create_initial_images.py

@@ -8,15 +8,15 @@
 
 Options:
   -t <template_image>, --template_image=<template_image>  filename (relative to <data_path>) of image to use
-                                                          for data sizes [default: PETRIC/VOI_whole_phantom.hv]
+                                                          for data sizes [default: PETRIC/VOI_whole_object.hv]
   -s <xy-size>, --xy-size=<xy-size>  force xy-size (do not use when using VOIs as init) [default: -1]
   -S <subsets>, --subsets=<subsets>  number of subsets [default: 2]
   -i <subiterations>, --subiterations=<subiterations>     number of sub-iterations [default: 14]
 """
 # Copyright 2024 Rutherford Appleton Laboratory STFC
 # Copyright 2024 University College London
 # Licence: Apache-2.0
-__version__ = '0.1.0'
+__version__ = '0.2.0'
 
 import logging
 import math

SIRF_data_preparation/data_QC.py

@@ -6,7 +6,7 @@
 
 Options:
   -h, --help
-  --srcdir=<path>        pathname (must have trailing slash!) [default: ./]
+  --srcdir=<path>        pathname. Will default to current directory unless dataset is set
   --skip_sino_profiles   do not plot the sinogram profiles
   --transverse_slice=<i>  idx [default: -1]
   --coronal_slice=<c>    idx [default: -1]
@@ -17,7 +17,7 @@
 """
 # Copyright 2024 University College London
 # Licence: Apache-2.0
-__version__ = '0.3.0'
+__version__ = '0.4.0'
 
 import os
 import os.path
@@ -31,6 +31,7 @@
 
 import sirf.STIR as STIR
 from SIRF_data_preparation.dataset_settings import get_settings
+from SIRF_data_preparation.data_utilities import the_data_path
 
 STIR.AcquisitionData.set_storage_scheme('memory')
 
@@ -163,11 +164,15 @@ def main(argv=None):
     slices["sagittal_slice"] = literal_eval(args['--sagittal_slice'])
 
     if (dataset):
+        if srcdir is None:
+            srcdir = the_data_path(dataset)
         settings = get_settings(dataset)
         for key in slices.keys():
             if slices[key] == -1 and key in settings.slices:
                 slices[key] = settings.slices[key]
-    print(slices)
+    else:
+        if srcdir is None:
+            srcdir = os.getcwd()
 
     if not skip_sino_profiles:
         acquired_data = STIR.AcquisitionData(os.path.join(srcdir, 'prompts.hs'))