Labelfree prediction from Multi-channel input to single-channel output

[KartikeyKansal1]

Hi I've been trying to play around with the labelfree configuration settings to have a 2-channel input image to predict 1-channel output image. I changed spatial_dims to 3 and raw_im_channels to 2 and changed the data config settings. I added an additional transform to load the image from the second channel though not sure if this is the correct way to go.

- _target_: monai.transforms.LoadImaged
        keys: ${source_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            dimension_order_out: ZYX
            C: 1
            Z: 0

- _target_: monai.transforms.LoadImaged
        keys: ${source_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            dimension_order_out: ZYX
            C: 2
            Z: 0

patch_shape is still [64, 64]

Do these changes look alright? as I get the following error:

Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/io/dictionary.py", line 162, in __call__
    data = self._loader(d[key], reader)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/io/array.py", line 282, in __call__
    img_array, meta_data = reader.get_data(img)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/cyto_dl/image/io/monai_bio_reader.py", line 36, in get_data
    data = img_obj.get_image_dask_data(**self.reader_kwargs).compute()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/aicsimageio/aics_image.py", line 649, in get_image_dask_data
    return transforms.reshape_data(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/aicsimageio/transforms.py", line 261, in reshape_data
    data = data[tuple(dim_specs)]
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/dask/array/core.py", line 1980, in __getitem__
    index2 = normalize_index(index, self.shape)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/dask/array/slicing.py", line 917, in normalize_index
    check_index(axis, i, d)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/dask/array/slicing.py", line 989, in check_index
    raise IndexError(
IndexError: Index 1 is out of bounds for axis 1 with size 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/cyto_dl/utils/template_utils.py", line 53, in wrap
    out = task_func(cfg=cfg)
  File "/Users/kartikeykansal/Documents/tensionGAN/Local Conda experiments/240320/cyto-dl-240320/cyto_dl/train.py", line 88, in train
    trainer.fit(model=model, datamodule=data, ckpt_path=cfg.get("ckpt_path"))
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1021, in _run_stage
    self._run_sanity_check()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1050, in _run_sanity_check
    val_loop.run()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 181, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 108, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 126, in __next__
    batch = super().__next__()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 58, in __next__
    batch = next(self.iterator)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 285, in __next__
    out = next(self._iterator)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 123, in __next__
    out = next(self.iterators[0])
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 677, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 298, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 112, in __getitem__
    return self._transform(index)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 418, in _transform
    pre_random_item = self._cachecheck(self.data[index])
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 391, in _cachecheck
    _item_transformed = self._pre_transform(deepcopy(item_transformed))  # keep the original hashed
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 332, in _pre_transform
    item_transformed = self.transform(item_transformed, end=first_random, threading=True)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/compose.py", line 335, in __call__
    result = execute_compose(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.io.dictionary.LoadImaged object at 0x7ff61c7ee020>

Do I only have to worry about the configuration settings or change the loss function as well? My input block has 16 filters and I'm thinking about having 8 filters from each image. Does this sound something that cyto-dl will be able to implement?

[benjijamorris]

the Image loader expects 0-indexed channels, so if you have a 2-channel image it expects channel indices [0,1]. Instead of having two image loading transforms, you can load multiple channels from the same image with C: [0,1]. The default model will pass your two channel input directly to the model, so you'll end up having 16 filters per channel in your first layer. If you want to do 8 filters per channel and combine those somehow, cytodl lets you write your own model and include it in the model config!

[KartikeyKansal1]

Hi I keep running into this error with the suggested model and data configurations.

In 'experiment/im2im/labelfree.yaml': Could not override '[email protected]'.
Did you mean to override data?

These are my labelfree data configurations:

_target_: cyto_dl.datamodules.dataframe.DataframeDatamodule

path: 
cache_dir:

num_workers: 0
batch_size: 1
pin_memory: True
split_column:
columns:
  - ${source_col}
  - ${target_col}

transforms:
  train:
    _target_: monai.transforms.Compose
    transforms:
      # channels are [blank, membrane,blank, structure, blank, nuclear dye, brightfield ]
      - _target_: monai.transforms.LoadImaged
        keys: ${target_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            # NOTE: eval is used so only the experiment file is required to change for beginning users. This is not recommended when creating your own configs.
            dimension_order_out: YX
            C: 0
            Z: 0
      # channels are [blank, membrane,blank, structure, blank, nuclear dye, brightfield ]
      - _target_: monai.transforms.LoadImaged
        keys: ${source_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            dimension_order_out: ZYX
            C: [0,1]
            Z: 0
      - _target_: monai.transforms.EnsureChannelFirstd
        channel_dim: "no_channel"
        keys: ${data.columns}
      - _target_: monai.transforms.Zoomd
        keys: ${data.columns}
        zoom: 1
        keep_size: False
      - _target_: monai.transforms.ToTensord
        keys: ${data.columns}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: ${data.columns}
        channel_wise: True
      - _target_: cyto_dl.image.transforms.RandomMultiScaleCropd
        keys: ${data.columns}
        patch_shape: ${data._aux.patch_shape}
        patch_per_image: 1
        scales_dict: ${kv_to_dict:${data._aux._scales_dict}}

  test:
    _target_: monai.transforms.Compose
    transforms:
      # channels are [blank, membrane,blank, structure, blank, nuclear dye, brightfield ]
      - _target_: monai.transforms.LoadImaged
        keys: ${target_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            # NOTE: eval is used so only the experiment file is required to change for beginning users. This is not recommended when creating your own configs.
            dimension_order_out: YX
            C: 0
            Z: 0
      # channels are [blank, membrane,blank, structure, blank, nuclear dye, brightfield ]
      - _target_: monai.transforms.LoadImaged
        keys: ${source_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            dimension_order_out: ZYX
            C: [0,1]
            Z: 0
      - _target_: monai.transforms.EnsureChannelFirstd
        channel_dim: "no_channel"
        keys: ${data.columns}
      - _target_: monai.transforms.Zoomd
        keys: ${data.columns}
        zoom: 1
        keep_size: False
      - _target_: monai.transforms.ToTensord
        keys: ${data.columns}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: ${data.columns}

  predict:
    _target_: monai.transforms.Compose
    transforms:
      # channels are [blank, membrane,blank, structure, blank, nuclear dye, brightfield ]
      - _target_: monai.transforms.LoadImaged
        keys: ${source_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            dimension_order_out: ZYX
            C: [0,1]
            Z: 0
      - _target_: monai.transforms.EnsureChannelFirstd
        channel_dim: "no_channel"
        keys: ${source_col}
      - _target_: monai.transforms.Zoomd
        keys: ${source_col}
        zoom: 1
        keep_size: False
      - _target_: monai.transforms.ToTensord
        keys: ${source_col}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: ${source_col}
        channel_wise: True

  valid:
    _target_: monai.transforms.Compose
    transforms:
      # channels are [blank, membrane,blank, structure, blank, nuclear dye, brightfield ]
      - _target_: monai.transforms.LoadImaged
        keys: ${target_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            # NOTE: eval is used so only the experiment file is required to change for beginning users. This is not recommended when creating your own configs.
            dimension_order_out: YX
            C: 0
            Z: 0
      # channels are [blank, membrane,blank, structure, blank, nuclear dye, brightfield ]
      - _target_: monai.transforms.LoadImaged
        keys: ${source_col}
        reader:
          - _target_: cyto_dl.image.io.MonaiBioReader
            dimension_order_out: ZYX
            C: [0,1]
            Z: 0
      - _target_: monai.transforms.EnsureChannelFirstd
        channel_dim: "no_channel"
        keys: ${data.columns}
      - _target_: monai.transforms.Zoomd
        keys: ${data.columns}
        zoom: 1
        keep_size: False
      - _target_: monai.transforms.ToTensord
        keys: ${data.columns}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: ${data.columns}
        channel_wise: True
      - _target_: cyto_dl.image.transforms.RandomMultiScaleCropd
        keys: ${data.columns}
        patch_shape: ${data._aux.patch_shape}
        patch_per_image: 1
        scales_dict: ${kv_to_dict:${data._aux._scales_dict}}

_aux:
  _scales_dict:
    - - ${target_col}
      - [1]
    - - ${source_col}
      - [1]

These are my experiment configurations:

## @package _global_

# to execute this experiment run:
# python train.py experiment=example

defaults:
  - override /data: im2im/labelfree.yaml
  - override /model: im2im/labelfree.yaml
  - override /callbacks: default.yaml
  - override /trainer: gpu.yaml
  - override /logger: csv.yaml

# all parameters below will be merged with parameters from default configurations set above
# this allows you to overwrite only specified parameters

tags: ["dev"]
seed: 12345

experiment_name: YOUR_EXP_NAME
run_name: YOUR_RUN_NAME
source_col: brightfield
target_col: signal
spatial_dims: 3
raw_im_channels: 2


trainer:
  max_epochs: 1

data:
  path: ${paths.data_dir}/labelfree
  cache_dir: ${paths.data_dir}/cache
  columns:
  - ${source_col}
  - ${target_col}
  batch_size: 1
  _aux:
    # 2D
    #patch_shape: [64, 64]
    # 3D
    patch_shape: [16, 32, 32]

[benjijamorris]

Nothing looks out of the ordinary in these configs, what is the python command you're running?

[KartikeyKansal1]

I was running python cyto_dl/train.py experiment=im2im/labelfree.yaml

I was able to get past that error and am getting the following error. Can you suggest a way to specify the return_dims @benjijamorris

`Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/io/dictionary.py", line 162, in __call__
    data = self._loader(d[key], reader)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/io/array.py", line 282, in __call__
    img_array, meta_data = reader.get_data(img)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/cyto_dl/image/io/monai_bio_reader.py", line 36, in get_data
    data = img_obj.get_image_dask_data(**self.reader_kwargs).compute()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/aicsimageio/aics_image.py", line 649, in get_image_dask_data
    return transforms.reshape_data(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/aicsimageio/transforms.py", line 183, in reshape_data
    raise ConflictingArgumentsError(
aicsimageio.exceptions.ConflictingArgumentsError: When selecting a multiple dimension indices, the specified dimension must be provided in return_dims. return_dims=ZYX, dimension C = [0, 1]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/cyto_dl/utils/template_utils.py", line 53, in wrap
    out = task_func(cfg=cfg)
  File "/Users/kartikeykansal/Documents/tensionGAN/Local Conda experiments/240322/Experiment 1/cyto-dl-staging-240321/cyto_dl/train.py", line 88, in train
    trainer.fit(model=model, datamodule=data, ckpt_path=cfg.get("ckpt_path"))
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1021, in _run_stage
    self._run_sanity_check()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1050, in _run_sanity_check
    val_loop.run()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 181, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 108, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 126, in __next__
    batch = super().__next__()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/loops/fetchers.py", line 58, in __next__
    batch = next(self.iterator)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 285, in __next__
    out = next(self._iterator)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/lightning/pytorch/utilities/combined_loader.py", line 123, in __next__
    out = next(self.iterators[0])
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 677, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/torch/utils/data/dataset.py", line 298, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 112, in __getitem__
    return self._transform(index)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 418, in _transform
    pre_random_item = self._cachecheck(self.data[index])
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 391, in _cachecheck
    _item_transformed = self._pre_transform(deepcopy(item_transformed))  # keep the original hashed
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/data/dataset.py", line 332, in _pre_transform
    item_transformed = self.transform(item_transformed, end=first_random, threading=True)
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/compose.py", line 335, in __call__
    result = execute_compose(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/Users/kartikeykansal/miniconda3/envs/cytoenv240131/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.io.dictionary.LoadImaged object at 0x7f814a24d390>