New API design for datasets (#206)

* new API design for datasets

* tuorial fix

* clean up for naming fix and docstring fixes

* more test cases for collate function, batching, internal dataset etc

* updates to the getitem/reduce functions, but this does not work

* fixed collate function bug for string datasets, not sure if working as intended

* rebased and updated broken documentation

* fixed issues where nested columns did not work. Changed nested column __getitem__() method to accept subsample  key names as well

* making it work

* updated docstrings

Co-authored-by: Rick Izzo <[email protected]>

Author: Sherin Thomas

.github/workflows/testsuite.yml

@@ -29,6 +29,9 @@ jobs:
           # build time with limited macos jobs
           - platform: macos-latest
             python-version: 3.7
+          - platform: windows-latest
+            python-version: 3.7
+            testml: yes
 
     steps:
     - uses: actions/checkout@v2
@@ -43,14 +46,14 @@ jobs:
         python -m pip install tox-gh-actions
     - name: Run Tests Without Coverage Report
       if: matrix.testcover == 'no'
-      run: tox -- -p no:sugar
+      run: tox
       env:
         PYTEST_XDIST_PROC_NR: 2
         TESTCOVER: ${{ matrix.testcover }}
         TESTML: ${{ matrix.testml }}
     - name: Run Tests With Coverage Report
       if: matrix.testcover == 'yes'
-      run: tox -- --cov-report xml -p no:sugar
+      run: tox -- --cov-report xml
       env:
         PYTEST_XDIST_PROC_NR: 2
         TESTCOVER: ${{ matrix.testcover }}

CHANGELOG.rst

@@ -3,6 +3,15 @@ Change Log
 ==========
 
 
+_`In-Progress`
+==============
+
+Improvements
+------------
+
+* New API design for datasets (previously dataloaders) for machine learning libraries.
+  (`#187 <https://github.com/tensorwerk/hangar-py/pull/187>`__) `@hhsecond <<https://github.com/hhsecond>>`__
+
 `v0.5.2`_ (2020-05-08)
 ======================
 

MANIFEST.in

@@ -13,7 +13,9 @@ include CODE_OF_CONDUCT.rst
 include LICENSE
 include README.rst
 
-include tox.ini .travis.yml mypy.ini
+include tox.ini
+include mypy.ini
+include setup.py
 
 global-exclude *.py[cod] *.so *.DS_Store
 global-exclude __pycache__ .mypy_cache .pytest_cache .hypothesis

docs/Tutorial-Dataloader.ipynb renamed to docs/Tutorial-Dataset.ipynb

@@ -161,7 +161,7 @@
    },
    "source": [
     "### Let's make a Tensorflow dataloader\n",
-    "Hangar provides `make_tf_dataset` & `make_torch_dataset` for creating Tensorflow & PyTorch datasets from Hangar columns. You can read more about it in the [documentation](https://hangar-py.readthedocs.io/en/latest/api.html#ml-framework-dataloaders). Next we'll make a Tensorflow dataset and loop over it to make sure we have got a proper Tensorflow dataset."
+    "Hangar provides `make_numpy_dataset`, `make_tensorflow_dataset` & `make_torch_dataset` for creating Tensorflow & PyTorch datasets from Hangar columns. You can read more about it in the [documentation](https://hangar-py.readthedocs.io/en/latest/api.html#ml-framework-dataloaders). Next we'll make a Tensorflow dataset and loop over it to make sure we have got a proper Tensorflow dataset."
    ]
   },
   {
@@ -174,7 +174,7 @@
    },
    "outputs": [],
    "source": [
-    "from hangar import make_tf_dataset"
+    "from hangar.dataset import make_tensorflow_dataset"
    ]
   },
   {
@@ -223,7 +223,7 @@
     "from matplotlib.pyplot import imshow\n",
     "co = repo.checkout()\n",
     "image_column = co.columns['images']\n",
-    "dataset = make_tf_dataset(image_column)\n",
+    "dataset = make_tensorflow_dataset(image_column)\n",
     "for image in dataset:\n",
     "    imshow(image[0].numpy())\n",
     "    break"
@@ -530,7 +530,7 @@
     "### Dataloaders for training\n",
     "We are using Tensorflow to build the network but how do we load this data from Hangar repository to Tensorflow?\n",
     "\n",
-    "A naive option would be to run through the samples and load the numpy arrays and pass that to the `sess.run` of Tensorflow. But that would be quite inefficient. Tensorflow uses multiple threads to load the data in memory and its dataloaders can prefetch the data before-hand so that your training loop doesn't get blocked while loading the data. Also, Tensoflow dataloaders brings batching, shuffling, etc. to the table prebuilt. That's cool but how to load data from Hangar to Tensorflow using TF dataset? Well, we have `make_tf_dataset` which accepts the list of columns as a parameter and returns a TF dataset object."
+    "A naive option would be to run through the samples and load the numpy arrays and pass that to the `sess.run` of Tensorflow. But that would be quite inefficient. Tensorflow uses multiple threads to load the data in memory and its dataloaders can prefetch the data before-hand so that your training loop doesn't get blocked while loading the data. Also, Tensoflow dataloaders brings batching, shuffling, etc. to the table prebuilt. That's cool but how to load data from Hangar to Tensorflow using TF dataset? Well, we have `make_tensorflow_dataset` which accepts the list of columns as a parameter and returns a TF dataset object."
    ]
   },
   {
@@ -555,7 +555,7 @@
     }
    ],
    "source": [
-    "from hangar import make_tf_dataset\n",
+    "from hangar.dataset import make_tensorflow_dataset\n",
     "co = repo.checkout()  # we don't need write checkout here"
    ]
   },
@@ -601,7 +601,7 @@
     "captions_dset = co.columns['captions']\n",
     "pimages_dset = co.columns['processed_images']\n",
     "\n",
-    "dataset = make_tf_dataset([pimages_dset, captions_dset], shuffle=True)"
+    "dataset = make_tensorflow_dataset([pimages_dset, captions_dset], shuffle=True)"
    ]
   },
   {
@@ -613,7 +613,7 @@
    "source": [
     "### Padded Batching\n",
     "\n",
-    "Batching needs a bit more explanation here since the dataset does not just consist of fixed shaped data. We have two dataset in which one is for captions. As you know captions are sequences which can be variably shaped. So instead of using `dataset.batch` we need to use `dataset.padded_batch` which takes care of padding the tensors with the longest value in each dimension for each batch. This `padded_batch` needs the shape by which the user needs the batch to be padded. Unless you need customization, you can use the shape stored in the `dataset` object by `make_tf_dataset` function."
+    "Batching needs a bit more explanation here since the dataset does not just consist of fixed shaped data. We have two dataset in which one is for captions. As you know captions are sequences which can be variably shaped. So instead of using `dataset.batch` we need to use `dataset.padded_batch` which takes care of padding the tensors with the longest value in each dimension for each batch. This `padded_batch` needs the shape by which the user needs the batch to be padded. Unless you need customization, you can use the shape stored in the `dataset` object by `make_tensorflow_dataset` function."
    ]
   },
   {
@@ -965,7 +965,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.3"
+   "version": "3.7.7"
   }
  },
  "nbformat": 4,

docs/api.rst

@@ -132,9 +132,14 @@ ML Framework Dataloaders
 Tensorflow
 ----------
 
-.. autofunction:: hangar.make_tf_dataset
+.. autofunction:: hangar.dataset.make_tensorflow_dataset
 
 Pytorch
 -------
 
-.. autofunction:: hangar.make_torch_dataset
+.. autofunction:: hangar.dataset.make_torch_dataset
+
+Numpy
+-----
+
+.. autofunction:: hangar.dataset.make_numpy_dataset

setup.py

@@ -119,6 +119,7 @@ def run(self):
     join('src', 'hangar', 'records', 'hashmachine.pyx'),
 ]
 CYTHON_HEADERS = [
+    join('src', 'hangar', 'external_cpython.pxd'),
     join('src', 'hangar', 'optimized_utils.pxd'),
     join('src', 'hangar', 'backends', 'specs.pxd'),
     join('src', 'hangar', 'records', 'recordstructs.pxd'),

src/hangar/__init__.py

@@ -1,21 +1,4 @@
 __version__ = '0.5.2'
-__all__ = ('make_torch_dataset', 'make_tf_dataset', 'Repository')
+__all__ = ('Repository',)
 
-from functools import partial
 from .repository import Repository
-
-
-def raise_ImportError(message, *args, **kwargs):
-    raise ImportError(message)
-
-
-try:
-    from .dataloaders.tfloader import make_tf_dataset
-except ImportError:
-    make_tf_dataset = partial(raise_ImportError, "Could not import tensorflow. Install dependencies")
-
-try:
-    from .dataloaders.torchloader import make_torch_dataset
-except ImportError:
-    make_torch_dataset = partial(raise_ImportError, "Could not import torch. Install dependencies")
-

src/hangar/_version.py

@@ -16,6 +16,7 @@
          https://github.com/pypa/packaging/blob/6a09d4015b/LICENSE.BSD
 """
 import re
+import typing
 from collections import namedtuple
 from itertools import dropwhile
 from typing import Callable, Optional, SupportsInt, Tuple, Union
@@ -99,24 +100,25 @@ def __neg__(self) -> InfinityType:
 
 # -------------------- Type Definitions ---------------------------------------
 
-InfiniteTypes = Union[InfinityType, NegativeInfinityType]
-PrePostDevType = Union[InfiniteTypes, Tuple[str, int]]
-SubLocalType = Union[InfiniteTypes, int, str]
-LocalType = Union[
-    NegativeInfinityType,
-    Tuple[
-        Union[
-            SubLocalType,
-            Tuple[SubLocalType, str],
-            Tuple[NegativeInfinityType, SubLocalType],
+if typing.TYPE_CHECKING:
+    InfiniteTypes = Union[InfinityType, NegativeInfinityType]
+    PrePostDevType = Union[InfiniteTypes, Tuple[str, int]]
+    SubLocalType = Union[InfiniteTypes, int, str]
+    LocalType = Union[
+        NegativeInfinityType,
+        Tuple[
+            Union[
+                SubLocalType,
+                Tuple[SubLocalType, str],
+                Tuple[NegativeInfinityType, SubLocalType],
+            ],
+            ...,
         ],
-        ...,
-    ],
-]
-CmpKey = Tuple[
-    int, Tuple[int, ...], PrePostDevType, PrePostDevType, PrePostDevType, LocalType
-]
-VersionComparisonMethod = Callable[[CmpKey, CmpKey], bool]
+    ]
+    CmpKey = Tuple[
+        int, Tuple[int, ...], PrePostDevType, PrePostDevType, PrePostDevType, LocalType
+    ]
+    VersionComparisonMethod = Callable[[CmpKey, CmpKey], bool]
 
 
 # ---------------------------- Version Parsing --------------------------------
@@ -142,7 +144,7 @@ class _BaseVersion(object):
     __slots__ = ('_key',)
 
     def __init__(self):
-        self._key: CmpKey = None
+        self._key: 'CmpKey' = None
 
     def __hash__(self) -> int:
         return hash(self._key)
@@ -165,7 +167,7 @@ def __gt__(self, other: '_BaseVersion') -> bool:
     def __ne__(self, other: object) -> bool:
         return self._compare(other, ne)
 
-    def _compare(self, other: object, method: VersionComparisonMethod
+    def _compare(self, other: object, method: 'VersionComparisonMethod'
                  ) -> Union[bool, type(NotImplemented)]:
         if isinstance(other, _BaseVersion):
             return method(self._key, other._key)
@@ -385,7 +387,7 @@ def _parse_letter_version(
 _local_version_separators = re.compile(r"[\._-]")
 
 
-def _parse_local_version(local: str) -> Optional[LocalType]:
+def _parse_local_version(local: str) -> Optional['LocalType']:
     """
     Takes a string like abc.1.twelve and turns it into ("abc", 1, "twelve").
     """
@@ -403,8 +405,8 @@ def _cmpkey(
         pre: Optional[Tuple[str, int]],
         post: Optional[Tuple[str, int]],
         dev: Optional[Tuple[str, int]],
-        local: Optional[Tuple[SubLocalType]],
-) -> CmpKey:
+        local: Optional[Tuple['SubLocalType']],
+) -> 'CmpKey':
 
     # When we compare a release version, we want to compare it with all of the
     # trailing zeros removed. So we'll use a reverse the list, drop all the now

src/hangar/backends/hdf5_00.py

@@ -186,7 +186,8 @@
 from .. import __version__
 from ..optimized_utils import SizedDict
 from ..constants import DIR_DATA_REMOTE, DIR_DATA_STAGE, DIR_DATA_STORE, DIR_DATA
-from ..utils import find_next_prime, random_string, set_blosc_nthreads
+from ..utils import random_string, set_blosc_nthreads
+from ..optimized_utils import find_next_prime
 from ..op_state import reader_checkout_only, writer_checkout_only
 from ..typesystem import Descriptor, OneOf, DictItems, SizedIntegerTuple, checkedmeta
 

src/hangar/backends/hdf5_01.py

@@ -228,7 +228,8 @@
 from ..optimized_utils import SizedDict
 from ..constants import DIR_DATA_REMOTE, DIR_DATA_STAGE, DIR_DATA_STORE, DIR_DATA
 from ..op_state import writer_checkout_only, reader_checkout_only
-from ..utils import find_next_prime, random_string, set_blosc_nthreads
+from ..utils import random_string, set_blosc_nthreads
+from ..optimized_utils import find_next_prime
 from ..typesystem import Descriptor, OneOf, DictItems, SizedIntegerTuple, checkedmeta
 
 set_blosc_nthreads()

src/hangar/columns/__init__.py

@@ -5,6 +5,7 @@
     generate_nested_column,
     column_type_object_from_schema
 )
+from .introspection import is_column, is_writer_column
 
 __all__ = (
     'Columns',
@@ -13,4 +14,6 @@
     'generate_nested_column',
     'column_type_object_from_schema',
     'ColumnTxn',
+    'is_column',
+    'is_writer_column'
 )

src/hangar/columns/introspection.py

@@ -0,0 +1,27 @@
+from .layout_flat import FlatSampleReader, FlatSampleWriter
+from .layout_nested import (
+    FlatSubsampleReader,
+    FlatSubsampleWriter,
+    NestedSampleReader,
+    NestedSampleWriter
+)
+
+
+def is_column(obj) -> bool:
+    """Determine if arbitrary input is an instance of a column layout.
+
+    Returns
+    -------
+    bool: True if input is an column, otherwise False.
+    """
+    return isinstance(obj, (FlatSampleReader, FlatSubsampleReader, NestedSampleReader))
+
+
+def is_writer_column(obj) -> bool:
+    """Determine if arbitrary input is an instance of a write-enabled column layout.
+
+    Returns
+    -------
+    bool: True if input is write-enabled column, otherwise False.
+    """
+    return isinstance(obj, (FlatSampleWriter, FlatSubsampleWriter, NestedSampleWriter))

src/hangar/columns/layout_flat.py

@@ -7,6 +7,7 @@
 """
 from contextlib import ExitStack
 from pathlib import Path
+from operator import attrgetter as op_attrgetter
 from typing import Tuple, Union, Iterable, Optional, Any
 
 from .common import open_file_handles
@@ -23,7 +24,8 @@
 from ..records.parsing import generate_sample_name
 from ..backends import backend_decoder
 from ..op_state import reader_checkout_only
-from ..utils import is_suitable_user_key, valfilter, valfilterfalse
+from ..utils import is_suitable_user_key
+from ..optimized_utils import valfilter, valfilterfalse
 
 
 KeyType = Union[str, int]
@@ -324,7 +326,8 @@ def contains_remote_references(self) -> bool:
             on some remote server. True if all sample data is available on the
             machine's local disk.
         """
-        return not all(map(lambda x: x.islocal, self._samples.values()))
+        _islocal_func = op_attrgetter('islocal')
+        return not all(map(_islocal_func, self._samples.values()))
 
     @property
     def remote_reference_keys(self) -> Tuple[KeyType]:
@@ -336,7 +339,8 @@ def remote_reference_keys(self) -> Tuple[KeyType]:
             list of sample keys in the column whose data references indicate
             they are stored on a remote server.
         """
-        return tuple(valfilterfalse(lambda x: x.islocal, self._samples).keys())
+        _islocal_func = op_attrgetter('islocal')
+        return tuple(valfilterfalse(_islocal_func, self._samples).keys())
 
     def _mode_local_aware_key_looper(self, local: bool) -> Iterable[KeyType]:
         """Generate keys for iteration with dict update safety ensured.
@@ -352,11 +356,12 @@ def _mode_local_aware_key_looper(self, local: bool) -> Iterable[KeyType]:
         Iterable[KeyType]
             Sample keys conforming to the `local` argument spec.
         """
+        _islocal_func = op_attrgetter('islocal')
         if local:
             if self._mode == 'r':
-                yield from valfilter(lambda x: x.islocal, self._samples).keys()
+                yield from valfilter(_islocal_func, self._samples).keys()
             else:
-                yield from tuple(valfilter(lambda x: x.islocal, self._samples).keys())
+                yield from tuple(valfilter(_islocal_func, self._samples).keys())
         else:
             if self._mode == 'r':
                 yield from self._samples.keys()