refactor!: Introduce new storage clients [WIP]

[vdusek]

Draft version

• This is a draft PR; currently, only Dataset and KVS-related components are implemented.
• Let's discuss the current state before proceeding with further implementation.

Description

• In this first iteration, the following have been updated or implemented:
  • DatasetClient,
  • FileSystemDatasetClient,
  • MemoryDatasetClient,
  • and the Dataset has been updated accordingly.
• A lot of things from the Dataset were removed and will be implemented in the specific storage clients instead.
• The memory client is now split into file system and memory implementations, eliminating the need for a persist_storage flag.
• All collection clients have been removed.
• Storage client method names have been aligned with storage naming.
• Users are now warned when using method arguments that are not supported.
• Creation management in the storage clients has been removed; creation management in the storages/ module will be removed later.

Suggested Dataset breaking changes

• storage_object → metadata property
• get_info → metadata property
• set_metadata method has been removed - Do we want to support it (e.g. for renaming)?
• from_storage_object method has been removed - Use the open method with name and/or id instead.
• check_and_serialize method has been removed - The clients should handle serialization.
• iterate_items → iterate method

Suggested KeyValueStore breaking changes

• storage_object → metadata property
• get_info → metadata property
• set_metadata method has been removed - Do we want to support it (e.g. for renaming)?
• from_storage_object method has been removed - Use the open method with name and/or id instead.
• persist_autosaved_values - It should be managed by the underlying client.
• get_auto_saved_value - It should be managed by the underlying client.

Issues

• Closes: Separate MemoryStorageClient and FilesystemStorageClient #92
• Closes: Improve storage manager and merge it with creation_management module #147
• Closes: Introduce new base & memory storage clients #783
• Relates: Add support for non-default unnamed storages #1175
• Relates: Update push_data annotations to use JsonSerializable type #1191

Dataset usage example

import asyncio

from crawlee.storage_clients import DatasetStorageClient
from crawlee.storages import Dataset


async def main() -> None:
    dataset = await Dataset.open(
        purge_on_start=False,
        storage_client=DatasetStorageClient(),
    )
    print(f'default dataset - ID: {dataset.id}, name: {dataset.name}')

    await dataset.push_data({'name': 'John'})
    await dataset.push_data({'name': 'John', 'age': 20})
    await dataset.push_data({})

    dataset_with_name = await Dataset.open(
        name='my_dataset',
        storage_client=file_system_storage_client,
    )
    print(f'named dataset - ID: {dataset_with_name.id}, name: {dataset_with_name.name}')

    await dataset_with_name.push_data([{'age': 30}, {'age': 25}])

    print('Default dataset items:')
    async for item in dataset.iterate(skip_empty=True):
        print(item)

    print('Named dataset items:')
    async for item in dataset_with_name.iterate():
        print(item)

    items = await dataset.get_data()
    print(items)


if __name__ == '__main__':
    asyncio.run(main())

KVS usage example

import asyncio

import requests

from crawlee.storage_clients import FileSystemStorageClient
from crawlee.storages import KeyValueStore


async def main() -> None:
    print('Opening key-value store "my_kvs"...')
    storage_client = FileSystemStorageClient()
    kvs = await KeyValueStore.open(name='my_kvs', storage_client=storage_client)

    print('Setting value to "file.json"...')
    await kvs.set_value('file.json', {'key': 'value'})

    print('Setting value to "file.jpg"...')
    response = requests.get('https://avatars.githubusercontent.com/u/25082181')
    await kvs.set_value('file.jpg', response.content)

    print('Iterating over keys:')
    async for key in kvs.iterate_keys():
        print(f'Key: {key}')

    print('Listing keys:')
    keys = [key.key for key in await kvs.list_keys()]
    print(f'Keys: {keys}')

    for key in keys:
        print(f'Getting value of {key}...')
        value = await kvs.get_value(key)
        print(f'Value: {str(value)[:100]}')

    print('Deleting value of "file.json"...')
    await kvs.delete_value('file.json')


if __name__ == '__main__':
    asyncio.run(main())

Testing

• Adjust existing tests and add new ones if necessary once there is agreement on the final form.

Checklist

• CI passed

[Pijukatel]

It would be good to explicitly say in what order some arguments take effect to avoid misinterpretations.

For example: offset + desc

• reverse and then offset
  or
• offset and then reverse?

[vdusek]

Closing in favor of #1194.