refactor!: Introduce new storage clients

[vdusek]

Draft version

• This is a draft PR; only dataset-related components are implemented at this stage.
• Let's discuss the current state before proceeding with the KVS and RQ implementations.

Description

• In this first iteration, the following have been updated or implemented:
  • DatasetClient,
  • FileSystemDatasetClient,
  • MemoryDatasetClient,
  • and the Dataset has been updated accordingly.
• A lot of things from the Dataset were removed and will be implemented in the specific storage clients instead.
• The memory client is now split into file system and memory implementations, eliminating the need for a persist_storage flag.
• All collection clients have been removed.
• Storage client method names have been aligned with storage naming.
• Users are now warned when using method arguments that are not supported.
• Creation management in the storage clients has been removed; creation management in the storages/ module will be removed later.

Suggested Dataset breaking changes

• iterate_items → iterate method
• storage_object → metadata property
• get_info → metadata property
• check_and_serialize method has been removed - The clients should handle serialization.
• from_storage_object method has been removed - Use the open method with name and/or id instead.
• set_metadata method has been removed - Do we want to support it (e.g. for renaming)?

Issues

• Closes: Separate MemoryStorageClient and FilesystemStorageClient #92
• Closes: Improve storage manager and merge it with creation_management module #147
• Closes: Introduce new base & memory storage clients #783

Todo

• I am not sure about the wrapping storage clients - StorageClient, file_system_storage_client, memory_storage_client.
• Implement KVS and RQ once there is a consensus on the final form.

Usage example

import asyncio

from crawlee.storage_clients import file_system_storage_client
from crawlee.storages import Dataset


async def main() -> None:
    dataset = await Dataset.open(
        purge_on_start=False,
        storage_client=file_system_storage_client,
    )
    print(f'default dataset - ID: {dataset.id}, name: {dataset.name}')

    await dataset.push_data({'name': 'John'})
    await dataset.push_data({'name': 'John', 'age': 20})
    await dataset.push_data({})

    dataset_with_name = await Dataset.open(
        name='my_dataset',
        storage_client=file_system_storage_client,
    )
    print(f'named dataset - ID: {dataset_with_name.id}, name: {dataset_with_name.name}')

    await dataset_with_name.push_data([{'age': 30}, {'age': 25}])

    print('Default dataset items:')
    async for item in dataset.iterate(skip_empty=True):
        print(item)

    print('Named dataset items:')
    async for item in dataset_with_name.iterate():
        print(item)

    items = await dataset.get_data()
    print(items)


if __name__ == '__main__':
    asyncio.run(main())

Testing

• Adjust existing tests and add new ones if necessary once there is agreement on the final form.

Checklist

• CI passed

[Pijukatel]

f'The arguments {invalid} of get_data ...

[Pijukatel]

It would be good to explicitly say in what order some arguments take effect to avoid misinterpretations.

For example: offset + desc

• reverse and then offset
  or
• offset and then reverse?