Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve storage manager and merge it with creation_management module #147

Open
vdusek opened this issue May 10, 2024 · 2 comments · May be fixed by #1107
Open

Improve storage manager and merge it with creation_management module #147

vdusek opened this issue May 10, 2024 · 2 comments · May be fixed by #1107
Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@vdusek
Copy link
Collaborator

vdusek commented May 10, 2024

The current Crawlee / StorageClientManager is more or less just copied from the Python SDK / StorageClientManager and is extremely simple. Its primary role is to maintain and provide access to storage client instances based on specific input parameters.

The Crawlee TS / StorageManager is more complex and it takes care of more things - creating instances of storages & their caching.

Currently, we have a helper module "creation_management" in storages/ which helps with it.

Let's move logic from storages/creation_management to StorageClientManager and improve the creation & caching process.

Functions get_or_create, find_or_create_client_by_id_or_name a create_*_from_directory should be refactored.

@vdusek vdusek added enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team. labels May 10, 2024
@janbuchar
Copy link
Collaborator

I could also imagine putting the functionality into a module instead of a singleton class, so basically StorageManager -> creation_management, not vice versa.

@vdusek vdusek changed the title Improve storage manager Improve storage manager and merge it with creation_management module May 13, 2024
@janbuchar
Copy link
Collaborator

This code should not check the implementation in use - it's a generic storage manager that should not be concerned with the concrete implementation.

@vdusek vdusek added this to the 0.2 milestone Jun 10, 2024
@vdusek vdusek removed this from the v0.2 milestone Jul 15, 2024
@vdusek vdusek added this to the 110th sprint - Tooling team milestone Mar 11, 2025
@vdusek vdusek linked a pull request Mar 19, 2025 that will close this issue
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants