Skip to content

Extract folder scanning module #212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Extract folder scanning module #212

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
43dfd5f
Move chunks to `scan` module
agoncharov-reef May 17, 2022
ba50848
Fix file headers
agoncharov-reef May 17, 2022
d1ca3ce
UnSyncableFilename -> UnsupportedFilename
agoncharov-reef May 17, 2022
11cb325
Refactor super()
agoncharov-reef May 17, 2022
bc38109
SyncPath -> ScanPath
agoncharov-reef May 17, 2022
eddc479
Separate Report from SyncReport
agoncharov-reef May 17, 2022
48dcf4c
Reformat imports
agoncharov-reef May 17, 2022
bf42cb7
Don't create redundant `keep_days` variable
agoncharov-reef May 17, 2022
8d0023f
Move `zip_folders` to `scan/scan.py`
agoncharov-reef May 17, 2022
266aa6c
Fix imports
agoncharov-reef May 17, 2022
95c39b8
Fix tests
agoncharov-reef May 17, 2022
19fb2b5
Lint fix
agoncharov-reef May 17, 2022
cdba814
Update changelog
agoncharov-reef May 17, 2022
8ac1500
Update CHANGELOG.md
agoncharov-reef May 19, 2022
df25245
Avoid calling time.time()
agoncharov-reef May 23, 2022
60b0641
Minor fixes
agoncharov-reef May 23, 2022
d8a4c45
Extract scan tests from sync unit tests
agoncharov-reef May 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 17 additions & 13 deletions b2sdk/_v3/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,14 +164,7 @@
from b2sdk.sync.action import B2HideAction
from b2sdk.sync.action import B2UploadAction
from b2sdk.sync.action import LocalDeleteAction
from b2sdk.sync.exception import EnvironmentEncodingError
from b2sdk.sync.exception import IncompleteSync
from b2sdk.sync.exception import InvalidArgument
from b2sdk.sync.folder import AbstractFolder
from b2sdk.sync.folder import B2Folder
from b2sdk.sync.folder import LocalFolder
from b2sdk.sync.folder_parser import parse_sync_folder
from b2sdk.sync.path import AbstractSyncPath, B2SyncPath, LocalSyncPath
from b2sdk.sync.policy import AbstractFileSyncPolicy
from b2sdk.sync.policy import CompareVersionMode
from b2sdk.sync.policy import NewerFileSyncMode
Expand All @@ -189,19 +182,30 @@
from b2sdk.sync.policy_manager import POLICY_MANAGER
from b2sdk.sync.report import SyncFileReporter
from b2sdk.sync.report import SyncReport
from b2sdk.sync.scan_policies import DEFAULT_SCAN_MANAGER
from b2sdk.sync.scan_policies import IntegerRange
from b2sdk.sync.scan_policies import RegexSet
from b2sdk.sync.scan_policies import ScanPoliciesManager
from b2sdk.sync.scan_policies import convert_dir_regex_to_dir_prefix_regex
from b2sdk.sync.sync import KeepOrDeleteMode
from b2sdk.sync.sync import Synchronizer
from b2sdk.sync.sync import zip_folders
from b2sdk.sync.encryption_provider import AbstractSyncEncryptionSettingsProvider
from b2sdk.sync.encryption_provider import BasicSyncEncryptionSettingsProvider
from b2sdk.sync.encryption_provider import ServerDefaultSyncEncryptionSettingsProvider
from b2sdk.sync.encryption_provider import SERVER_DEFAULT_SYNC_ENCRYPTION_SETTINGS_PROVIDER

# scan

from b2sdk.scan.exception import EnvironmentEncodingError
from b2sdk.scan.exception import InvalidArgument
from b2sdk.scan.folder import AbstractFolder
from b2sdk.scan.folder import B2Folder
from b2sdk.scan.folder import LocalFolder
from b2sdk.scan.folder_parser import parse_folder
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in v2 both parse_sync_folder and parse_folder should be available and there should be a test that makes sure we have it

from b2sdk.scan.path import AbstractPath, B2Path, LocalPath
from b2sdk.scan.policies import convert_dir_regex_to_dir_prefix_regex
from b2sdk.scan.policies import DEFAULT_SCAN_MANAGER
from b2sdk.scan.policies import IntegerRange
from b2sdk.scan.policies import RegexSet
from b2sdk.scan.policies import ScanPoliciesManager
from b2sdk.scan.report import Report
from b2sdk.scan.scan import zip_folders

# replication

from b2sdk.replication.setting import ReplicationConfigurationFactory
Expand Down
14 changes: 7 additions & 7 deletions b2sdk/_v3/exception.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,14 +71,14 @@
from b2sdk.exception import SSECKeyError
from b2sdk.exception import WrongEncryptionModeForBucketDefault
from b2sdk.exception import interpret_b2_error
from b2sdk.sync.exception import EmptyDirectory
from b2sdk.sync.exception import EnvironmentEncodingError
from b2sdk.sync.exception import IncompleteSync
from b2sdk.sync.exception import InvalidArgument
from b2sdk.sync.exception import NotADirectory
from b2sdk.sync.exception import UnableToCreateDirectory
from b2sdk.sync.exception import UnSyncableFilename
from b2sdk.sync.exception import check_invalid_argument
from b2sdk.scan.exception import UnableToCreateDirectory
from b2sdk.scan.exception import EmptyDirectory
from b2sdk.scan.exception import EnvironmentEncodingError
from b2sdk.scan.exception import InvalidArgument
from b2sdk.scan.exception import NotADirectory
from b2sdk.scan.exception import UnsupportedFilename
from b2sdk.scan.exception import check_invalid_argument

__all__ = (
'AccessDenied',
Expand Down
2 changes: 1 addition & 1 deletion b2sdk/scan/folder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ def __init__(self, bucket_name, folder_name, api):
self.prefix = '' if self.folder_name == '' else self.folder_name + '/'

def all_files(
self, reporter: Report, policies_manager: ScanPoliciesManager = DEFAULT_SCAN_MANAGER
self, reporter, policies_manager: ScanPoliciesManager = DEFAULT_SCAN_MANAGER
):
"""
Yield all files.
Expand Down
1 change: 1 addition & 0 deletions b2sdk/sync/exception.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
######################################################################

from ..exception import B2SimpleError
from ..scan.exception import BaseDirectoryError


class IncompleteSync(B2SimpleError):
Expand Down
6 changes: 3 additions & 3 deletions doc/source/api/sync.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Following are the important optional arguments that can be provided while initia
.. code-block:: python

>>> from b2sdk.v2 import ScanPoliciesManager
>>> from b2sdk.v2 import parse_sync_folder
>>> from b2sdk.v2 import parse_folder
>>> from b2sdk.v2 import Synchronizer
>>> from b2sdk.v2 import KeepOrDeleteMode, CompareVersionMode, NewerFileSyncMode
>>> import time
Expand All @@ -42,8 +42,8 @@ Following are the important optional arguments that can be provided while initia
>>> source = '/home/user1/b2_example'
>>> destination = 'b2://example-mybucket-b2'

>>> source = parse_sync_folder(source, b2_api)
>>> destination = parse_sync_folder(destination, b2_api)
>>> source = parse_folder(source, b2_api)
>>> destination = parse_folder(destination, b2_api)

>>> policies_manager = ScanPoliciesManager(exclude_all_symlinks=True)

Expand Down
6 changes: 3 additions & 3 deletions doc/source/quick_start.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Synchronization
.. code-block:: python

>>> from b2sdk.v2 import ScanPoliciesManager
>>> from b2sdk.v2 import parse_sync_folder
>>> from b2sdk.v2 import parse_folder
>>> from b2sdk.v2 import Synchronizer
>>> from b2sdk.v2 import SyncReport
>>> import time
Expand All @@ -37,8 +37,8 @@ Synchronization
>>> source = '/home/user1/b2_example'
>>> destination = 'b2://example-mybucket-b2'

>>> source = parse_sync_folder(source, b2_api)
>>> destination = parse_sync_folder(destination, b2_api)
>>> source = parse_folder(source, b2_api)
>>> destination = parse_folder(destination, b2_api)

>>> policies_manager = ScanPoliciesManager(exclude_all_symlinks=True)

Expand Down