Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finalize switch flow #207

Open
wants to merge 51 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
4d1a825
refactor: Introduce generator class to handle global flow
Hellgartner Jan 27, 2025
fee2d18
refactor: Introduce argument class
Hellgartner Jan 27, 2025
3a118fd
refactor: Remove unnecessary comment
Hellgartner Jan 27, 2025
168ddc1
refactor: Introduce core package
Hellgartner Jan 27, 2025
1ade466
refactor: Introduce file list
Hellgartner Jan 27, 2025
0441c60
refactor: Make internal method "private"
Hellgartner Jan 28, 2025
2c734e6
refactor: Make scancode converter publish to the internal model
Hellgartner Jan 28, 2025
e471c65
refactor: Make opossum converter publish to the internal model
Hellgartner Jan 28, 2025
dd7e456
refactor: Extract reading into the internal format
Hellgartner Jan 28, 2025
08b8a67
refactor: Define interface and implement for opossum file
Hellgartner Jan 28, 2025
95b5523
refactor: Implement for interface for scancode file
Hellgartner Jan 28, 2025
996483a
Merge branch 'main' into finalize_switch_flow
Hellgartner Jan 28, 2025
57adfde
fix: remove spdx remainings
Hellgartner Jan 28, 2025
86ea7cf
refactor: introduce can handle pattern
Hellgartner Jan 28, 2025
5ab9635
refactor: rename
Hellgartner Jan 28, 2025
ed40898
refactor: use Path instead of strings for file paths
Hellgartner Jan 28, 2025
1051de7
refactor: rename
Hellgartner Jan 28, 2025
854e16e
refactor: remove not used merging functionality
Hellgartner Jan 28, 2025
fd3a285
refactor: use common generator helpers for testing
Hellgartner Jan 28, 2025
b1d0e2d
feat: introduce import linter
Hellgartner Jan 28, 2025
b58ba84
refactor: dedicated package for input formats
Hellgartner Jan 28, 2025
18bb9d4
refactor: review-comment: Rename input file type
Hellgartner Jan 28, 2025
308d175
refactor: review-comment: Using property annotation
Hellgartner Jan 28, 2025
c46f1a2
feat: Remove misleading log statement
Hellgartner Jan 28, 2025
51a4603
refactor: review comment: further improve the readability of the gene…
Hellgartner Jan 28, 2025
93ecfe1
refactor: review comment: remove for loop to select reader
Hellgartner Jan 28, 2025
f58490a
refactor: move opossum file model to dedicated package
Hellgartner Jan 29, 2025
d24c105
refactor: move also file writing into the package
Hellgartner Jan 29, 2025
5dd9461
refactor: DIY dependency injection: Define input readers at top level
Hellgartner Jan 29, 2025
780a80b
feat: Add importlinter rule to keep file format independent
Hellgartner Jan 29, 2025
bf6ab6e
fix: Add missing license header
Hellgartner Jan 29, 2025
137a3d0
fix: Remove surplus pass
Hellgartner Jan 29, 2025
9d5f9c9
feat: review-comment: New package structure
Hellgartner Jan 29, 2025
7707e8f
refactor: review-comment: Filename to match main class name
Hellgartner Jan 29, 2025
2348a1a
refactor: review-comment: Move file writer to core
Hellgartner Jan 29, 2025
72dd03e
refactor: review-comment: Fuse closely connected classes
Hellgartner Jan 29, 2025
b8848d5
refactor: review-comment: Fix typo
Hellgartner Jan 29, 2025
7c95235
refactor: review-comment: Smaller fixes
Hellgartner Jan 29, 2025
7b1547a
refactor: review-comment: rename opossum model files
Hellgartner Jan 29, 2025
bc8a7d6
refactor: review-comment: rename all file specific models to end with…
Hellgartner Jan 29, 2025
378c276
refactor: review-comment: use the renamings in converters
Hellgartner Jan 29, 2025
c4fd699
refactor: review-comment: Make tests adhere to the new structure
Hellgartner Jan 29, 2025
ea536f1
refactor: review -- inline validation logic
Hellgartner Jan 30, 2025
30bd42a
refactor: review -- static only classes to methods
Hellgartner Jan 30, 2025
df51d5a
refactor: review -- expand opossum model to multiple files
Hellgartner Jan 30, 2025
c06bb6d
refactor: review -- improve writing to opossum files
Hellgartner Jan 30, 2025
927b3eb
refactor: review -- rename scancode data model
Hellgartner Jan 30, 2025
d774d13
refactor: review -- improve scancode file reader
Hellgartner Jan 30, 2025
a2afebb
refactor: review -- extract shared CamelBaseModel
Hellgartner Jan 30, 2025
5087102
refactor: review -- adapt test structure
Hellgartner Jan 30, 2025
938c94e
refactor: review -- group tests in class structures
Hellgartner Jan 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 10 additions & 51 deletions src/opossum_lib/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,12 @@
# SPDX-License-Identifier: Apache-2.0


import logging
import sys
from pathlib import Path

import click

from opossum_lib.opossum.file_generation import OpossumFileWriter
from opossum_lib.opossum.opossum_file_content import OpossumFileContent
from opossum_lib.opossum.read_opossum_file import read_opossum_file
from opossum_lib.scancode.convert_scancode_to_opossum import (
convert_scancode_file_to_opossum,
from opossum_lib.core.opossum_generator import (
OpossumGenerationArguments,
OpossumGenerator,
)
from opossum_lib.spdx.convert_to_opossum import convert_spdx_to_opossum_information


@click.group()
Expand Down Expand Up @@ -73,49 +66,15 @@ def generate(
- ScanCode
- Opossum
"""
validate_input_and_exit_on_error(spdx_files, scancode_json_files, opossum_files)
opossum_file_content = convert_after_valid_input(
spdx_files, scancode_json_files, opossum_files
)

if not outfile.endswith(".opossum"):
outfile += ".opossum"

if Path.is_file(Path(outfile)):
logging.warning(f"{outfile} already exists and will be overwritten.")

OpossumFileWriter.write_opossum_information_to_file(
opossum_file_content, Path(outfile)
OpossumGenerator.generate(
opossum_generation_arguments=OpossumGenerationArguments(
spdx_files=spdx_files,
opossum_files=opossum_files,
scancode_json_files=scancode_json_files,
outfile=outfile,
)
)


def validate_input_and_exit_on_error(
spdx_files: list[str], scancode_json_files: list[str], opossum_files: list[str]
) -> None:
total_number_of_files = (
len(spdx_files) + len(scancode_json_files) + len(opossum_files)
)
if total_number_of_files == 0:
logging.warning("No input provided. Exiting.")
sys.exit(1)
if total_number_of_files > 1:
logging.error("Merging of multiple files not yet supported!")
sys.exit(1)


def convert_after_valid_input(
spdx_files: list[str], scancode_json_files: list[str], opossum_files: list[str]
) -> OpossumFileContent:
if len(spdx_files) == 1:
spdx_input_file = spdx_files[0]
return convert_spdx_to_opossum_information(spdx_input_file)
elif len(scancode_json_files) == 1:
scancode_json_input_file = scancode_json_files[0]
return convert_scancode_file_to_opossum(scancode_json_input_file)
else:
opossum_input_file = opossum_files[0]
return read_opossum_file(opossum_input_file)


if __name__ == "__main__":
opossum_file()
Empty file.
17 changes: 17 additions & 0 deletions src/opossum_lib/core/input_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0
from enum import Enum, auto

from pydantic import BaseModel


class FileType(Enum):
Hellgartner marked this conversation as resolved.
Show resolved Hide resolved
SPDX = (auto(),)
OPOSSUM = (auto(),)
SCAN_CODE = (auto(),)


class InputFile(BaseModel):
path: str
type: FileType
50 changes: 50 additions & 0 deletions src/opossum_lib/core/opossum_generation_arguments.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0

import logging
import sys
from pathlib import Path

from pydantic import BaseModel

from opossum_lib.core.input_file import FileType, InputFile


class OpossumGenerationArguments(BaseModel):
spdx_files: list[str]
scancode_json_files: list[str]
opossum_files: list[str]
outfile: str
Hellgartner marked this conversation as resolved.
Show resolved Hide resolved

def validate_input_and_exit_on_error(self) -> None:
total_number_of_files = (
len(self.spdx_files)
+ len(self.scancode_json_files)
+ len(self.opossum_files)
)
if total_number_of_files == 0:
logging.warning("No input provided. Exiting.")
sys.exit(1)
if total_number_of_files > 1:
logging.error("Merging of multiple files not yet supported!")
sys.exit(1)

def add_outfile_ending_and_warn_on_existing_outfile(self) -> None:
if not self.outfile.endswith(".opossum"):
self.outfile += ".opossum"

if Path.is_file(Path(self.outfile)):
logging.warning(f"{self.outfile} already exists and will be overwritten.")
Hellgartner marked this conversation as resolved.
Show resolved Hide resolved

def generate_input_file_list(self) -> list[InputFile]:
result = []
result += [
InputFile(path=path, type=FileType.SCAN_CODE)
for path in self.scancode_json_files
]
result += [
InputFile(path=path, type=FileType.OPOSSUM) for path in self.opossum_files
]
result += [InputFile(path=path, type=FileType.SPDX) for path in self.spdx_files]
return result
41 changes: 41 additions & 0 deletions src/opossum_lib/core/opossum_generator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0

from pathlib import Path

from opossum_lib.core.input_file import FileType, InputFile
from opossum_lib.core.opossum_generation_arguments import OpossumGenerationArguments
from opossum_lib.opossum.file_generation import OpossumFileWriter
from opossum_lib.opossum.opossum_file_content import OpossumFileContent
from opossum_lib.opossum.read_opossum_file import read_opossum_file
from opossum_lib.scancode.convert_scancode_to_opossum import (
convert_scancode_file_to_opossum,
)
from opossum_lib.spdx.convert_to_opossum import convert_spdx_to_opossum_information


class OpossumGenerator:
@staticmethod
def generate(opossum_generation_arguments: OpossumGenerationArguments) -> None:
opossum_generation_arguments.validate_input_and_exit_on_error()
input_files = opossum_generation_arguments.generate_input_file_list()
opossum_file_content = OpossumGenerator._convert_after_valid_input(input_files)
opossum_generation_arguments.add_outfile_ending_and_warn_on_existing_outfile()

OpossumFileWriter.write_opossum_information_to_file(
opossum_file_content, Path(opossum_generation_arguments.outfile)
)

@staticmethod
def _convert_after_valid_input(
input_file_list: list[InputFile],
) -> OpossumFileContent:
input_file = input_file_list[0]

if input_file.type == FileType.SPDX:
return convert_spdx_to_opossum_information(input_file.path)
elif input_file.type == FileType.SCAN_CODE:
return convert_scancode_file_to_opossum(input_file.path)
else:
return read_opossum_file(input_file.path)
File renamed without changes.
12 changes: 6 additions & 6 deletions src/opossum_lib/opossum/opossum_file_to_opossum_converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,7 @@
from pathlib import PurePath

import opossum_lib.opossum.opossum_file
from opossum_lib.opossum.opossum_file import (
ExternalAttributionSource as FileExternalAttributionSource,
)
from opossum_lib.opossum.opossum_file import OpossumInformation
from opossum_lib.opossum.opossum_file_content import OpossumFileContent
from opossum_lib.opossum_model import (
from opossum_lib.core.opossum_model import (
BaseUrlsForSources,
ExternalAttributionSource,
FrequentLicense,
Expand All @@ -25,6 +20,11 @@
SourceInfo,
_convert_path_to_str,
)
from opossum_lib.opossum.opossum_file import (
ExternalAttributionSource as FileExternalAttributionSource,
)
from opossum_lib.opossum.opossum_file import OpossumInformation
from opossum_lib.opossum.opossum_file_content import OpossumFileContent


class OpossumFileToOpossumConverter:
Expand Down
2 changes: 1 addition & 1 deletion src/opossum_lib/scancode/convert_scancode_to_opossum.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import uuid
from pathlib import PurePath

import opossum_lib.opossum_model as opossum_model
import opossum_lib.core.opossum_model as opossum_model
from opossum_lib.opossum.opossum_file_content import OpossumFileContent
from opossum_lib.scancode.constants import SCANCODE_SOURCE_NAME
from opossum_lib.scancode.model import File, FileType, Header, ScanCodeData
Expand Down
Empty file added tests/core/__init__.py
Empty file.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from faker.providers.misc import Provider as MiscProvider
from faker.providers.person import Provider as PersonProvider

from opossum_lib.opossum_model import ExternalAttributionSource
from opossum_lib.core.opossum_model import ExternalAttributionSource
from tests.util.generator_helpers import entry_or_none


Expand Down
2 changes: 1 addition & 1 deletion tests/opossum_model_generators/metadata_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from faker.providers.lorem.en_US import Provider as LoremProvider
from faker.providers.misc import Provider as MiscProvider

from opossum_lib.opossum_model import Metadata
from opossum_lib.core.opossum_model import Metadata
from tests.util.generator_helpers import entry_or_none


Expand Down
2 changes: 1 addition & 1 deletion tests/opossum_model_generators/opossum_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

from faker.providers import BaseProvider

from opossum_lib.core.opossum_model import Opossum, ScanResults
from opossum_lib.opossum.output_model import OpossumOutputFile
from opossum_lib.opossum_model import Opossum, ScanResults
from tests.opossum_model_generators.scan_results_provider import ScanResultsProvider
from tests.test_opossum.generators.generate_outfile_information import (
OpossumOutputFileProvider,
Expand Down
2 changes: 1 addition & 1 deletion tests/opossum_model_generators/package_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from faker.providers.misc.en_US import Provider as MiscProvider
from faker.providers.person.en_US import Provider as PersonProvider

from opossum_lib.opossum_model import OpossumPackage, SourceInfo
from opossum_lib.core.opossum_model import OpossumPackage, SourceInfo
from tests.opossum_model_generators.source_info_provider import SourceInfoProvider
from tests.util.generator_helpers import entry_or_none, random_list

Expand Down
2 changes: 1 addition & 1 deletion tests/opossum_model_generators/resource_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from faker.providers.file import Provider as FileProvider
from faker.providers.misc import Provider as MiscProvider

from opossum_lib.opossum_model import OpossumPackage, Resource, ResourceType
from opossum_lib.core.opossum_model import OpossumPackage, Resource, ResourceType
from tests.opossum_model_generators.package_provider import PackageProvider
from tests.util.generator_helpers import random_list

Expand Down
2 changes: 1 addition & 1 deletion tests/opossum_model_generators/scan_results_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from faker.providers.file.en_US import Provider as FileProvider
from faker.providers.misc.en_US import Provider as MiscProvider

from opossum_lib.opossum_model import (
from opossum_lib.core.opossum_model import (
BaseUrlsForSources,
ExternalAttributionSource,
FrequentLicense,
Expand Down
2 changes: 1 addition & 1 deletion tests/opossum_model_generators/source_info_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from faker.providers.misc import Provider as MiscProvider
from faker.providers.person.en_US import Provider as PersonProvider

from opossum_lib.opossum_model import SourceInfo
from opossum_lib.core.opossum_model import SourceInfo
from tests.util.generator_helpers import entry_or_none


Expand Down
2 changes: 1 addition & 1 deletion tests/test_scancode/test_convert_scancode_to_opossum.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import pytest
from _pytest.logging import LogCaptureFixture

from opossum_lib.opossum_model import Resource
from opossum_lib.core.opossum_model import Resource
from opossum_lib.scancode.convert_scancode_to_opossum import (
convert_scancode_to_opossum,
)
Expand Down
Loading