diff --git a/docs/source/api.rst b/docs/source/api.rst index 7f34c5b0b..d83c09f8e 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -144,9 +144,9 @@ API endpoints reference There are two primary endpoints: -- packages/: this is the main endpoint where you can lookup vulnerabilities by package. +- packages/: this is the main endpoint where you can look up vulnerabilities by package. -- vulnerabilities/: to lookup by vulnerabilities +- vulnerabilities/: to look up by vulnerabilities And two secondary endpoints, used to query vulnerability aliases (such as CVEs) and vulnerability by CPEs: cpes/ and aliases/ diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst index ba634ecc8..08025157a 100644 --- a/docs/source/contributing.rst +++ b/docs/source/contributing.rst @@ -17,7 +17,7 @@ resources to help you get started. Do Your Homework ---------------- -Before adding a contribution or create a new issue, take a look at the project’s +Before adding a contribution or creating a new issue, take a look at the project’s `README `_, read through our `documentation `_, and browse existing `issues `_, @@ -52,7 +52,7 @@ For more established contributors, you can contribute to the codebase in several - Submit a `bug fix `_ for any existing issue. - Create a `new issue `_ to request a - feature, submit a feedback, or ask a question. + feature, submit feedback, or ask a question. * Want to add support for a new importer pipeline? See the detailed tutorial here: :ref:`tutorial_add_importer_pipeline`. diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 655504d3d..1d3f4169c 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -8,7 +8,7 @@ Installation installations are likely to not produce enough results. This is tracked in https://github.com/aboutcode-org/vulnerablecode/issues/597 -Welcome to **VulnerableCode** installation guide! This guide describes how to install +Welcome to the **VulnerableCode** installation guide! This guide describes how to install VulnerableCode on various platforms. Please read and follow the instructions carefully to ensure your installation is functional and smooth. @@ -63,7 +63,7 @@ Run the App At this point, the VulnerableCode app should be running at port ``8000`` on your Docker host. Go to http://localhost:8000/ on a web browser to access the web UI. -Optionally, you can set ``NGINX_PORT`` environment variable in your shell or in the `.env` file +Optionally, you can set the ``NGINX_PORT`` environment variable in your shell or in the `.env` file to run on a different port than 8000. .. note:: @@ -80,14 +80,14 @@ to run on a different port than 8000. .. warning:: - Serving VulnerableCode on a network could lead to security issues and there - are several steps that may be needed to secure such a deployment. - Currently, this is not recommendend. + Serving VulnerableCode on a network could lead to security issues and + several steps may be needed to secure such a deployment. + Currently, this is not recommended. Execute a Command ^^^^^^^^^^^^^^^^^ -You can execute a one of ``manage.py`` commands through the Docker command line +You can execute one of the ``manage.py`` commands through the Docker command line interface, for example:: docker compose run vulnerablecode ./manage.py import --list @@ -180,7 +180,7 @@ production servers. make postgres .. note:: - You can also use a **SQLite** database for local development as a single user + You can also use an **SQLite** database for local development as a single user with:: make sqlite @@ -244,7 +244,7 @@ There are several options to use the Nix version:: > ../../manage.py ... # invoke the local checkout > vulnerablecode-manage.py ... # invoke manage.py as installed in the nix store - # Test the import prodecure using the Nix version. + # Test the import procedure using the Nix version. etc/nix/test-import-using-nix.sh --all # import everything # Test the import using the local checkout. INSTALL_DIR=. etc/nix/test-import-using-nix.sh ruby # import ruby only diff --git a/docs/source/introduction.rst b/docs/source/introduction.rst index 5143343a6..01003b74c 100644 --- a/docs/source/introduction.rst +++ b/docs/source/introduction.rst @@ -24,12 +24,12 @@ fixes those vulnerabilities.** - With the web UI, you can search by package using Package URLs or search by vulnerability, e.g., by CVE. From there you can navigate to the package - vulnerabilities and to the vulnerable packages. + vulnerabilities and the vulnerable packages. - With the JSON API, you can perform package queries using Package URLs (`purl `__) or query by vulnerability id ("VCID"). You can also query by CPEs and other vulnerability aliases. - The API provides paginated index and detail endpoints and includes indexes + The API provides a paginated index and detailed endpoints and includes indexes of vulnerable CPEs and vulnerable Package URLs. You can install VulnerableCode locally or use the provided publicly hosted instance, @@ -79,7 +79,7 @@ How does it work? VulnerableCode independently aggregates many software vulnerability data sources and supports data re-creation in a decentralized fashion. These data sources -(see complete list +(see the complete list `here `__) include security advisories published by Linux and BSD distributions, application software package managers and package repositories, FOSS projects, @@ -96,9 +96,9 @@ to vulnerability *bar*?" more accurate and easier to interpret. The primary access to VulnerableCode data is through a REST API, but there is also a Web UI for searching and browsing vulnerabilities by package -or by vulnerability. For the initial releases both access modes are +or by the vulnerability. For the initial releases both access modes are read-only, but our longer-term goal is to enable community curation of -the data including addition of new packages and vulnerabilities, and +the data including the addition of new packages and vulnerabilities, and reviewing and updating their relationships. We also plan to mine for vulnerabilities that have not received any diff --git a/docs/source/misc.rst b/docs/source/misc.rst index c2aab16a9..1cb1e4825 100644 --- a/docs/source/misc.rst +++ b/docs/source/misc.rst @@ -7,7 +7,7 @@ Continuous periodic Data import ------------------------------- -If you want to run the import periodically, you can use a systemd timer. +If you want to run the import periodically, you can use a system timer. Here is an example:: $ cat ~/.config/systemd/user/vulnerablecode.service diff --git a/docs/source/reference_improver_overview.rst b/docs/source/reference_improver_overview.rst index 4de0ff1c0..847b23e89 100644 --- a/docs/source/reference_improver_overview.rst +++ b/docs/source/reference_improver_overview.rst @@ -27,8 +27,8 @@ An ``Inference`` is more explicit than an ``Advisory`` and is able to answer que package A vulnerable to Vulnerability B ?". Of course, there is some confidence attached to the answer, which could also be ``MAX_CONFIDENCE`` in certain cases. -The possibilities with improvers are endless; they are not restricted to take one approach. Features -like *Time Travel* and *finding fix commits* could be implemented as well. +The possibilities with improvers are endless; they are not restricted to taking one approach. +Features like *Time Travel* and *finding fix commits* could be implemented as well. You can find more in-code documentation about improvers in :file:`vulnerabilites/improver.py` and the framework responsible for invoking these improvers in :file:`vulnerabilites/improve_runner.py`. diff --git a/docs/source/tutorial_add_importer_pipeline.rst b/docs/source/tutorial_add_importer_pipeline.rst index 3f5b6b785..6854d4341 100644 --- a/docs/source/tutorial_add_importer_pipeline.rst +++ b/docs/source/tutorial_add_importer_pipeline.rst @@ -9,34 +9,34 @@ TL;DR #. Create a new file ``{name}_importer.py`` inside **vulnerabilities/pipelines/**. #. Create a new importer pipeline by inheriting **VulnerableCodeBaseImporterPipeline** - defined in **vulnerabilities.pipelines**. By convention the importer pipeline + defined in **vulnerabilities.pipelines**. By convention, the importer pipeline class should end with **ImporterPipeline**. -#. Specify the license of upstream data being imported. +#. Specify the license of the upstream data being imported. #. Implement the ``advisories_count`` and ``collect_advisories`` methods. #. Add the newly created importer pipeline to the importers registry at - **vulnerabilities/importers/__init__.py** + **vulnerabilities/importers/__init__.py**. Pipeline -------- We use `aboutcode.pipeline `_ -for importing and improving data. At a very high level, a working pipeline contains classmethod +for importing and improving data. At a very high level, a working pipeline contains a classmethod ``steps`` that defines what steps to run and in what order. These steps are essentially just -functions. Pipeline provides an easy and effective way to log events inside these steps (it -automatically handles rendering and dissemination for these logs.) +functions. The pipeline provides an easy and effective way to log events inside these steps (it +automatically handles rendering and dissemination for these logs). -It also includes built-in progress indicator, which is essential since some of the jobs we run -in the pipeline are long-running tasks that require proper progress indicators. Pipeline provides -way to seamlessly records the progress (it automatically takes care of rendering and dissemination -of these progress). +It also includes a built-in progress indicator, which is essential since some of the jobs we run +in the pipeline are long-running tasks that require proper progress indicators. +The pipeline provides a way to seamlessly record the progress (it automatically takes care of +rendering and dissemination of this progress). Additionally, the pipeline offers a consistent structure, making it easy to run these pipeline steps -with message queue like RQ and store all events related to a particular pipeline for -debugging/improvements. +with a message queue like RQ and store all events related to a particular pipeline for +debugging or improvements. -This tutorial contains all the things one should know to quickly implement an importer pipeline. -Many internal details about importer pipeline can be found inside the +This tutorial contains all the information one should know to quickly implement an importer pipeline. +Many internal details about the importer pipeline can be found inside the `vulnerabilities/pipelines/__init__.py `_ file. @@ -46,7 +46,7 @@ Many internal details about importer pipeline can be found inside the Prerequisites -------------- -Before writing pipeline to import advisories, it is important to familiarize yourself with +Before writing a pipeline to import advisories, it is important to familiarize yourself with the following concepts. PackageURL @@ -69,8 +69,8 @@ AdvisoryData ``AdvisoryData`` is an intermediate data format: it is expected that your importer will convert the raw scraped data into ``AdvisoryData`` objects. -All the fields in ``AdvisoryData`` dataclass are optional; it is the importer's responsibility to -ensure that it contains meaningful information about a vulnerability. +All the fields in the ``AdvisoryData`` dataclass are optional; it is the importer's responsibility +to ensure that it contains meaningful information about a vulnerability. AffectedPackage ~~~~~~~~~~~~~~~ @@ -84,7 +84,7 @@ Univers `univers `_ is a Python implementation of the `vers specification `_. It can parse and compare all the package versions and all the ranges, -from debian, npm, pypi, ruby and more. +from Debian, npm, PyPI, Ruby, and more. It processes all the version range specs and expressions. @@ -92,8 +92,8 @@ Writing an Importer Pipeline ----------------------------- -Create file for the new importer pipeline -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Create a file for the new importer pipeline +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All pipelines, including the importer pipeline, are located in the `vulnerabilities/pipelines/ @@ -110,7 +110,7 @@ parsing it to create proper **AdvisoryData** objects. Specify the importer license ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The pipeline scrape data off the internet. In order to make sure the data is useable, a license +The pipeline scrapes data off the internet. In order to make sure the data is usable, a license must be provided. Populate the ``spdx_license_expression`` with the appropriate value. The SPDX license identifiers @@ -145,7 +145,7 @@ Implement the ``collect_advisories`` method ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``collect_advisories`` method collects and parses the advisories from the data source and -yield an *AdvisoryData*. +yields an *AdvisoryData* object. At this point, an example importer will look like this: diff --git a/docs/source/tutorial_add_improver_pipeline.rst b/docs/source/tutorial_add_improver_pipeline.rst index feed3556b..7b40a5311 100644 --- a/docs/source/tutorial_add_improver_pipeline.rst +++ b/docs/source/tutorial_add_improver_pipeline.rst @@ -9,8 +9,8 @@ TL;DR #. Create a new file ``{improver_name}.py`` inside **vulnerabilities/pipelines/**. #. Create a new improver pipeline by inheriting **VulnerableCodePipeline** defined in **vulnerabilities.pipelines**. -#. Implement ``steps`` **classmethod** to define what function to run and in which order. -#. Implement the individual function defined in ``steps`` +#. Implement the ``steps`` **classmethod** to define what functions to run and in which order. +#. Implement the individual function defined in ``steps``. #. Add the newly created pipeline to the improvers registry at **vulnerabilities/improvers/__init__.py**. @@ -18,21 +18,22 @@ Pipeline -------- We use `aboutcode.pipeline `_ -for importing and improving data. At a very high level, a working pipeline contains classmethod +for importing and improving data. At a very high level, a working pipeline contains a classmethod ``steps`` that defines what steps to run and in what order. These steps are essentially just -functions. Pipeline provides an easy and effective way to log events inside these steps (it -automatically handles rendering and dissemination for these logs.) +functions. The pipeline provides an easy and effective way to log events inside these steps (it +automatically handles rendering and dissemination for these logs). -It also includes built-in progress indicator, which is essential since some of the jobs we run -in the pipeline are long-running tasks that require proper progress indicators. Pipeline provides -way to seamlessly records the progress (it automatically takes care of rendering and dissemination -of these progress). +It also includes a built-in progress indicator, which is essential since some of the jobs we run +in the pipeline are long-running tasks that require proper progress indicators. The pipeline +provides a way to seamlessly record the progress (it automatically takes care of rendering +and dissemination of this progress). Additionally, the pipeline offers a consistent structure, making it easy to run these pipeline steps -with message queue like RQ and store all events related to a particular pipeline for -debugging/improvements. +with a message queue like RQ and store all events related to a particular pipeline for +debugging or improvements. -This tutorial contains all the things one should know to quickly implement an improver pipeline. +This tutorial contains all the information one should know to quickly implement an improver +pipeline. Prerequisites @@ -57,24 +58,24 @@ Some of those are: So you see, the new improver pipeline is very powerful in what you can achieve, but as always, with -great power comes great responsibility. By design, the new improver are unconstrained, and you must +great power comes great responsibility. By design, the new improver is unconstrained, and you must be absolutely sure of what you're doing and should have robust tests for these pipelines in place. Writing an Improver Pipeline ----------------------------- -**Scenario:** Suppose we come around a source that curates and stores the list of packages that +**Scenario:** Suppose we come across a source that curates and stores the list of packages that don't exist upstream and makes it available through the REST API endpoint https://example.org/api/non-existent-packages, which gives a JSON response with a list of non-existent packages. -Let's write a pipeline that will use this source to flag these non-existent package as -ghost package. +Let's write a pipeline that will use this source to flag these non-existent packages as +ghost packages. -Create file for the new improver pipeline -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Create a file for the new improver pipeline +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All pipelines, including the improver pipeline, are located in the `vulnerabilities/pipelines/ @@ -99,7 +100,8 @@ can be found at `ScanCode LicenseDB ` Add skeleton for new pipeline ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In this scenario pipeline needs to do two thing fetch raw data and use that to flag those packages. +In this scenario pipeline needs to do two things, fetch raw data and use that to flag those +packages. At this point improver will look like this: