From f048611ee0d2cd12817034341e7ac45861c77542 Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Thu, 7 Apr 2022 12:52:13 +0200 Subject: [PATCH 1/3] doc: Remove obsolete remark about absolute assets paths Asset paths are no longer changed to be absolute paths. Besides, this remark about an internal implementation detail is likely not very useful to mention in the introduction of assets. --- docs/UsersGuide.asciidoc | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/UsersGuide.asciidoc b/docs/UsersGuide.asciidoc index d71075104a4..59cf4b16113 100644 --- a/docs/UsersGuide.asciidoc +++ b/docs/UsersGuide.asciidoc @@ -1082,11 +1082,10 @@ openqa-load-templates test.json == Asset handling == -Multiple parameters exist to reference "assets" to be used by tests. "Assets" are essentially -content that is stored by the openQA web-UI and provided to the workers; when sending jobs to -os-autoinst on the workers, openQA adjusts the parameter values to refer to an absolute path -where the worker will be able to access the content. Things that are typically assets include the -ISOs and other images that are tested, for example. +Multiple parameters exist to reference "assets" to be used by tests. "Assets" +are essentially content that is stored by the openQA web-UI and provided to the +workers. Things that are typically assets include the ISOs and other images that +are tested, for example. Some assets can also be produced by a job, sent back to the web-UI, and used by a later job (see explanation of 'storing' and 'publishing' assets, below). Assets can also be seen in the web-UI From 10b06263cc4016d1da4db85ef5cade42756ab0ea Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Thu, 7 Apr 2022 13:07:10 +0200 Subject: [PATCH 2/3] doc: Split asset handling section into more subsections For better readability, it is generally a good idea to keep sections short. --- docs/UsersGuide.asciidoc | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/UsersGuide.asciidoc b/docs/UsersGuide.asciidoc index 59cf4b16113..1ee5ef666cb 100644 --- a/docs/UsersGuide.asciidoc +++ b/docs/UsersGuide.asciidoc @@ -1092,8 +1092,13 @@ explanation of 'storing' and 'publishing' assets, below). Assets can also be see and downloaded directly (though there is a configuration option to hide some or all asset types from public view in the web-UI). -The parameters treated as assets are as follows. Where you see e.g. `ISO_n`, that means `ISO_1`, -`ISO_2` etc. will all be treated as assets. +Assets may be shared between the web-UI and the workers by having them literally use a shared +filesystem (this used to be the only option), or by having the workers download them from the +server when needed and cache them locally. Checkout the documentation about +<> for more on this. + +=== Specifying assets required by a job === +The following job settings are specifying that an asset is required by a job: * `ISO` (type `iso`) * `ISO_n` (type `iso`) @@ -1104,6 +1109,8 @@ The parameters treated as assets are as follows. Where you see e.g. `ISO_n`, tha * `KERNEL` (type `other`) * `INITRD` (type `other`) +Where you see e.g. `ISO_n`, that means `ISO_1`, `ISO_2` etc. will all be treated as assets. + The values of the above parameters are expected to be the name of a file - or, in the case of `REPO_n`, a directory - that exists under the path `/var/lib/openqa/share/factory` on the openQA web-UI. That path has subdirectories for each of the asset types, and the file or directory must @@ -1141,11 +1148,7 @@ uncompress it to `foo2.iso`, store it in `/var/lib/openqa/share/factory/iso` and `ISO_1=foo2.iso`. Again, you can also set `ISO_1` to change the name the file will be downloaded and uncompressed as. -Assets may be shared between the web-UI and the workers by having them literally use a shared -filesystem (this used to be the only option), or by having the workers download them from the -server when needed and cache them locally. Checkout the documentation about -<> for more on this. - +=== Specifying assets created by a job === `HDD_n` assets can be 'stored' or 'published' by a job, and `UEFI_PFLASH_VARS` assets can be 'published'. These both mean that if the job completes successfully, the resulting state of those disk assets will be sent back to the web-UI and made available as an `hdd`-type asset. To 'store' From a6b4ca2acfe3741867afcd0f1c6386fc1f050eec Mon Sep 17 00:00:00 2001 From: Marius Kittler Date: Thu, 7 Apr 2022 13:46:24 +0200 Subject: [PATCH 3/3] doc: Mention the problem of conflicting asset names more explicitly * State the problem more explicitly, also considering concurrent uploads * Move private assets (which are a solution to avoid the problem for jobs with chained dependencies) into their own section * Avoid mentioning private assets directly at the beginning to simplify the wording * Avoid confusion with reference to variable expansion (it does *not* relate to private assets specifically) * See https://progress.opensuse.org/issues/109319 --- docs/UsersGuide.asciidoc | 58 +++++++++++++++++++++++++--------------- 1 file changed, 37 insertions(+), 21 deletions(-) diff --git a/docs/UsersGuide.asciidoc b/docs/UsersGuide.asciidoc index 1ee5ef666cb..76470ea5770 100644 --- a/docs/UsersGuide.asciidoc +++ b/docs/UsersGuide.asciidoc @@ -1149,29 +1149,45 @@ uncompress it to `foo2.iso`, store it in `/var/lib/openqa/share/factory/iso` and and uncompressed as. === Specifying assets created by a job === -`HDD_n` assets can be 'stored' or 'published' by a job, and `UEFI_PFLASH_VARS` assets can be -'published'. These both mean that if the job completes successfully, the resulting state of those -disk assets will be sent back to the web-UI and made available as an `hdd`-type asset. To 'store' -an asset, you can specify e.g. `STORE_HDD_1`. To 'publish' it, you can specify e.g. -`PUBLISH_HDD_1` or `PUBLISH_PFLASH_VARS`. If you specify `PUBLISH_HDD_1=updated.qcow2`, the -`HDD_1` disk image as it exists at the end of the test will be uploaded back to the web-UI and -stored under the name `updated.qcow2`; any other job can then specify `HDD_1=updated.qcow2` to use -this published image as its `HDD_1`. To force publishing assets even in case -of a failed job one can try the `FORCE_PUBLISH_HDD_` variable. - -The difference between 'storing' and 'publishing' is that when 'storing' an asset, it will be -altered in some way (currently, by prepending the job ID to the filename) to associate it with -the particular job that produced it. That means that many jobs can 'store' an asset under "the -same name" without conflicting. Of course, that would seem to make it hard for other jobs to use -the 'stored' image - but for "chained" jobs, the reverse operation is done transparently. This -all means that a 'parent' job template can specify `STORE_HDD_1=somename.qcow2` and its 'child' -job template(s) can specify `HDD_1=somename.qcow2`, and everything will work, without multiple -runs of the same jobs overwriting the asset. For more on "chained" jobs, see the documentation -of <>. - -When using this mechanism you will often also want to use the +Jobs can upload assets to the web-UI so other jobs can used them as `HDD_n` and +`UEFI_PFLASH_VARS` assets as described in the previous section. + +To declare an asset to be uploaded, you can use the job settings `PUBLISH_HDD_n` +and `PUBLISH_PFLASH_VARS`. For instance, if you specify +`PUBLISH_HDD_1=updated.qcow2`, the `HDD_1` disk image as it exists at the end of +the test will be uploaded back to the web-UI and stored under the name +`updated.qcow2`. Any other job can then specify `HDD_1=updated.qcow2` to use +this published image as its `HDD_1`. + +IMPORTANT: Assets that are already existing will be overridden. If the same +asset is uploaded by multiple jobs concurrently this will lead to file +corruption. So be sure to use unique names or use private assets as explained +in the subsection below. + +NOTE: Note that assets are by default only uploaded if the job completes +successfully. To force publishing assets even in case of a failed job one can +specify the `FORCE_PUBLISH_HDD_` variable. + +NOTE: When using this mechanism you will often also want to use the <> mechanism. +==== Private assets ==== +There is a mechanism to alter an asset's file name automatically to associate +it with the particular job that produced it (currently, by prepending the job ID +to the filename). To make use of it, use `STORE_HDD_n` (instead of +`PUBLISH_HDD_n`). Those assets can then be consumed by chained jobs. For +instance, if a parent job uploads an asset via `STORE_HDD_1=somename.qcow2`, its +children can use it via `HDD_1=somename.qcow2` without having to worry about +naming conflicts. + +IMPORTANT: This only works if the jobs uploading and consuming jobs have a +chained dependency. For more on "chained" jobs, see the documentation of +<>. + +NOTE: Access to private assets is not protected. Theoretically, jobs outside the +chain can still access the asset by explicitly prepending the ID of the creating +job. + [id="asset_cleanup"] === Asset cleanup === For more information on assets, start reading from the beginning of the