From 82b9d7a2be90cb9b656d5eef503759bdcdefd28b Mon Sep 17 00:00:00 2001 From: Konrad Ueltzen <73957305+konradgithuup@users.noreply.github.com> Date: Thu, 4 Apr 2024 21:25:41 +0200 Subject: [PATCH 1/2] Add specification --- doc/implementing-backend.md | 6 ++ doc/object-backend-specs.md | 194 ++++++++++++++++++++++++++++++++++++ 2 files changed, 200 insertions(+) create mode 100644 doc/object-backend-specs.md diff --git a/doc/implementing-backend.md b/doc/implementing-backend.md index 939b13e72..fb9db83c2 100644 --- a/doc/implementing-backend.md +++ b/doc/implementing-backend.md @@ -101,3 +101,9 @@ foreach backend: julea_backends ... ``` + +## General (Object) Backend Workflow + +This section describes an object backend's lifecycle to provide a better understanding regarding the backend's various functions and how they ultimately work together. This is only a surface level overview. For more information, see the [formal object backend specification](./object-backend-specs.md). + +The general workflow begins when JULEA initializes the backend by calling `backend_init`. From there, objects can be created and opened using `backend_create` and `backend_open`. For opening existing objects, the iterator functions `backend_get_all`, `backend_get_by_prefix`, and `backend_iterate` can be used to explore the file system. The `backend_open` and `backend_create` functions return a handle for the object. This handle can then be used with the `backend_read`, `backend_write`, `backend_sync`, and `backend_status` functions. Once work on an object has concluded it should be removed from the backend by passing the object handle to `backend_close`. If the object should also be removed from the file system use `backend_delete` instead. Both the close and delete functions consume the object handle, meaning it is no longer valid after passing it to either of these functions. Finally, JULEA will eventually call `backend_fini` to destroy the backend. diff --git a/doc/object-backend-specs.md b/doc/object-backend-specs.md new file mode 100644 index 000000000..5ff5621d2 --- /dev/null +++ b/doc/object-backend-specs.md @@ -0,0 +1,194 @@ + +# Object Backend Specification + +This specification aims to help with implementing new backends by defining what JULEA expects a backend to do when it calls one of its functions. + +> [!NOTE] +> Unfortunately, the current test suite do not cover every aspect of the specification nor does every violation of the specification immediately lead to unexpected behavior. Thus, for the time being, special attention and care on the side of the backend implementor is required. + +### Error Handling + +By default, the backend's functions are expected to return `TRUE`. +If, for whatever reason, the backend cannot perform a task following the specifications below, it must fail by returning `FALSE`. +Additional failure conditions are outlined in the sections below. + +### backend_init + +JULEA calls this function to allow the backend to initialize itself under the given path. The path consists of an arbitrary number of directories. It's the backend's responsibility to create any missing directories. + +**Parameters** +- `path`: The path under which to initialize the backend. +- `backend_data`: Output parameter for the backend. + +### backend_fini + +JULEA calls this function to allow the backend to perform clean-up before shutdown. + +**Parameters** +- `backend_data`: The backend JULEA is about to destroy. + +**Postcondtions** +- the backend does not have any I/O operations in flight +- all memory allocated by the backend is freed + +### backend_create + +This function creates a new object specified by its full path. + +Both the namespace and the path can contain an arbitrary number of parent directories. It's the responsibility of the backend to create any missing parent directories. + +**Parameters** +- `backend_data`: The backend in which to create the object. +- `namespace`: The namespace used to identify the created object. +- `path`: The path used to identify the created object. +- `backend_object`: Output parameter for the created object. + +**Failure Conditions** +- the full path resolves to an existing object + +### backend_open + +This function opens an existing object specified by its full path. + +Both the namespace and the path can contain an arbitrary number of parent directories. It's the responsibility of the backend to create any missing parent directories. + +**Parameters** +- `backend_data`: The backend in which to open the object. +- `namespace`: The namespace used to identify the opened object. +- `path`: The path used to identify the opened object. +- `backend_object`: Output parameter for the opened object. + +**Failure Conditions** +- the full path does not resolve to an existing object + +**Output** +- a reference to an object handle must be stored in `backend_object`. + +### backend_delete + +This function deletes an existing object, specified by its handle. + +**Parameters** +- `backend_data`: The backend containing the object. +- `backend_object`: The object that is to be deleted. + +**Postconditions** +- the object handle no longer resolves to a valid object +- the object cannot be reloaded from persistent storage + +**Failure Conditions** +- the object handle does not resolve to an object + +### backend_close + +This function closes an existing object, specified by its handle. + +**Parameters** +- `backend_data`: The backend containing the object. +- `backend_object`: The object that is to be closed. + +**Postconditions** +- the object handle no longer resolves to a valid object + +**Failure Conditions** +- the object handle does not resolve to an object +- the object was already closed + +### backend_read + +This function reads a specified number of bytes at a specified offset from an object specified by its handle into a buffer. Short reads are allowed. + +**Parameters** +- `backend_data`: The backend containing the object. +- `backend_object`: The object to read from. +- `buffer`: Output parameter to write the content of the object into. +- `length`: The number of bytes to read. +- `offset`: The offset at which to read from the object. +- `bytes_read`: Output parameter for the number of bytes that have been read from the object. + +**Preconditions** +- the buffer's size is at least `length` + +**Failure Conditions** +- the object handle does not resolve to an object. +- read stops early due to an error (to distinguish from a valid short read) + +### backend_write + +This function writes a specified number of bytes from a buffer into an object specified by its handle at the specified offset. The object must be appended, if necessary. + +- `backend_data`: The backend containing the object. +- `backend_object`: The object to write to. +- `buffer`: The buffer to write to the object. +- `length`: The number of bytes to write. +- `offset`: The offset at which to write to the object. +- `bytes_written`: Output parameter for the number of bytes that have been written to the object. + +**Preconditions** +- the buffer's size is at least `length` + +**Failure Conditions** +- the object handle does not resolve to an object +- unable to write entire buffer to object (for example because the system ran out of storage space) + +### backend_status + +This function fetches metadata from an object specified by its handle. + +**Parameters** +- `backend_data`: The backend containing the object. +- `backend_object`: The object to acquire metadata from. +- `modification_time`: Output parameter for the time of the object's most recent modification. +- `size`: Output parameter for the object's size in bytes. + +**Failure Conditions** +- the object handle does not resolve to an object + +### backend_sync + +This function flushes an object specified by its handle. + +**Parameters** +- `backend_data`: The backend containing the object. +- `backend_object`: The object to synchronize. + +**Post Conditions** +- no I/O operations on the specified object are in flight +- the object's state matches the state in persistent storage + +**Failure Conditions** +- the object handle does not resolve to an object + +### backend_get_all + +Creates a new iterator that iterates over all objects under a specified path. The path is defined as `${backend_data.path}/${namespace}`. + +**Parameters** +- `backend_data`: The backend containing on which to iterate. +- `backend_iterator`: Output parameter for the newly created iterator. + +**Failure Conditions** +- the specified path does not resolve to an existing directory + +### backend_get_by_prefix + +Creates a new iterator that iterates over all objects under a specified path matching a specified prefix. The path is defined as `${backend_data.path}/${namespace}`. + +**Parameters** +- `backend_data`: The backend containing a directory on which to iterate. +- `backend_iterator`: Output parameter for the newly created iterator. + +**Failure Conditions** +- the specified path does not resolve to an existing directory + +### backend_iterate + +This function advances the iterator to the next object. If the end of the iterator is reached the iterator must be freed. + +**Parameters** +- `backend_data`: The backend containing a directory on which to iterate. +- `backend_iterator`: The backend iterator to advance to the next entry. +- `name`: Output parameter for the file the iterator advanced to. + +**Failure Conditions** +- reached the end of the iterator From cade4c50631c567573c3cebff3e04319ded66714 Mon Sep 17 00:00:00 2001 From: Konrad Ueltzen Date: Tue, 4 Jun 2024 11:48:40 +0200 Subject: [PATCH 2/2] Update formal specs --- doc/object-backend-specs.md | 58 +++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 31 deletions(-) diff --git a/doc/object-backend-specs.md b/doc/object-backend-specs.md index 5ff5621d2..572983300 100644 --- a/doc/object-backend-specs.md +++ b/doc/object-backend-specs.md @@ -14,7 +14,8 @@ Additional failure conditions are outlined in the sections below. ### backend_init -JULEA calls this function to allow the backend to initialize itself under the given path. The path consists of an arbitrary number of directories. It's the backend's responsibility to create any missing directories. +JULEA calls this function to allow the backend to initialize itself under the given path. The path consists of an arbitrary number of directories. +It’s the backend’s responsibility to create any missing directories. **Parameters** - `path`: The path under which to initialize the backend. @@ -33,9 +34,7 @@ JULEA calls this function to allow the backend to perform clean-up before shutdo ### backend_create -This function creates a new object specified by its full path. - -Both the namespace and the path can contain an arbitrary number of parent directories. It's the responsibility of the backend to create any missing parent directories. +This function creates a new object specified by its full path. Both the namespace and the path can contain an arbitrary number of parent directories. It’s the responsibility of the backend to create any missing parent directories. **Parameters** - `backend_data`: The backend in which to create the object. @@ -43,14 +42,12 @@ Both the namespace and the path can contain an arbitrary number of parent direct - `path`: The path used to identify the created object. - `backend_object`: Output parameter for the created object. -**Failure Conditions** +**Additional Failure Conditions** - the full path resolves to an existing object ### backend_open -This function opens an existing object specified by its full path. - -Both the namespace and the path can contain an arbitrary number of parent directories. It's the responsibility of the backend to create any missing parent directories. +This function opens an existing object specified by its full path. Both the namespace and the path can contain an arbitrary number of parent directories. It’s the responsibility of the backend to create any missing parent directories. **Parameters** - `backend_data`: The backend in which to open the object. @@ -58,11 +55,11 @@ Both the namespace and the path can contain an arbitrary number of parent direct - `path`: The path used to identify the opened object. - `backend_object`: Output parameter for the opened object. -**Failure Conditions** +**Additional Failure Conditions** - the full path does not resolve to an existing object **Output** -- a reference to an object handle must be stored in `backend_object`. +- a reference to an object handle must be stored in backend_object ### backend_delete @@ -76,21 +73,20 @@ This function deletes an existing object, specified by its handle. - the object handle no longer resolves to a valid object - the object cannot be reloaded from persistent storage -**Failure Conditions** +**Additional Failure Conditions** - the object handle does not resolve to an object ### backend_close This function closes an existing object, specified by its handle. -**Parameters** - `backend_data`: The backend containing the object. - `backend_object`: The object that is to be closed. **Postconditions** - the object handle no longer resolves to a valid object -**Failure Conditions** +**Additional Failure Conditions** - the object handle does not resolve to an object - the object was already closed @@ -107,15 +103,15 @@ This function reads a specified number of bytes at a specified offset from an ob - `bytes_read`: Output parameter for the number of bytes that have been read from the object. **Preconditions** -- the buffer's size is at least `length` +- the buffer’s size is at least length -**Failure Conditions** +**Additional Failure Conditions** - the object handle does not resolve to an object. - read stops early due to an error (to distinguish from a valid short read) ### backend_write -This function writes a specified number of bytes from a buffer into an object specified by its handle at the specified offset. The object must be appended, if necessary. +This function writes a specified number of bytes from a buffer into an object specified by its handle at the specified offset. The backend must try to write the entire buffer to the best of its abilities. - `backend_data`: The backend containing the object. - `backend_object`: The object to write to. @@ -125,9 +121,9 @@ This function writes a specified number of bytes from a buffer into an object sp - `bytes_written`: Output parameter for the number of bytes that have been written to the object. **Preconditions** -- the buffer's size is at least `length` +- the buffer’s size is at least `length` -**Failure Conditions** +**Additional Failure Conditions** - the object handle does not resolve to an object - unable to write entire buffer to object (for example because the system ran out of storage space) @@ -138,10 +134,10 @@ This function fetches metadata from an object specified by its handle. **Parameters** - `backend_data`: The backend containing the object. - `backend_object`: The object to acquire metadata from. -- `modification_time`: Output parameter for the time of the object's most recent modification. -- `size`: Output parameter for the object's size in bytes. +- `modification_time`: Output parameter for the time of the object’s most recent modification. +- `size`: Output parameter for the object’s size in bytes. -**Failure Conditions** +**Additional Failure Conditions** - the object handle does not resolve to an object ### backend_sync @@ -149,36 +145,36 @@ This function fetches metadata from an object specified by its handle. This function flushes an object specified by its handle. **Parameters** -- `backend_data`: The backend containing the object. -- `backend_object`: The object to synchronize. +- backend_data: The backend containing the object. +- backend_object: The object to synchronize. **Post Conditions** -- no I/O operations on the specified object are in flight -- the object's state matches the state in persistent storage +- the backend does not have any I/O operations on the specified object in flight +- the object’s state matches the state in persistent storage -**Failure Conditions** +**Additional Failure Conditions** - the object handle does not resolve to an object ### backend_get_all -Creates a new iterator that iterates over all objects under a specified path. The path is defined as `${backend_data.path}/${namespace}`. +Creates a new iterator that iterates over all objects under a specified path. **Parameters** - `backend_data`: The backend containing on which to iterate. - `backend_iterator`: Output parameter for the newly created iterator. -**Failure Conditions** +**Additional Failure Conditions** - the specified path does not resolve to an existing directory ### backend_get_by_prefix -Creates a new iterator that iterates over all objects under a specified path matching a specified prefix. The path is defined as `${backend_data.path}/${namespace}`. +Creates a new iterator that iterates over all objects under a specified path matching a specified prefix. **Parameters** - `backend_data`: The backend containing a directory on which to iterate. - `backend_iterator`: Output parameter for the newly created iterator. -**Failure Conditions** +**Additional Failure Conditions** - the specified path does not resolve to an existing directory ### backend_iterate @@ -190,5 +186,5 @@ This function advances the iterator to the next object. If the end of the iterat - `backend_iterator`: The backend iterator to advance to the next entry. - `name`: Output parameter for the file the iterator advanced to. -**Failure Conditions** +**Additional Failure Conditions** - reached the end of the iterator