diff --git a/disk-buffering/DESIGN.md b/disk-buffering/DESIGN.md index 01f6048da..3bd1e3f01 100644 --- a/disk-buffering/DESIGN.md +++ b/disk-buffering/DESIGN.md @@ -1,59 +1,62 @@ # Design Overview -There are three main disk-writing exporters provided by this module: +The core of disk buffering +is [SignalStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/SignalStorage.java). +SignalStorage is an abstraction that defines the bare minimum functionalities needed for +implementations to allow writing and reading signals. -* [LogRecordToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporter.java) -* [MetricToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporter.java) -* [SpanToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporter.java)) +There is a default implementation per signal that writes serialized signal items to protobuf +delimited messages into files, where each file's name represents a timestamp of when it was created, +which will help later to know when it's ready to read, as well as when it's expired. These +implementations are the following: -Each is responsible for writing a specific type of telemetry to disk storage for later -harvest/ingest. +* [FileSpanStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileSpanStorage.java) +* [FileLogRecordStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileLogRecordStorage.java) +* [FileMetricStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileMetricStorage.java) -For later reading, there are: +Each one has a `create()` method that takes a destination directory (to store data into) and an +optional [FileStorageConfiguration](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileStorageConfiguration.java) +to have a finer control of the storing behavior. -* [LogRecordFromToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordFromDiskExporter.java) -* [MetricFromDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/MetricFromDiskExporter.java) -* [SpanFromDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/SpanFromDiskExporter.java)) +Even +though [SignalStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/SignalStorage.java) +can receive signal items directly to be stored in disk, there are convenience exporter +implementations for each signal that handle the storing process on your behalf. Those are the +following: -Each one of those has a `create()` method that takes a delegate exporter (to send data -to ingest) and the `StorageConfiguration` that tells them where to find buffered data. +* [SpanToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SpanToDiskExporter.java) +* [LogRecordToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/LogRecordToDiskExporter.java) +* [MetricToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/MetricToDiskExporter.java) -As explained in the [README](README.md), this has to be triggered manually by the consumer of -this library and does not happen automatically. +Each receive their +respective [SignalStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/SignalStorage.java) +object to delegate signals to as well as an optional callback object to notify its operations. ## Writing overview ![Writing flow](assets/writing-flow.png) -* The writing process happens automatically within its `export(Collection signals)` - method, which is called by the configured signal processor. -* When a set of signals is received, these are delegated over to - a type-specific wrapper of [ToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporter.java) - class which then serializes them using an implementation - of [SignalSerializer](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/serializers/SignalSerializer.java) - and then the serialized data is appended into a File using an instance of - the [Storage](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/Storage.java) - class. +* Via the convenience toDisk exporters, the writing process happens automatically within their + `export(Collection signals)` method, which is called by the configured signal + processor. +* When a set of signals is received, these are delegated over to a type-specific serializer + and then the serialized data is appended into a file. * The data is written into a file directly, without the use of a buffer, to make sure no data gets lost in case the application ends unexpectedly. -* Each disk exporter stores its signals in its own folder, which is expected to contain files +* Each signal storage stores its signals in its own folder, which is expected to contain files that belong to that type of signal only. * Each file may contain more than a batch of signals if the configuration parameters allow enough limit size for it. * If the configured folder size for the signals has been reached and a new file is needed to be created to keep storing new data, the oldest available file will be removed to make space for the new one. -* The [Storage](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/Storage.java), - [FolderManager](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManager.java) - and [WritableFile](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFile.java) - files contain more information on the details of the writing process into a file. ## Reading overview ![Reading flow](assets/reading-flow.png) -* The reading process has to be triggered manually by the library consumer as explained in - the [README](README.md). +* The reading process has to be triggered manually by the library consumer via the signal storage + iterator. * A single file is read at a time and updated to remove the data gathered from it after it is successfully exported, until it's emptied. Each file previously created during the writing process has a timestamp in milliseconds, which is used to determine what file to start @@ -62,9 +65,3 @@ this library and does not happen automatically. the time of creating the disk exporter, then it will be ignored, and the next oldest (and unexpired) one will be used instead. * All the stale and empty files will be removed as a new file is created. -* The [Storage](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/Storage.java), - [FolderManager](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManager.java) - and [ReadableFile](src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFile.java) - files contain more information on the details of the file reading process. -* Note that the reader delegates the data to the exporter exactly in the way it has received the - data - it does not try to batch data (but this could be an optimization in the future). diff --git a/disk-buffering/README.md b/disk-buffering/README.md index 72e17144d..9178794ad 100644 --- a/disk-buffering/README.md +++ b/disk-buffering/README.md @@ -1,109 +1,132 @@ # Disk buffering -This module provides exporters that store telemetry data in files which can be -sent later on demand. A high level description of how it works is that there are two separate -processes in place, one for writing data in disk, and one for reading/exporting the previously -stored data. +This module provides an abstraction +named [SignalStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/SignalStorage.java), +as well as default implementations for each signal type that allow writing signals to disk and +reading them later. -* Each exporter stores the received data automatically in disk right after it's received from its - processor. -* The reading of the data back from disk and exporting process has to be done manually. At - the moment there's no automatic mechanism to do so. There's more information on how it can be - achieved, under [Reading data](#reading-data). +For a more detailed information on how the whole process works, take a look at +the [DESIGN.md](DESIGN.md) file. -> For a more detailed information on how the whole process works, take a look at -> the [DESIGN.md](DESIGN.md) file. +## Default implementation usage -## Configuration +The default implementations are the following: -The configurable parameters are provided **per exporter**, the available ones are: +* [FileSpanStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileSpanStorage.java) +* [FileLogRecordStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileLogRecordStorage.java) +* [FileMetricStorage](src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileMetricStorage.java) -* Max file size, defaults to 1MB. -* Max folder size, defaults to 10MB. All files are stored in a single folder per-signal, therefore - if all 3 types of signals are stored, the total amount of space from disk to be taken by default - would be of 30MB. -* Max age for file writing, defaults to 30 seconds. -* Min age for file reading, defaults to 33 seconds. It must be greater that the max age for file - writing. -* Max age for file reading, defaults to 18 hours. After that time passes, the file will be - considered stale and will be removed when new files are created. No more data will be read from a - file past this time. - -## Usage +### Set up -### Storing data +We need to create a signal storage object per signal type to start writing signals to disk. Each +`File*Storage` implementation has a `create()` function that receives: + +* A File directory to store the signal files. Note that each signal storage object must have a + dedicated directory to work properly. +* (Optional) a configuration object. -In order to use it, you need to wrap your own exporter with a new instance of -the ones provided in here: +The available configuration parameters are the following: -* For a LogRecordExporter, it must be wrapped within - a [LogRecordToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporter.java). -* For a MetricExporter, it must be wrapped within - a [MetricToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporter.java). -* For a SpanExporter, it must be wrapped within - a [SpanToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporter.java). +* Max file size, defaults to 1MB. +* Max folder size, defaults to 10MB. +* Max age for file writing. It sets the time window where a file can get signals appended to it. + Defaults to 30 seconds. +* Min age for file reading. It sets the time to wait before starting to read from a file after + its creation. Defaults to 33 seconds. It must be greater that the max age for file writing. +* Max age for file reading. After that time passes, the file will be considered stale and will be + removed when new files are created. No more data will be read from a file past this time. Defaults + to 18 hours. -Each wrapper will need the following when instantiating them: +```java +// Root dir +File rootDir = new File("/some/root"); -* The exporter to be wrapped. -* A File instance of the root directory where all the data is going to be written. The same root dir - can be used for all the wrappers, since each will create their own folder inside it. -* An instance - of [StorageConfiguration](src/main/java/io/opentelemetry/contrib/disk/buffering/config/StorageConfiguration.java) - with the desired parameters. You can create one with default values by - calling `StorageConfiguration.getDefault()`. +// Setting up span storage +SignalStorage.Span spanStorage = FileSpanStorage.create(new File(rootDir, "spans")); -After wrapping your exporters, you must register the wrapper as the exporter you'll use. It will -take care of always storing the data it receives. +// Setting up metric storage +SignalStorage.Metric metricStorage = FileMetricStorage.create(new File(rootDir, "metrics")); -#### Set up example for spans +// Setting up log storage +SignalStorage.LogRecord logStorage = FileLogRecordStorage.create(new File(rootDir, "logs")); +``` -### Writing data +### Storing data -The data is written in the disk by "ToDisk" exporters, these are exporters that serialize and store the data as received by their processors. If for some reason -the "ToDisk" cannot store data in the disk, they'll delegate the data to their wrapped exporter. +While you could manually call your `SignalStorage.write(items)` function, disk buffering +provides convenience exporters that you can use in your OpenTelemetry's instance, so +that all signals are automatically stored as they are created. -```java -// Creating the SpanExporter of our choice. -SpanExporter mySpanExporter = OtlpGrpcSpanExporter.getDefault(); +* For a span storage, use + a [SpanToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SpanToDiskExporter.java). +* For a log storage, use + a [LogRecordToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/LogRecordToDiskExporter.java). +* For a metric storage, use + a [MetricToDiskExporter](src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/MetricToDiskExporter.java). -// Wrapping our exporter with its "ToDisk" exporter. -SpanToDiskExporter toDiskExporter = SpanToDiskExporter.create(mySpanExporter, StorageConfiguration.getDefault(new File("/my/signals/cache/dir"))); +Each will wrap a signal storage for its respective signal type, as well as an optional callback +to notify when it succeeds, fails, and gets shutdown. - // Registering the disk exporter within our OpenTelemetry instance. -SdkTracerProvider myTraceProvider = SdkTracerProvider.builder() - .addSpanProcessor(SimpleSpanProcessor.create(toDiskExporter)) +```java +// Setting up span to disk exporter +SpanToDiskExporter spanToDiskExporter = + SpanToDiskExporter.builder(spanStorage).setExporterCallback(spanCallback).build(); +// Setting up metric to disk +MetricToDiskExporter metricToDiskExporter = + MetricToDiskExporter.builder(metricStorage).setExporterCallback(metricCallback).build(); +// Setting up log to disk exporter +LogRecordToDiskExporter logToDiskExporter = + LogRecordToDiskExporter.builder(logStorage).setExporterCallback(logCallback).build(); + +// Using exporters in your OpenTelemetry instance. +OpenTelemetry openTelemetry = + OpenTelemetrySdk.builder() + // Using span to disk exporter + .setTracerProvider( + SdkTracerProvider.builder() + .addSpanProcessor(BatchSpanProcessor.builder(spanToDiskExporter).build()) + .build()) + // Using log to disk exporter + .setLoggerProvider( + SdkLoggerProvider.builder() + .addLogRecordProcessor( + BatchLogRecordProcessor.builder(logToDiskExporter).build()) + .build()) + // Using metric to disk exporter + .setMeterProvider( + SdkMeterProvider.builder() + .registerMetricReader(PeriodicMetricReader.create(metricToDiskExporter)) + .build()) .build(); -OpenTelemetrySdk.builder() - .setTracerProvider(myTraceProvider) - .buildAndRegisterGlobal(); - ``` +Now when creating signals using your `OpenTelemetry` instance, those will get stored in disk. + ### Reading data -In order to read data, we need to create "FromDisk" exporters, which read data from the disk, parse it and delegate it -to their wrapped exporters. +In order to read data, we can iterate through our signal storage objects and then forward them to +a network exporter, as shown in the example for spans below. ```java -try { - SpanFromDiskExporter fromDiskExporter = SpanFromDiskExporter.create(memorySpanExporter, storageConfig); - if(fromDiskExporter.exportStoredBatch(1, TimeUnit.SECONDS)) { - // A batch was successfully exported and removed from disk. You can call this method for as long as it keeps returning true. - } else { - // Either there was no data in the disk or the wrapped exporter returned CompletableResultCode.ofFailure(). - } -} catch (IOException e) { - // Something unexpected happened. +// Example of reading an exporting spans from disk +OtlpHttpSpanExporter networkExporter; +Iterator> spanCollections = spanStorage.iterator(); +while(spanCollections.hasNext()){ + networkExporter.export(spanCollections.next()); } ``` +The `File*Storage` iterators delete the previously returned collection when `next()` is called, +assuming that if the next collection is requested is because the previous one was successfully +consumed. + Both the writing and reading processes can run in parallel and they don't overlap because each is supposed to happen in different files. We ensure that reader and writer don't -accidentally meet in the same file by using the configurable parameters. These parameters set non-overlapping time frames for each action to be done on a single file at a time. On top of that, there's a mechanism in -place to avoid overlapping on edge cases where the time frames ended but the resources haven't been -released. For that mechanism to work properly, this tool assumes that both the reading and the -writing actions are executed within the same application process. +accidentally meet in the same file by using the configurable parameters. These parameters set +non-overlapping time frames for each action to be done on a single file at a time. On top of that, +there's a mechanism in place to avoid overlapping on edge cases where the time frames ended but the +resources haven't been released. For that mechanism to work properly, this tool assumes that both +the reading and the writing actions are executed within the same application process. ## Component owners diff --git a/disk-buffering/assets/reading-flow.png b/disk-buffering/assets/reading-flow.png index 76b8de438..63750e5a3 100644 Binary files a/disk-buffering/assets/reading-flow.png and b/disk-buffering/assets/reading-flow.png differ diff --git a/disk-buffering/assets/writing-flow.png b/disk-buffering/assets/writing-flow.png index c6144b301..b4b21359d 100644 Binary files a/disk-buffering/assets/writing-flow.png and b/disk-buffering/assets/writing-flow.png differ diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordFromDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordFromDiskExporter.java deleted file mode 100644 index 6c38be04a..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordFromDiskExporter.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporter; -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.logs.data.LogRecordData; -import io.opentelemetry.sdk.logs.export.LogRecordExporter; -import java.io.IOException; -import java.util.concurrent.TimeUnit; - -public final class LogRecordFromDiskExporter implements FromDiskExporter { - - private final FromDiskExporterImpl delegate; - - public static LogRecordFromDiskExporter create(LogRecordExporter exporter, Storage storage) - throws IOException { - FromDiskExporterImpl delegate = - FromDiskExporterImpl.builder(storage) - .setDeserializer(SignalDeserializer.ofLogs()) - .setExportFunction(exporter::export) - .build(); - return new LogRecordFromDiskExporter(delegate); - } - - private LogRecordFromDiskExporter(FromDiskExporterImpl delegate) { - this.delegate = delegate; - } - - @Override - public boolean exportStoredBatch(long timeout, TimeUnit unit) throws IOException { - return delegate.exportStoredBatch(timeout, unit); - } - - @Override - public void shutdown() throws IOException { - delegate.shutdown(); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporter.java deleted file mode 100644 index 1524723e8..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporter.java +++ /dev/null @@ -1,65 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.ToDiskExporter; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.logs.data.LogRecordData; -import io.opentelemetry.sdk.logs.export.LogRecordExporter; -import java.io.IOException; -import java.util.Collection; - -/** - * This class implements a {@link LogRecordExporter} that delegates to an instance of {@code - * ToDiskExporter}. - */ -public final class LogRecordToDiskExporter implements LogRecordExporter { - - private final ToDiskExporter delegate; - - /** - * Creates a new LogRecordToDiskExporter that will buffer LogRecordData telemetry on disk storage. - * - * @param delegate - The LogRecordExporter to delegate to if disk writing fails. - * @param storage - The Storage instance that specifies how storage is managed. - * @return A new LogRecordToDiskExporter instance. - */ - public static LogRecordToDiskExporter create(LogRecordExporter delegate, Storage storage) { - ToDiskExporter toDisk = - ToDiskExporter.builder(storage) - .setSerializer(SignalSerializer.ofLogs()) - .setExportFunction(delegate::export) - .build(); - return new LogRecordToDiskExporter(toDisk); - } - - // Visible for testing - LogRecordToDiskExporter(ToDiskExporter delegate) { - this.delegate = delegate; - } - - @Override - public CompletableResultCode export(Collection logs) { - return delegate.export(logs); - } - - @Override - public CompletableResultCode flush() { - return CompletableResultCode.ofSuccess(); - } - - @Override - public CompletableResultCode shutdown() { - try { - delegate.shutdown(); - return CompletableResultCode.ofSuccess(); - } catch (IOException e) { - return CompletableResultCode.ofFailure(); - } - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/MetricFromDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/MetricFromDiskExporter.java deleted file mode 100644 index 36d478e7b..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/MetricFromDiskExporter.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporter; -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.metrics.data.MetricData; -import io.opentelemetry.sdk.metrics.export.MetricExporter; -import java.io.IOException; -import java.util.concurrent.TimeUnit; - -public final class MetricFromDiskExporter implements FromDiskExporter { - - private final FromDiskExporterImpl delegate; - - public static MetricFromDiskExporter create(MetricExporter exporter, Storage storage) - throws IOException { - FromDiskExporterImpl delegate = - FromDiskExporterImpl.builder(storage) - .setDeserializer(SignalDeserializer.ofMetrics()) - .setExportFunction(exporter::export) - .build(); - return new MetricFromDiskExporter(delegate); - } - - private MetricFromDiskExporter(FromDiskExporterImpl delegate) { - this.delegate = delegate; - } - - @Override - public boolean exportStoredBatch(long timeout, TimeUnit unit) throws IOException { - return delegate.exportStoredBatch(timeout, unit); - } - - @Override - public void shutdown() throws IOException { - delegate.shutdown(); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporter.java deleted file mode 100644 index 2e0848684..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporter.java +++ /dev/null @@ -1,76 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.ToDiskExporter; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.metrics.InstrumentType; -import io.opentelemetry.sdk.metrics.data.AggregationTemporality; -import io.opentelemetry.sdk.metrics.data.MetricData; -import io.opentelemetry.sdk.metrics.export.AggregationTemporalitySelector; -import io.opentelemetry.sdk.metrics.export.MetricExporter; -import java.io.IOException; -import java.util.Collection; - -/** - * This class implements a {@link MetricExporter} that delegates to an instance of {@code - * ToDiskExporter}. - */ -public final class MetricToDiskExporter implements MetricExporter { - - private final ToDiskExporter delegate; - private final AggregationTemporalitySelector aggregationTemporalitySelector; - - /** - * Creates a new MetricToDiskExporter that will buffer Metric telemetry on disk storage. - * - * @param delegate - The MetricExporter to delegate to if disk writing fails. - * @param storage - The Storage instance that specifies how storage is managed. - * @return A new MetricToDiskExporter instance. - */ - public static MetricToDiskExporter create(MetricExporter delegate, Storage storage) { - ToDiskExporter toDisk = - ToDiskExporter.builder(storage) - .setSerializer(SignalSerializer.ofMetrics()) - .setExportFunction(delegate::export) - .build(); - return new MetricToDiskExporter(toDisk, delegate); - } - - // VisibleForTesting - MetricToDiskExporter( - ToDiskExporter delegate, AggregationTemporalitySelector selector) { - this.delegate = delegate; - this.aggregationTemporalitySelector = selector; - } - - @Override - public CompletableResultCode export(Collection metrics) { - return delegate.export(metrics); - } - - @Override - public CompletableResultCode flush() { - return CompletableResultCode.ofSuccess(); - } - - @Override - public CompletableResultCode shutdown() { - try { - delegate.shutdown(); - } catch (IOException e) { - return CompletableResultCode.ofFailure(); - } - return CompletableResultCode.ofSuccess(); - } - - @Override - public AggregationTemporality getAggregationTemporality(InstrumentType instrumentType) { - return aggregationTemporalitySelector.getAggregationTemporality(instrumentType); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SignalType.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SignalType.java deleted file mode 100644 index c66ed940e..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SignalType.java +++ /dev/null @@ -1,12 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -public enum SignalType { - SPAN, - LOG, - METRIC -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SpanFromDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SpanFromDiskExporter.java deleted file mode 100644 index 9523c8a2f..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SpanFromDiskExporter.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporter; -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.trace.data.SpanData; -import io.opentelemetry.sdk.trace.export.SpanExporter; -import java.io.IOException; -import java.util.concurrent.TimeUnit; - -public final class SpanFromDiskExporter implements FromDiskExporter { - - private final FromDiskExporterImpl delegate; - - public static SpanFromDiskExporter create(SpanExporter exporter, Storage storage) - throws IOException { - FromDiskExporterImpl delegate = - FromDiskExporterImpl.builder(storage) - .setDeserializer(SignalDeserializer.ofSpans()) - .setExportFunction(exporter::export) - .build(); - return new SpanFromDiskExporter(delegate); - } - - private SpanFromDiskExporter(FromDiskExporterImpl delegate) { - this.delegate = delegate; - } - - @Override - public boolean exportStoredBatch(long timeout, TimeUnit unit) throws IOException { - return delegate.exportStoredBatch(timeout, unit); - } - - @Override - public void shutdown() throws IOException { - delegate.shutdown(); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporter.java deleted file mode 100644 index dcd79d3b0..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporter.java +++ /dev/null @@ -1,65 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.ToDiskExporter; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.trace.data.SpanData; -import io.opentelemetry.sdk.trace.export.SpanExporter; -import java.io.IOException; -import java.util.Collection; - -/** - * This class implements a SpanExporter that delegates to an instance of {@code - * ToDiskExporter}. - */ -public final class SpanToDiskExporter implements SpanExporter { - - private final ToDiskExporter delegate; - - /** - * Creates a new SpanToDiskExporter that will buffer Span telemetry on disk storage. - * - * @param delegate - The SpanExporter to delegate to if disk writing fails. - * @param storage - The Storage instance that specifies how storage is managed. - * @return A new SpanToDiskExporter instance. - */ - public static SpanToDiskExporter create(SpanExporter delegate, Storage storage) { - ToDiskExporter toDisk = - ToDiskExporter.builder(storage) - .setSerializer(SignalSerializer.ofSpans()) - .setExportFunction(delegate::export) - .build(); - return new SpanToDiskExporter(toDisk); - } - - // Visible for testing - SpanToDiskExporter(ToDiskExporter delegate) { - this.delegate = delegate; - } - - @Override - public CompletableResultCode export(Collection spans) { - return delegate.export(spans); - } - - @Override - public CompletableResultCode flush() { - return CompletableResultCode.ofSuccess(); - } - - @Override - public CompletableResultCode shutdown() { - try { - delegate.shutdown(); - } catch (IOException e) { - return CompletableResultCode.ofFailure(); - } - return CompletableResultCode.ofSuccess(); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/ExporterCallback.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/ExporterCallback.java deleted file mode 100644 index f877c7b7d..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/ExporterCallback.java +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.exporters; - -import io.opentelemetry.contrib.disk.buffering.SignalType; -import javax.annotation.Nullable; - -/** Notifies about exporter and storage-related operations from within a signal to disk exporter. */ -public interface ExporterCallback { - /** - * Called when an export to disk operation succeeded. - * - * @param type The type of signal associated to the exporter. - */ - void onExportSuccess(SignalType type); - - /** - * Called when an export to disk operation failed. - * - * @param type The type of signal associated to the exporter. - * @param error Optional - provides more information of why the operation failed. - */ - void onExportError(SignalType type, @Nullable Throwable error); - - /** - * Called when the exporter is closed. - * - * @param type The type of signal associated to the exporter. - */ - void onShutdown(SignalType type); - - static ExporterCallback noop() { - return NoopExporterCallback.INSTANCE; - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/LogRecordToDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/LogRecordToDiskExporter.java new file mode 100644 index 000000000..6ed7ae2b4 --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/LogRecordToDiskExporter.java @@ -0,0 +1,81 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.exporters; + +import com.google.errorprone.annotations.CanIgnoreReturnValue; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.ExporterCallback; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.NoopExporterCallback; +import io.opentelemetry.contrib.disk.buffering.internal.exporters.SignalStorageExporter; +import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; +import io.opentelemetry.sdk.common.CompletableResultCode; +import io.opentelemetry.sdk.logs.data.LogRecordData; +import io.opentelemetry.sdk.logs.export.LogRecordExporter; +import java.time.Duration; +import java.util.Collection; + +/** Exporter that stores logs into disk. */ +public final class LogRecordToDiskExporter implements LogRecordExporter { + private final SignalStorageExporter storageExporter; + private final ExporterCallback callback; + private static final ExporterCallback DEFAULT_CALLBACK = + new NoopExporterCallback<>(); + private static final Duration DEFAULT_EXPORT_TIMEOUT = Duration.ofSeconds(10); + + private LogRecordToDiskExporter( + SignalStorageExporter storageExporter, + ExporterCallback callback) { + this.storageExporter = storageExporter; + this.callback = callback; + } + + public static Builder builder(SignalStorage.LogRecord storage) { + return new Builder(storage); + } + + @Override + public CompletableResultCode export(Collection logs) { + return storageExporter.exportToStorage(logs); + } + + @Override + public CompletableResultCode flush() { + return CompletableResultCode.ofSuccess(); + } + + @Override + public CompletableResultCode shutdown() { + callback.onShutdown(); + return CompletableResultCode.ofSuccess(); + } + + public static final class Builder { + private final SignalStorage.LogRecord storage; + private ExporterCallback callback = DEFAULT_CALLBACK; + private Duration writeTimeout = DEFAULT_EXPORT_TIMEOUT; + + @CanIgnoreReturnValue + public Builder setExporterCallback(ExporterCallback value) { + callback = value; + return this; + } + + @CanIgnoreReturnValue + public Builder setWriteTimeout(Duration value) { + writeTimeout = value; + return this; + } + + public LogRecordToDiskExporter build() { + SignalStorageExporter storageExporter = + new SignalStorageExporter<>(storage, callback, writeTimeout); + return new LogRecordToDiskExporter(storageExporter, callback); + } + + private Builder(SignalStorage.LogRecord storage) { + this.storage = storage; + } + } +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/MetricToDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/MetricToDiskExporter.java new file mode 100644 index 000000000..fe7a86abf --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/MetricToDiskExporter.java @@ -0,0 +1,99 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.exporters; + +import com.google.errorprone.annotations.CanIgnoreReturnValue; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.ExporterCallback; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.NoopExporterCallback; +import io.opentelemetry.contrib.disk.buffering.internal.exporters.SignalStorageExporter; +import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; +import io.opentelemetry.sdk.common.CompletableResultCode; +import io.opentelemetry.sdk.metrics.InstrumentType; +import io.opentelemetry.sdk.metrics.data.AggregationTemporality; +import io.opentelemetry.sdk.metrics.data.MetricData; +import io.opentelemetry.sdk.metrics.export.AggregationTemporalitySelector; +import io.opentelemetry.sdk.metrics.export.MetricExporter; +import java.time.Duration; +import java.util.Collection; + +/** Exporter that stores metrics into disk. */ +public final class MetricToDiskExporter implements MetricExporter { + private final SignalStorageExporter storageExporter; + private final AggregationTemporalitySelector aggregationTemporalitySelector; + private final ExporterCallback callback; + private static final ExporterCallback DEFAULT_CALLBACK = new NoopExporterCallback<>(); + private static final Duration DEFAULT_EXPORT_TIMEOUT = Duration.ofSeconds(10); + + private MetricToDiskExporter( + SignalStorageExporter storageExporter, + AggregationTemporalitySelector aggregationTemporalitySelector, + ExporterCallback callback) { + this.storageExporter = storageExporter; + this.aggregationTemporalitySelector = aggregationTemporalitySelector; + this.callback = callback; + } + + public static Builder builder(SignalStorage.Metric storage) { + return new Builder(storage); + } + + @Override + public CompletableResultCode export(Collection metrics) { + return storageExporter.exportToStorage(metrics); + } + + @Override + public CompletableResultCode flush() { + return CompletableResultCode.ofSuccess(); + } + + @Override + public CompletableResultCode shutdown() { + callback.onShutdown(); + return CompletableResultCode.ofSuccess(); + } + + @Override + public AggregationTemporality getAggregationTemporality(InstrumentType instrumentType) { + return aggregationTemporalitySelector.getAggregationTemporality(instrumentType); + } + + public static final class Builder { + private final SignalStorage.Metric storage; + private AggregationTemporalitySelector aggregationTemporalitySelector = + AggregationTemporalitySelector.alwaysCumulative(); + private ExporterCallback callback = DEFAULT_CALLBACK; + private Duration writeTimeout = DEFAULT_EXPORT_TIMEOUT; + + @CanIgnoreReturnValue + public Builder setExporterCallback(ExporterCallback value) { + callback = value; + return this; + } + + @CanIgnoreReturnValue + public Builder setWriteTimeout(Duration value) { + writeTimeout = value; + return this; + } + + @CanIgnoreReturnValue + public Builder setAggregationTemporalitySelector(AggregationTemporalitySelector value) { + aggregationTemporalitySelector = value; + return this; + } + + public MetricToDiskExporter build() { + SignalStorageExporter storageExporter = + new SignalStorageExporter<>(storage, callback, writeTimeout); + return new MetricToDiskExporter(storageExporter, aggregationTemporalitySelector, callback); + } + + private Builder(SignalStorage.Metric storage) { + this.storage = storage; + } + } +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/NoopExporterCallback.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/NoopExporterCallback.java deleted file mode 100644 index 2dd4f2f70..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/NoopExporterCallback.java +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.exporters; - -import io.opentelemetry.contrib.disk.buffering.SignalType; -import javax.annotation.Nullable; - -final class NoopExporterCallback implements ExporterCallback { - static final NoopExporterCallback INSTANCE = new NoopExporterCallback(); - - private NoopExporterCallback() {} - - @Override - public void onExportSuccess(SignalType type) {} - - @Override - public void onExportError(SignalType type, @Nullable Throwable error) {} - - @Override - public void onShutdown(SignalType type) {} -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SpanToDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SpanToDiskExporter.java index 3efe5c367..9558a2767 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SpanToDiskExporter.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SpanToDiskExporter.java @@ -6,7 +6,9 @@ package io.opentelemetry.contrib.disk.buffering.exporters; import com.google.errorprone.annotations.CanIgnoreReturnValue; -import io.opentelemetry.contrib.disk.buffering.SignalType; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.ExporterCallback; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.NoopExporterCallback; +import io.opentelemetry.contrib.disk.buffering.internal.exporters.SignalStorageExporter; import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; import io.opentelemetry.sdk.common.CompletableResultCode; import io.opentelemetry.sdk.trace.data.SpanData; @@ -16,13 +18,13 @@ /** Exporter that stores spans into disk. */ public final class SpanToDiskExporter implements SpanExporter { - private static final SignalType TYPE = SignalType.SPAN; - private final SignalStorageExporter storageExporter; - private final ExporterCallback callback; + private final ExporterCallback callback; + private static final ExporterCallback DEFAULT_CALLBACK = new NoopExporterCallback<>(); + private static final Duration DEFAULT_EXPORT_TIMEOUT = Duration.ofSeconds(10); private SpanToDiskExporter( - SignalStorageExporter storageExporter, ExporterCallback callback) { + SignalStorageExporter storageExporter, ExporterCallback callback) { this.storageExporter = storageExporter; this.callback = callback; } @@ -43,21 +45,21 @@ public CompletableResultCode flush() { @Override public CompletableResultCode shutdown() { - callback.onShutdown(TYPE); + callback.onShutdown(); return CompletableResultCode.ofSuccess(); } public static final class Builder { private final SignalStorage.Span storage; - private ExporterCallback callback = ExporterCallback.noop(); - private Duration writeTimeout = Duration.ofSeconds(10); + private ExporterCallback callback = DEFAULT_CALLBACK; + private Duration writeTimeout = DEFAULT_EXPORT_TIMEOUT; private Builder(SignalStorage.Span storage) { this.storage = storage; } @CanIgnoreReturnValue - public Builder setExporterCallback(ExporterCallback value) { + public Builder setExporterCallback(ExporterCallback value) { callback = value; return this; } @@ -70,7 +72,7 @@ public Builder setWriteTimeout(Duration value) { public SpanToDiskExporter build() { SignalStorageExporter storageExporter = - new SignalStorageExporter<>(storage, callback, writeTimeout, TYPE); + new SignalStorageExporter<>(storage, callback, writeTimeout); return new SpanToDiskExporter(storageExporter, callback); } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/callback/ExporterCallback.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/callback/ExporterCallback.java new file mode 100644 index 000000000..9c3c816ea --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/callback/ExporterCallback.java @@ -0,0 +1,30 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.exporters.callback; + +import java.util.Collection; +import javax.annotation.Nullable; + +/** Notifies about exporter and storage-related operations from within a signal to disk exporter. */ +public interface ExporterCallback { + /** + * Called when an export to disk operation succeeded. + * + * @param items The items successfully stored in disk. + */ + void onExportSuccess(Collection items); + + /** + * Called when an export to disk operation failed. + * + * @param items The items that couldn't get stored in disk. + * @param error Optional - provides more information of why the operation failed. + */ + void onExportError(Collection items, @Nullable Throwable error); + + /** Called when the exporter is closed. */ + void onShutdown(); +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/callback/NoopExporterCallback.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/callback/NoopExporterCallback.java new file mode 100644 index 000000000..6313d1a5b --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/callback/NoopExporterCallback.java @@ -0,0 +1,21 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.exporters.callback; + +import java.util.Collection; +import javax.annotation.Nullable; + +public final class NoopExporterCallback implements ExporterCallback { + + @Override + public void onExportSuccess(Collection items) {} + + @Override + public void onExportError(Collection items, @Nullable Throwable error) {} + + @Override + public void onShutdown() {} +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporter.java deleted file mode 100644 index fdc3bb796..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporter.java +++ /dev/null @@ -1,15 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.exporter; - -import java.io.IOException; -import java.util.concurrent.TimeUnit; - -public interface FromDiskExporter { - boolean exportStoredBatch(long timeout, TimeUnit unit) throws IOException; - - void shutdown() throws IOException; -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporterBuilder.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporterBuilder.java deleted file mode 100644 index a91ded1f3..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporterBuilder.java +++ /dev/null @@ -1,55 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.exporter; - -import static java.util.Collections.emptyList; - -import com.google.errorprone.annotations.CanIgnoreReturnValue; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.common.CompletableResultCode; -import java.io.IOException; -import java.util.Collection; -import java.util.function.Function; -import org.jetbrains.annotations.NotNull; - -public class FromDiskExporterBuilder { - - private SignalDeserializer serializer = noopDeserializer(); - private final Storage storage; - - private Function, CompletableResultCode> exportFunction = - x -> CompletableResultCode.ofFailure(); - - public FromDiskExporterBuilder(Storage storage) { - if (storage == null) { - throw new NullPointerException("Storage cannot be null"); - } - this.storage = storage; - } - - @NotNull - private static SignalDeserializer noopDeserializer() { - return x -> emptyList(); - } - - @CanIgnoreReturnValue - public FromDiskExporterBuilder setDeserializer(SignalDeserializer serializer) { - this.serializer = serializer; - return this; - } - - @CanIgnoreReturnValue - public FromDiskExporterBuilder setExportFunction( - Function, CompletableResultCode> exportFunction) { - this.exportFunction = exportFunction; - return this; - } - - public FromDiskExporterImpl build() throws IOException { - return new FromDiskExporterImpl<>(serializer, exportFunction, storage); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporterImpl.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporterImpl.java deleted file mode 100644 index 5ba5c2390..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/FromDiskExporterImpl.java +++ /dev/null @@ -1,86 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.exporter; - -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.DeserializationException; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.ProcessResult; -import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.ReadableResult; -import io.opentelemetry.contrib.disk.buffering.internal.utils.DebugLogger; -import io.opentelemetry.sdk.common.CompletableResultCode; -import java.io.IOException; -import java.util.Collection; -import java.util.List; -import java.util.concurrent.TimeUnit; -import java.util.function.Function; -import java.util.logging.Logger; - -/** - * Signal-type generic class that can read telemetry previously buffered on disk and send it to - * another delegated exporter. - */ -public final class FromDiskExporterImpl implements FromDiskExporter { - private final DebugLogger logger; - private final Storage storage; - private final SignalDeserializer deserializer; - private final Function, CompletableResultCode> exportFunction; - - FromDiskExporterImpl( - SignalDeserializer deserializer, - Function, CompletableResultCode> exportFunction, - Storage storage) { - this.deserializer = deserializer; - this.exportFunction = exportFunction; - this.storage = storage; - this.logger = - DebugLogger.wrap( - Logger.getLogger(FromDiskExporterImpl.class.getName()), storage.isDebugEnabled()); - } - - public static FromDiskExporterBuilder builder(Storage storage) { - return new FromDiskExporterBuilder<>(storage); - } - - /** - * Reads data from the disk and attempts to export it. - * - * @param timeout The amount of time to wait for the wrapped exporter to finish. - * @param unit The unit of the time provided. - * @return true if there was data available, and it was successfully exported within the timeout - * provided. false otherwise. - * @throws IOException If an unexpected error happens. - */ - @Override - public boolean exportStoredBatch(long timeout, TimeUnit unit) throws IOException { - logger.log("Attempting to export " + deserializer.signalType() + " batch from disk."); - ReadableResult result = - storage.readAndProcess( - bytes -> { - logger.log( - "Read " - + bytes.length - + " " - + deserializer.signalType() - + " bytes from storage."); - try { - List telemetry = deserializer.deserialize(bytes); - logger.log( - "Now exporting batch of " + telemetry.size() + " " + deserializer.signalType()); - CompletableResultCode join = exportFunction.apply(telemetry).join(timeout, unit); - return join.isSuccess() ? ProcessResult.SUCCEEDED : ProcessResult.TRY_LATER; - } catch (DeserializationException e) { - return ProcessResult.CONTENT_INVALID; - } - }); - return result == ReadableResult.SUCCEEDED; - } - - @Override - public void shutdown() throws IOException { - storage.close(); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/NoopSerializer.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/NoopSerializer.java deleted file mode 100644 index 715f538c4..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/NoopSerializer.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.exporter; - -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import java.io.IOException; -import java.io.OutputStream; -import java.util.Collection; - -class NoopSerializer implements SignalSerializer { - - @Override - public NoopSerializer initialize(Collection data) { - return this; - } - - @Override - public void writeBinaryTo(OutputStream output) throws IOException {} - - @Override - public int getBinarySerializedSize() { - return 0; - } - - @Override - public void reset() {} -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporter.java deleted file mode 100644 index a3104a892..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporter.java +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.exporter; - -import static java.util.logging.Level.FINER; -import static java.util.logging.Level.WARNING; - -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.contrib.disk.buffering.internal.utils.DebugLogger; -import io.opentelemetry.sdk.common.CompletableResultCode; -import java.io.IOException; -import java.util.Collection; -import java.util.function.Function; -import java.util.logging.Logger; - -public class ToDiskExporter { - - private final DebugLogger logger; - private final Storage storage; - private final SignalSerializer serializer; - private final Function, CompletableResultCode> exportFunction; - - ToDiskExporter( - SignalSerializer serializer, - Function, CompletableResultCode> exportFunction, - Storage storage) { - this.serializer = serializer; - this.exportFunction = exportFunction; - this.storage = storage; - this.logger = - DebugLogger.wrap( - Logger.getLogger(ToDiskExporter.class.getName()), storage.isDebugEnabled()); - } - - public static ToDiskExporterBuilder builder(Storage storage) { - return new ToDiskExporterBuilder<>(storage); - } - - public synchronized CompletableResultCode export(Collection data) { - logger.log("Intercepting exporter batch.", FINER); - try { - serializer.initialize(data); - if (storage.write(serializer)) { - return CompletableResultCode.ofSuccess(); - } - logger.log("Could not store batch in disk. Exporting it right away."); - return exportFunction.apply(data); - } catch (IOException e) { - logger.log( - "An unexpected error happened while attempting to write the data in disk. Exporting it right away.", - WARNING, - e); - return exportFunction.apply(data); - } finally { - serializer.reset(); - } - } - - public void shutdown() throws IOException { - storage.close(); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporterBuilder.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporterBuilder.java deleted file mode 100644 index be75a3976..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporterBuilder.java +++ /dev/null @@ -1,47 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.exporter; - -import com.google.errorprone.annotations.CanIgnoreReturnValue; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.common.CompletableResultCode; -import java.util.Collection; -import java.util.function.Function; - -public final class ToDiskExporterBuilder { - - private SignalSerializer serializer = new NoopSerializer(); - - private final Storage storage; - - private Function, CompletableResultCode> exportFunction = - x -> CompletableResultCode.ofFailure(); - - ToDiskExporterBuilder(Storage storage) { - if (storage == null) { - throw new NullPointerException("Storage cannot be null"); - } - this.storage = storage; - } - - @CanIgnoreReturnValue - public ToDiskExporterBuilder setSerializer(SignalSerializer serializer) { - this.serializer = serializer; - return this; - } - - @CanIgnoreReturnValue - public ToDiskExporterBuilder setExportFunction( - Function, CompletableResultCode> exportFunction) { - this.exportFunction = exportFunction; - return this; - } - - public ToDiskExporter build() { - return new ToDiskExporter<>(serializer, exportFunction, storage); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SignalStorageExporter.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporters/SignalStorageExporter.java similarity index 76% rename from disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SignalStorageExporter.java rename to disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporters/SignalStorageExporter.java index c508d042f..22ba6b61c 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/exporters/SignalStorageExporter.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/exporters/SignalStorageExporter.java @@ -3,11 +3,11 @@ * SPDX-License-Identifier: Apache-2.0 */ -package io.opentelemetry.contrib.disk.buffering.exporters; +package io.opentelemetry.contrib.disk.buffering.internal.exporters; import static java.util.concurrent.TimeUnit.MILLISECONDS; -import io.opentelemetry.contrib.disk.buffering.SignalType; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.ExporterCallback; import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; import io.opentelemetry.contrib.disk.buffering.storage.result.WriteResult; import io.opentelemetry.sdk.common.CompletableResultCode; @@ -18,18 +18,16 @@ import java.util.concurrent.TimeoutException; /** Internal utility for common export to disk operations across all exporters. */ -final class SignalStorageExporter { +public final class SignalStorageExporter { private final SignalStorage storage; - private final ExporterCallback callback; + private final ExporterCallback callback; private final Duration writeTimeout; - private final SignalType type; public SignalStorageExporter( - SignalStorage storage, ExporterCallback callback, Duration writeTimeout, SignalType type) { + SignalStorage storage, ExporterCallback callback, Duration writeTimeout) { this.storage = storage; this.callback = callback; this.writeTimeout = writeTimeout; - this.type = type; } public CompletableResultCode exportToStorage(Collection items) { @@ -37,18 +35,18 @@ public CompletableResultCode exportToStorage(Collection items) { try { WriteResult operation = future.get(writeTimeout.toMillis(), MILLISECONDS); if (operation.isSuccessful()) { - callback.onExportSuccess(type); + callback.onExportSuccess(items); return CompletableResultCode.ofSuccess(); } Throwable error = operation.getError(); - callback.onExportError(type, error); + callback.onExportError(items, error); if (error != null) { return CompletableResultCode.ofExceptionalFailure(error); } return CompletableResultCode.ofFailure(); } catch (ExecutionException | InterruptedException | TimeoutException e) { - callback.onExportError(type, e); + callback.onExportError(items, e); return CompletableResultCode.ofExceptionalFailure(e); } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/LogRecordDataDeserializer.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/LogRecordDataDeserializer.java index 075a0f103..d4c877923 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/LogRecordDataDeserializer.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/LogRecordDataDeserializer.java @@ -6,7 +6,6 @@ package io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers; import io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.logs.ProtoLogsDataMapper; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; import io.opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest; import io.opentelemetry.sdk.logs.data.LogRecordData; import java.io.IOException; @@ -30,9 +29,4 @@ public List deserialize(byte[] source) throws DeserializationExce throw new DeserializationException(e); } } - - @Override - public String signalType() { - return SignalTypes.logs.name(); - } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/MetricDataDeserializer.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/MetricDataDeserializer.java index 463c07b75..eda886f89 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/MetricDataDeserializer.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/MetricDataDeserializer.java @@ -6,7 +6,6 @@ package io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers; import io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.metrics.ProtoMetricsDataMapper; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest; import io.opentelemetry.sdk.metrics.data.MetricData; import java.io.IOException; @@ -30,9 +29,4 @@ public List deserialize(byte[] source) throws DeserializationExcepti throw new DeserializationException(e); } } - - @Override - public String signalType() { - return SignalTypes.metrics.name(); - } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SignalDeserializer.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SignalDeserializer.java index dd56e356e..915868288 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SignalDeserializer.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SignalDeserializer.java @@ -26,9 +26,4 @@ static SignalDeserializer ofLogs() { /** Deserializes the given byte array into a list of telemetry items. */ List deserialize(byte[] source) throws DeserializationException; - - /** Returns the name of the stored type of signal -- one of "metrics", "spans", or "logs". */ - default String signalType() { - return "unknown"; - } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SpanDataDeserializer.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SpanDataDeserializer.java index b703c3ade..b3631ca4c 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SpanDataDeserializer.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/serialization/deserializers/SpanDataDeserializer.java @@ -6,7 +6,6 @@ package io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers; import io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.spans.ProtoSpansDataMapper; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; import io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest; import io.opentelemetry.sdk.trace.data.SpanData; import java.io.IOException; @@ -30,9 +29,4 @@ public List deserialize(byte[] source) throws DeserializationException throw new DeserializationException(e); } } - - @Override - public String signalType() { - return SignalTypes.spans.name(); - } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FileSignalStorage.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FileSignalStorage.java new file mode 100644 index 000000000..743dee757 --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FileSignalStorage.java @@ -0,0 +1,92 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.internal.storage; + +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; +import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; +import io.opentelemetry.contrib.disk.buffering.storage.result.WriteResult; +import java.io.IOException; +import java.util.Collection; +import java.util.Iterator; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.logging.Level; +import java.util.logging.Logger; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import javax.annotation.concurrent.GuardedBy; + +/** Default storage implementation where items are stored in multiple protobuf files. */ +public final class FileSignalStorage implements SignalStorage { + private final Storage storage; + private final SignalSerializer serializer; + private final SignalDeserializer deserializer; + private final Logger logger = Logger.getLogger(FileSignalStorage.class.getName()); + private final AtomicBoolean isClosed = new AtomicBoolean(false); + private final Object iteratorLock = new Object(); + + @GuardedBy("iteratorLock") + @Nullable + private Iterator> iterator; + + public FileSignalStorage( + Storage storage, SignalSerializer serializer, SignalDeserializer deserializer) { + this.storage = storage; + this.serializer = serializer; + this.deserializer = deserializer; + } + + @Override + public CompletableFuture write(Collection items) { + logger.finer("Intercepting batch."); + try { + serializer.initialize(items); + if (storage.write(serializer)) { + return CompletableFuture.completedFuture(WriteResult.successful()); + } + logger.fine("Could not store batch in disk."); + return CompletableFuture.completedFuture( + WriteResult.error(new Exception("Could not store batch in disk for an unknown reason."))); + } catch (IOException e) { + logger.log( + Level.WARNING, + "An unexpected error happened while attempting to write the data in disk.", + e); + return CompletableFuture.completedFuture(WriteResult.error(e)); + } finally { + serializer.reset(); + } + } + + @Override + public CompletableFuture clear() { + try { + storage.clear(); + return CompletableFuture.completedFuture(WriteResult.successful()); + } catch (IOException e) { + return CompletableFuture.completedFuture(WriteResult.error(e)); + } + } + + @Override + public void close() throws IOException { + if (isClosed.compareAndSet(false, true)) { + storage.close(); + } + } + + @Nonnull + @Override + public Iterator> iterator() { + synchronized (iteratorLock) { + if (iterator == null) { + iterator = new StorageIterator<>(storage, deserializer); + } + return iterator; + } + } +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FileSpanStorage.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FileSpanStorage.java deleted file mode 100644 index 5ba51790f..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FileSpanStorage.java +++ /dev/null @@ -1,39 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.storage; - -import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; -import io.opentelemetry.contrib.disk.buffering.storage.result.WriteResult; -import io.opentelemetry.sdk.trace.data.SpanData; -import java.util.Collection; -import java.util.Iterator; -import java.util.concurrent.CompletableFuture; -import javax.annotation.Nonnull; - -/** Default storage implementation where items are stored in multiple protobuf files. */ -public final class FileSpanStorage implements SignalStorage.Span { - - @Override - public CompletableFuture write(Collection items) { - throw new UnsupportedOperationException("For next PR"); - } - - @Override - public CompletableFuture clear() { - throw new UnsupportedOperationException("For next PR"); - } - - @Override - public void close() { - throw new UnsupportedOperationException("For next PR"); - } - - @Nonnull - @Override - public Iterator> iterator() { - throw new UnsupportedOperationException("For next PR"); - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManager.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManager.java index 1f76419ab..d93751fa1 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManager.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManager.java @@ -7,13 +7,15 @@ import static io.opentelemetry.contrib.disk.buffering.internal.storage.util.ClockBuddy.nowMillis; -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; import io.opentelemetry.contrib.disk.buffering.internal.storage.files.ReadableFile; import io.opentelemetry.contrib.disk.buffering.internal.storage.files.WritableFile; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileStorageConfiguration; import io.opentelemetry.sdk.common.Clock; import java.io.Closeable; import java.io.File; import java.io.IOException; +import java.util.ArrayList; +import java.util.List; import java.util.Objects; import javax.annotation.Nullable; import org.jetbrains.annotations.NotNull; @@ -21,11 +23,24 @@ public final class FolderManager implements Closeable { private final File folder; private final Clock clock; - private final StorageConfiguration configuration; + private final FileStorageConfiguration configuration; @Nullable private ReadableFile currentReadableFile; @Nullable private WritableFile currentWritableFile; - public FolderManager(File folder, StorageConfiguration configuration, Clock clock) { + public static FolderManager create( + File destinationDir, FileStorageConfiguration configuration, Clock clock) { + if (destinationDir.isFile()) { + throw new IllegalArgumentException("destinationDir must be a directory"); + } + if (!destinationDir.exists()) { + if (!destinationDir.mkdirs()) { + throw new IllegalStateException("Could not create dir: " + destinationDir); + } + } + return new FolderManager(destinationDir, configuration, clock); + } + + public FolderManager(File folder, FileStorageConfiguration configuration, Clock clock) { this.folder = folder; this.configuration = configuration; this.clock = clock; @@ -33,12 +48,7 @@ public FolderManager(File folder, StorageConfiguration configuration, Clock cloc @Override public void close() throws IOException { - if (currentReadableFile != null) { - currentReadableFile.close(); - } - if (currentWritableFile != null) { - currentWritableFile.close(); - } + closeCurrentFiles(); } @Nullable @@ -68,6 +78,21 @@ public synchronized WritableFile createWritableFile() throws IOException { return currentWritableFile; } + public synchronized void clear() throws IOException { + closeCurrentFiles(); + List undeletedFiles = new ArrayList<>(); + + for (File file : Objects.requireNonNull(folder.listFiles())) { + if (!file.delete()) { + undeletedFiles.add(file); + } + } + + if (!undeletedFiles.isEmpty()) { + throw new IOException("Could not delete files " + undeletedFiles); + } + } + @Nullable private File findReadableFile() throws IOException { long currentTime = nowMillis(clock); @@ -152,4 +177,13 @@ private boolean hasExpiredForReading(long systemCurrentTimeMillis, long createdT return systemCurrentTimeMillis > (createdTimeInMillis + configuration.getMaxFileAgeForReadMillis()); } + + private synchronized void closeCurrentFiles() throws IOException { + if (currentReadableFile != null) { + currentReadableFile.close(); + } + if (currentWritableFile != null) { + currentWritableFile.close(); + } + } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/Storage.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/Storage.java index 86b5284ca..6b2ee05f7 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/Storage.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/Storage.java @@ -7,44 +7,34 @@ import static java.util.logging.Level.WARNING; -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.DeserializationException; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; import io.opentelemetry.contrib.disk.buffering.internal.storage.files.ReadableFile; import io.opentelemetry.contrib.disk.buffering.internal.storage.files.WritableFile; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.ProcessResult; import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.ReadableResult; import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.WritableResult; -import io.opentelemetry.contrib.disk.buffering.internal.utils.DebugLogger; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; import java.io.Closeable; import java.io.IOException; +import java.util.Collection; +import java.util.List; +import java.util.Objects; import java.util.concurrent.atomic.AtomicBoolean; -import java.util.function.Function; +import java.util.concurrent.atomic.AtomicReference; import java.util.logging.Logger; import javax.annotation.Nullable; -public final class Storage implements Closeable { +public final class Storage implements Closeable { private static final int MAX_ATTEMPTS = 3; - private final DebugLogger logger; + private final Logger logger = Logger.getLogger(Storage.class.getName()); private final FolderManager folderManager; - private final boolean debugEnabled; private final AtomicBoolean isClosed = new AtomicBoolean(false); - @Nullable private WritableFile writableFile; - @Nullable private ReadableFile readableFile; + private final AtomicBoolean activeReadResultAvailable = new AtomicBoolean(false); + private final AtomicReference writableFileRef = new AtomicReference<>(); + private final AtomicReference readableFileRef = new AtomicReference<>(); - public Storage(FolderManager folderManager, boolean debugEnabled) { + public Storage(FolderManager folderManager) { this.folderManager = folderManager; - this.logger = - DebugLogger.wrap(Logger.getLogger(FromDiskExporterImpl.class.getName()), debugEnabled); - this.debugEnabled = debugEnabled; - } - - public static StorageBuilder builder(SignalTypes types) { - return new StorageBuilder(types); - } - - public boolean isDebugEnabled() { - return debugEnabled; } /** @@ -53,91 +43,151 @@ public boolean isDebugEnabled() { * @param marshaler - The data that would be appended to the file. * @throws IOException If an unexpected error happens. */ - public boolean write(SignalSerializer marshaler) throws IOException { + public boolean write(SignalSerializer marshaler) throws IOException { return write(marshaler, 1); } - private boolean write(SignalSerializer marshaler, int attemptNumber) throws IOException { + private boolean write(SignalSerializer marshaler, int attemptNumber) throws IOException { if (isClosed.get()) { - logger.log("Refusing to write to storage after being closed."); + logger.fine("Refusing to write to storage after being closed."); return false; } if (attemptNumber > MAX_ATTEMPTS) { - logger.log("Max number of attempts to write buffered data exceeded.", WARNING); + logger.log(WARNING, "Max number of attempts to write buffered data exceeded."); return false; } + WritableFile writableFile = writableFileRef.get(); if (writableFile == null) { writableFile = folderManager.createWritableFile(); - logger.log("Created new writableFile: " + writableFile); + writableFileRef.set(writableFile); + logger.finer("Created new writableFile: " + writableFile); } WritableResult result = writableFile.append(marshaler); if (result != WritableResult.SUCCEEDED) { // Retry with new file - writableFile = null; + writableFileRef.set(null); return write(marshaler, ++attemptNumber); } return true; } public void flush() throws IOException { + WritableFile writableFile = writableFileRef.get(); if (writableFile != null) { writableFile.flush(); } else { - logger.log("No writable file to flush."); + logger.info("No writable file to flush."); } } /** * Attempts to read an item from a ready-to-read file. * - * @param processing Is passed over to {@link ReadableFile#readAndProcess(Function)}. * @throws IOException If an unexpected error happens. */ - public ReadableResult readAndProcess(Function processing) - throws IOException { - return readAndProcess(processing, 1); + @Nullable + public ReadableResult readNext(SignalDeserializer deserializer) throws IOException { + if (activeReadResultAvailable.get()) { + throw new IllegalStateException( + "You must close any previous ReadableResult before requesting a new one"); + } + return doReadNext(deserializer, 1); } - private ReadableResult readAndProcess( - Function processing, int attemptNumber) throws IOException { + @Nullable + private ReadableResult doReadNext(SignalDeserializer deserializer, int attemptNumber) + throws IOException { if (isClosed.get()) { - logger.log("Refusing to read from storage after being closed."); - return ReadableResult.FAILED; + logger.fine("Refusing to read from storage after being closed."); + return null; } if (attemptNumber > MAX_ATTEMPTS) { - logger.log("Maximum number of attempts to read and process buffered data exceeded.", WARNING); - return ReadableResult.FAILED; + logger.log(WARNING, "Maximum number of attempts to read buffered data exceeded."); + return null; } + ReadableFile readableFile = readableFileRef.get(); if (readableFile == null) { - logger.log("Obtaining a new readableFile from the folderManager."); + logger.finer("Obtaining a new readableFile from the folderManager."); readableFile = folderManager.getReadableFile(); + readableFileRef.set(readableFile); if (readableFile == null) { - logger.log("Unable to get or create readable file."); - return ReadableResult.FAILED; + logger.fine("Unable to get or create readable file."); + return null; } } - logger.log("Attempting to read data from " + readableFile); - ReadableResult result = readableFile.readAndProcess(processing); - switch (result) { - case SUCCEEDED: - case TRY_LATER: - return result; - default: - // Retry with new file - readableFile = null; - return readAndProcess(processing, ++attemptNumber); + + logger.finer("Attempting to read data from " + readableFile); + byte[] result = readableFile.readNext(); + if (result != null) { + try { + List items = deserializer.deserialize(result); + activeReadResultAvailable.set(true); + return new FileReadResult(items, readableFile); + } catch (DeserializationException e) { + // Data corrupted, clear file. + readableFile.clear(); + } } + + // Retry with new file + readableFileRef.set(null); + return doReadNext(deserializer, ++attemptNumber); + } + + public void clear() throws IOException { + folderManager.clear(); + } + + public boolean isClosed() { + return isClosed.get(); } @Override public void close() throws IOException { - logger.log("Closing disk buffering storage."); + logger.fine("Closing disk buffering storage."); if (isClosed.compareAndSet(false, true)) { - if (writableFile != null) { - writableFile.close(); + folderManager.close(); + writableFileRef.set(null); + readableFileRef.set(null); + } + } + + class FileReadResult implements ReadableResult { + private final Collection content; + private final AtomicBoolean itemDeleted = new AtomicBoolean(false); + private final AtomicBoolean closed = new AtomicBoolean(false); + private final AtomicReference readableFile = new AtomicReference<>(); + + FileReadResult(Collection content, ReadableFile readableFile) { + this.content = content; + this.readableFile.set(readableFile); + } + + @Override + public Collection getContent() { + return content; + } + + @Override + public void delete() throws IOException { + if (closed.get()) { + return; } - if (readableFile != null) { - readableFile.close(); + if (itemDeleted.compareAndSet(false, true)) { + try { + Objects.requireNonNull(readableFile.get()).removeTopItem(); + } catch (IOException e) { + itemDeleted.set(false); + throw e; + } + } + } + + @Override + public void close() throws IOException { + if (closed.compareAndSet(false, true)) { + activeReadResultAvailable.set(false); + readableFile.set(null); } } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageBuilder.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageBuilder.java deleted file mode 100644 index ebea37171..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageBuilder.java +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.storage; - -import static java.util.logging.Level.INFO; - -import com.google.errorprone.annotations.CanIgnoreReturnValue; -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; -import io.opentelemetry.sdk.common.Clock; -import java.io.File; -import java.io.IOException; -import java.util.logging.Logger; - -public class StorageBuilder { - - private static final Logger logger = Logger.getLogger(StorageBuilder.class.getName()); - - private final String folderName; - private StorageConfiguration configuration = StorageConfiguration.getDefault(new File(".")); - private Clock clock = Clock.getDefault(); - - StorageBuilder(SignalTypes types) { - folderName = types.name(); - } - - @CanIgnoreReturnValue - public StorageBuilder setStorageConfiguration(StorageConfiguration configuration) { - validateConfiguration(configuration); - this.configuration = configuration; - return this; - } - - @CanIgnoreReturnValue - public StorageBuilder setStorageClock(Clock clock) { - this.clock = clock; - return this; - } - - public Storage build() throws IOException { - File folder = ensureSubdir(configuration.getRootDir(), folderName); - FolderManager folderManager = new FolderManager(folder, configuration, clock); - if (configuration.isDebugEnabled()) { - logger.log(INFO, "Building storage with configuration => " + configuration); - } - return new Storage(folderManager, configuration.isDebugEnabled()); - } - - private static File ensureSubdir(File rootDir, String child) throws IOException { - File subdir = new File(rootDir, child); - if (subdir.exists() || subdir.mkdirs()) { - return subdir; - } - throw new IOException("Could not create the subdir: '" + child + "' inside: " + rootDir); - } - - private static void validateConfiguration(StorageConfiguration configuration) { - if (configuration.getMinFileAgeForReadMillis() <= configuration.getMaxFileAgeForWriteMillis()) { - throw new IllegalArgumentException( - "The configured max file age for writing must be lower than the configured min file age for reading"); - } - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageIterator.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageIterator.java new file mode 100644 index 000000000..871238b03 --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageIterator.java @@ -0,0 +1,90 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.internal.storage; + +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; +import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.ReadableResult; +import java.io.IOException; +import java.util.Collection; +import java.util.Iterator; +import java.util.Objects; +import java.util.logging.Level; +import java.util.logging.Logger; +import javax.annotation.Nullable; +import javax.annotation.concurrent.GuardedBy; + +final class StorageIterator implements Iterator> { + private final Storage storage; + private final SignalDeserializer deserializer; + private final Logger logger = Logger.getLogger(StorageIterator.class.getName()); + + @GuardedBy("this") + @Nullable + private ReadableResult currentResult; + + @GuardedBy("this") + private boolean currentResultConsumed = false; + + StorageIterator(Storage storage, SignalDeserializer deserializer) { + this.storage = storage; + this.deserializer = deserializer; + } + + @Override + public synchronized boolean hasNext() { + if (storage.isClosed()) { + return false; + } + return findNext(); + } + + @Override + @Nullable + public synchronized Collection next() { + if (storage.isClosed()) { + return null; + } + if (findNext()) { + currentResultConsumed = true; + return Objects.requireNonNull(currentResult).getContent(); + } + return null; + } + + @Override + public synchronized void remove() { + if (currentResult != null) { + try { + currentResult.delete(); + } catch (IOException e) { + logger.log(Level.SEVERE, "Error deleting stored item", e); + } + } + } + + private synchronized boolean findNext() { + try { + if (currentResult != null) { + if (!currentResultConsumed) { + return true; + } + currentResult.delete(); + currentResult.close(); + currentResult = null; + } + + currentResultConsumed = false; + ReadableResult result = storage.readNext(deserializer); + if (result != null) { + currentResult = result; + return true; + } + } catch (IOException e) { + logger.log(Level.SEVERE, "Error reading from storage", e); + } + return false; + } +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFile.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFile.java index 710e192bb..59429d187 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFile.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFile.java @@ -7,20 +7,17 @@ import static io.opentelemetry.contrib.disk.buffering.internal.storage.util.ClockBuddy.nowMillis; -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.DelimitedProtoStreamReader; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.ProcessResult; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.ReadResult; import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.StreamReader; import io.opentelemetry.contrib.disk.buffering.internal.storage.files.utils.FileStream; -import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.ReadableResult; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileStorageConfiguration; import io.opentelemetry.sdk.common.Clock; import java.io.File; import java.io.IOException; import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Function; +import javax.annotation.Nonnull; import javax.annotation.Nullable; -import org.jetbrains.annotations.NotNull; /** * Reads from a file and updates it in parallel in order to avoid re-reading the same items later. @@ -32,16 +29,15 @@ *

More information on the overall storage process in the CONTRIBUTING.md file. */ public final class ReadableFile implements FileOperations { - @NotNull private final File file; + @Nonnull private final File file; private final FileStream fileStream; private final StreamReader reader; private final Clock clock; private final long expireTimeMillis; private final AtomicBoolean isClosed = new AtomicBoolean(false); - @Nullable private ReadResult unconsumedResult; public ReadableFile( - File file, long createdTimeMillis, Clock clock, StorageConfiguration configuration) + File file, long createdTimeMillis, Clock clock, FileStorageConfiguration configuration) throws IOException { this( file, @@ -52,10 +48,10 @@ public ReadableFile( } public ReadableFile( - @NotNull File file, + @Nonnull File file, long createdTimeMillis, Clock clock, - StorageConfiguration configuration, + FileStorageConfiguration configuration, StreamReader.Factory readerFactory) throws IOException { this.file = file; @@ -68,49 +64,22 @@ public ReadableFile( /** * Reads the next line available in the file and provides it to a {@link Function processing} * which will determine whether to remove the provided line or not. - * - * @param processing - A function that receives the line that has been read and returns a boolean. - * If the processing function returns TRUE, then the provided line will be deleted from the - * source file. If the function returns FALSE, no changes will be applied to the source file. */ - public synchronized ReadableResult readAndProcess(Function processing) - throws IOException { + @Nullable + public synchronized byte[] readNext() throws IOException { if (isClosed.get()) { - return ReadableResult.FAILED; + return null; } if (hasExpired()) { close(); - return ReadableResult.FAILED; - } - ReadResult read = readNextItem(); - if (read == null) { - cleanUp(); - return ReadableResult.FAILED; + return null; } - switch (processing.apply(read.content)) { - case SUCCEEDED: - unconsumedResult = null; - fileStream.truncateTop(); - if (fileStream.size() == 0) { - cleanUp(); - } - return ReadableResult.SUCCEEDED; - case TRY_LATER: - unconsumedResult = read; - return ReadableResult.TRY_LATER; - case CONTENT_INVALID: - cleanUp(); - return ReadableResult.FAILED; - } - return ReadableResult.FAILED; - } - - @Nullable - private ReadResult readNextItem() throws IOException { - if (unconsumedResult != null) { - return unconsumedResult; + byte[] resultBytes = reader.readNext(); + if (resultBytes == null) { + clear(); + return null; } - return reader.readNext(); + return resultBytes; } @Override @@ -123,23 +92,29 @@ public synchronized boolean isClosed() { return isClosed.get(); } - @NotNull + @Nonnull @Override public File getFile() { return file; } - private void cleanUp() throws IOException { + public synchronized void clear() throws IOException { close(); if (!file.delete()) { throw new IOException("Could not delete file: " + file); } } + public synchronized void removeTopItem() throws IOException { + fileStream.truncateTop(); + if (fileStream.size() == 0) { + clear(); + } + } + @Override public synchronized void close() throws IOException { if (isClosed.compareAndSet(false, true)) { - unconsumedResult = null; reader.close(); } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFile.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFile.java index ce4e87ddf..325d30856 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFile.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFile.java @@ -7,9 +7,9 @@ import static io.opentelemetry.contrib.disk.buffering.internal.storage.util.ClockBuddy.nowMillis; -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.WritableResult; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileStorageConfiguration; import io.opentelemetry.sdk.common.Clock; import java.io.File; import java.io.FileOutputStream; @@ -21,7 +21,7 @@ public final class WritableFile implements FileOperations { private final File file; - private final StorageConfiguration configuration; + private final FileStorageConfiguration configuration; private final Clock clock; private final long expireTimeMillis; private final OutputStream out; @@ -29,7 +29,7 @@ public final class WritableFile implements FileOperations { private int size; public WritableFile( - File file, long createdTimeMillis, StorageConfiguration configuration, Clock clock) + File file, long createdTimeMillis, FileStorageConfiguration configuration, Clock clock) throws IOException { this.file = file; this.configuration = configuration; diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/DelimitedProtoStreamReader.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/DelimitedProtoStreamReader.java index 60a8e4f45..d638c118b 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/DelimitedProtoStreamReader.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/DelimitedProtoStreamReader.java @@ -19,16 +19,22 @@ public DelimitedProtoStreamReader(InputStream inputStream) { @Override @Nullable - public ReadResult readNext() throws IOException { + public byte[] readNext() throws IOException { int itemSize = getNextItemSize(); if (itemSize < 1) { return null; } byte[] bytes = new byte[itemSize]; - if (inputStream.read(bytes) < 0) { - return null; + int offset = 0; + int readCt; + do { + readCt = inputStream.read(bytes, offset, itemSize - offset); + offset += readCt; + } while (readCt != -1 && offset < itemSize); + if (offset != itemSize) { + return null; // unable to read the whole item correctly } - return new ReadResult(bytes); + return bytes; } private int getNextItemSize() { diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/ProcessResult.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/ProcessResult.java deleted file mode 100644 index 696d98d2c..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/ProcessResult.java +++ /dev/null @@ -1,13 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader; - -/** Result of processing the contents of a file. */ -public enum ProcessResult { - SUCCEEDED, - TRY_LATER, - CONTENT_INVALID -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/ReadResult.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/ReadResult.java deleted file mode 100644 index a9f5d1116..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/ReadResult.java +++ /dev/null @@ -1,15 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader; - -public final class ReadResult { - /** The consumable data. */ - public final byte[] content; - - public ReadResult(byte[] content) { - this.content = content; - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/StreamReader.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/StreamReader.java index 447315f1e..925422f67 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/StreamReader.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/reader/StreamReader.java @@ -12,7 +12,7 @@ public interface StreamReader extends Closeable { @Nullable - ReadResult readNext() throws IOException; + byte[] readNext() throws IOException; interface Factory { StreamReader create(InputStream stream); diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/responses/ReadableResult.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/responses/ReadableResult.java index 295bc2289..1e2877c7b 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/responses/ReadableResult.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/storage/responses/ReadableResult.java @@ -5,8 +5,14 @@ package io.opentelemetry.contrib.disk.buffering.internal.storage.responses; -public enum ReadableResult { - SUCCEEDED, - FAILED, - TRY_LATER +import java.io.Closeable; +import java.io.IOException; +import java.util.Collection; + +public interface ReadableResult extends Closeable { + /** The consumable data. */ + Collection getContent(); + + /** Delete the items provided in {@link #getContent()} */ + void delete() throws IOException; } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/utils/DebugLogger.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/utils/DebugLogger.java deleted file mode 100644 index b0bb67624..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/utils/DebugLogger.java +++ /dev/null @@ -1,41 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.utils; - -import static java.util.logging.Level.INFO; - -import java.util.logging.Level; -import java.util.logging.Logger; - -public class DebugLogger { - private final Logger logger; - private final boolean debugEnabled; - - private DebugLogger(Logger logger, boolean debugEnabled) { - this.logger = logger; - this.debugEnabled = debugEnabled; - } - - public static DebugLogger wrap(Logger logger, boolean debugEnabled) { - return new DebugLogger(logger, debugEnabled); - } - - public void log(String msg) { - log(msg, INFO); - } - - public void log(String msg, Level level) { - if (debugEnabled) { - logger.log(level, msg); - } - } - - public void log(String msg, Level level, Throwable e) { - if (debugEnabled) { - logger.log(level, msg, e); - } - } -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/utils/SignalTypes.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/utils/SignalTypes.java deleted file mode 100644 index c0a7f5765..000000000 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/internal/utils/SignalTypes.java +++ /dev/null @@ -1,12 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.utils; - -public enum SignalTypes { - metrics, - spans, - logs -} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileLogRecordStorage.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileLogRecordStorage.java new file mode 100644 index 000000000..71d5d884b --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileLogRecordStorage.java @@ -0,0 +1,63 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.storage.impl; + +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; +import io.opentelemetry.contrib.disk.buffering.internal.storage.FileSignalStorage; +import io.opentelemetry.contrib.disk.buffering.internal.storage.FolderManager; +import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; +import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; +import io.opentelemetry.contrib.disk.buffering.storage.result.WriteResult; +import io.opentelemetry.sdk.common.Clock; +import io.opentelemetry.sdk.logs.data.LogRecordData; +import java.io.File; +import java.io.IOException; +import java.util.Collection; +import java.util.Iterator; +import java.util.concurrent.CompletableFuture; +import javax.annotation.Nonnull; + +public final class FileLogRecordStorage implements SignalStorage.LogRecord { + private final FileSignalStorage fileSignalStorage; + + public static FileLogRecordStorage create(File destinationDir) { + return create(destinationDir, FileStorageConfiguration.getDefault()); + } + + public static FileLogRecordStorage create( + File destinationDir, FileStorageConfiguration configuration) { + Storage storage = + new Storage<>(FolderManager.create(destinationDir, configuration, Clock.getDefault())); + return new FileLogRecordStorage( + new FileSignalStorage<>(storage, SignalSerializer.ofLogs(), SignalDeserializer.ofLogs())); + } + + private FileLogRecordStorage(FileSignalStorage fileSignalStorage) { + this.fileSignalStorage = fileSignalStorage; + } + + @Override + public CompletableFuture write(Collection items) { + return fileSignalStorage.write(items); + } + + @Override + public CompletableFuture clear() { + return fileSignalStorage.clear(); + } + + @Override + public void close() throws IOException { + fileSignalStorage.close(); + } + + @Nonnull + @Override + public Iterator> iterator() { + return fileSignalStorage.iterator(); + } +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileMetricStorage.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileMetricStorage.java new file mode 100644 index 000000000..8f8b41508 --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileMetricStorage.java @@ -0,0 +1,64 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.storage.impl; + +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; +import io.opentelemetry.contrib.disk.buffering.internal.storage.FileSignalStorage; +import io.opentelemetry.contrib.disk.buffering.internal.storage.FolderManager; +import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; +import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; +import io.opentelemetry.contrib.disk.buffering.storage.result.WriteResult; +import io.opentelemetry.sdk.common.Clock; +import io.opentelemetry.sdk.metrics.data.MetricData; +import java.io.File; +import java.io.IOException; +import java.util.Collection; +import java.util.Iterator; +import java.util.concurrent.CompletableFuture; +import javax.annotation.Nonnull; + +public final class FileMetricStorage implements SignalStorage.Metric { + private final FileSignalStorage fileSignalStorage; + + public static FileMetricStorage create(File destinationDir) { + return create(destinationDir, FileStorageConfiguration.getDefault()); + } + + public static FileMetricStorage create( + File destinationDir, FileStorageConfiguration configuration) { + Storage storage = + new Storage<>(FolderManager.create(destinationDir, configuration, Clock.getDefault())); + return new FileMetricStorage( + new FileSignalStorage<>( + storage, SignalSerializer.ofMetrics(), SignalDeserializer.ofMetrics())); + } + + private FileMetricStorage(FileSignalStorage fileSignalStorage) { + this.fileSignalStorage = fileSignalStorage; + } + + @Override + public CompletableFuture write(Collection items) { + return fileSignalStorage.write(items); + } + + @Override + public CompletableFuture clear() { + return fileSignalStorage.clear(); + } + + @Override + public void close() throws IOException { + fileSignalStorage.close(); + } + + @Nonnull + @Override + public Iterator> iterator() { + return fileSignalStorage.iterator(); + } +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileSpanStorage.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileSpanStorage.java new file mode 100644 index 000000000..b8cfa8996 --- /dev/null +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileSpanStorage.java @@ -0,0 +1,63 @@ +/* + * Copyright The OpenTelemetry Authors + * SPDX-License-Identifier: Apache-2.0 + */ + +package io.opentelemetry.contrib.disk.buffering.storage.impl; + +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; +import io.opentelemetry.contrib.disk.buffering.internal.storage.FileSignalStorage; +import io.opentelemetry.contrib.disk.buffering.internal.storage.FolderManager; +import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; +import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; +import io.opentelemetry.contrib.disk.buffering.storage.result.WriteResult; +import io.opentelemetry.sdk.common.Clock; +import io.opentelemetry.sdk.trace.data.SpanData; +import java.io.File; +import java.io.IOException; +import java.util.Collection; +import java.util.Iterator; +import java.util.concurrent.CompletableFuture; +import javax.annotation.Nonnull; + +public final class FileSpanStorage implements SignalStorage.Span { + private final FileSignalStorage fileSignalStorage; + + public static FileSpanStorage create(File destinationDir) { + return create(destinationDir, FileStorageConfiguration.getDefault()); + } + + public static FileSpanStorage create( + File destinationDir, FileStorageConfiguration configuration) { + Storage storage = + new Storage<>(FolderManager.create(destinationDir, configuration, Clock.getDefault())); + return new FileSpanStorage( + new FileSignalStorage<>(storage, SignalSerializer.ofSpans(), SignalDeserializer.ofSpans())); + } + + private FileSpanStorage(FileSignalStorage fileSignalStorage) { + this.fileSignalStorage = fileSignalStorage; + } + + @Override + public CompletableFuture write(Collection items) { + return fileSignalStorage.write(items); + } + + @Override + public CompletableFuture clear() { + return fileSignalStorage.clear(); + } + + @Override + public void close() throws IOException { + fileSignalStorage.close(); + } + + @Nonnull + @Override + public Iterator> iterator() { + return fileSignalStorage.iterator(); + } +} diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/config/StorageConfiguration.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileStorageConfiguration.java similarity index 72% rename from disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/config/StorageConfiguration.java rename to disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileStorageConfiguration.java index e8a7f0bd0..0a34c12b4 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/config/StorageConfiguration.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/impl/FileStorageConfiguration.java @@ -3,23 +3,16 @@ * SPDX-License-Identifier: Apache-2.0 */ -package io.opentelemetry.contrib.disk.buffering.config; +package io.opentelemetry.contrib.disk.buffering.storage.impl; import static java.util.concurrent.TimeUnit.HOURS; import static java.util.concurrent.TimeUnit.SECONDS; import com.google.auto.value.AutoValue; -import java.io.File; /** Defines how the storage should be managed. */ @AutoValue -public abstract class StorageConfiguration { - - /** The root storage location for buffered telemetry. */ - public abstract File getRootDir(); - - /** Returns true if the storage has been configured with debug verbosity enabled. */ - public abstract boolean isDebugEnabled(); +public abstract class FileStorageConfiguration { /** The max amount of time a file can receive new data. */ public abstract long getMaxFileAgeForWriteMillis(); @@ -50,18 +43,17 @@ public abstract class StorageConfiguration { */ public abstract int getMaxFolderSize(); - public static StorageConfiguration getDefault(File rootDir) { - return builder().setRootDir(rootDir).build(); + public static FileStorageConfiguration getDefault() { + return builder().build(); } public static Builder builder() { - return new AutoValue_StorageConfiguration.Builder() + return new AutoValue_FileStorageConfiguration.Builder() .setMaxFileSize(1024 * 1024) // 1MB .setMaxFolderSize(10 * 1024 * 1024) // 10MB .setMaxFileAgeForWriteMillis(SECONDS.toMillis(30)) .setMinFileAgeForReadMillis(SECONDS.toMillis(33)) - .setMaxFileAgeForReadMillis(HOURS.toMillis(18)) - .setDebugEnabled(false); + .setMaxFileAgeForReadMillis(HOURS.toMillis(18)); } @AutoValue.Builder @@ -76,10 +68,16 @@ public abstract static class Builder { public abstract Builder setMaxFolderSize(int value); - public abstract Builder setRootDir(File rootDir); - - public abstract Builder setDebugEnabled(boolean debugEnabled); - - public abstract StorageConfiguration build(); + abstract FileStorageConfiguration autoBuild(); + + public final FileStorageConfiguration build() { + FileStorageConfiguration configuration = autoBuild(); + if (configuration.getMinFileAgeForReadMillis() + <= configuration.getMaxFileAgeForWriteMillis()) { + throw new IllegalArgumentException( + "The configured max file age for writing must be lower than the configured min file age for reading"); + } + return configuration; + } } } diff --git a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/result/WriteResult.java b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/result/WriteResult.java index 8e4534a55..85c08ed37 100644 --- a/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/result/WriteResult.java +++ b/disk-buffering/src/main/java/io/opentelemetry/contrib/disk/buffering/storage/result/WriteResult.java @@ -25,7 +25,11 @@ public interface WriteResult { @Nullable Throwable getError(); - static WriteResult create(boolean successful, @Nullable Throwable error) { - return new DefaultWriteResult(successful, error); + static WriteResult successful() { + return new DefaultWriteResult(/* successful= */ true, null); + } + + static WriteResult error(@Nullable Throwable t) { + return new DefaultWriteResult(/* successful= */ false, t); } } diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/FromDiskExporterImplTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/FromDiskExporterImplTest.java deleted file mode 100644 index e7995c675..000000000 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/FromDiskExporterImplTest.java +++ /dev/null @@ -1,114 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.MIN_FILE_AGE_FOR_READ_MILLIS; -import static java.util.Collections.singletonList; -import static java.util.concurrent.TimeUnit.MILLISECONDS; -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.ArgumentMatchers.any; -import static org.mockito.Mockito.doThrow; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.DeserializationException; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.TestData; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; -import io.opentelemetry.sdk.common.Clock; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.trace.data.SpanData; -import io.opentelemetry.sdk.trace.export.SpanExporter; -import java.io.File; -import java.io.IOException; -import java.nio.file.Files; -import java.util.Collections; -import java.util.List; -import java.util.concurrent.TimeUnit; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; - -@SuppressWarnings("unchecked") -class FromDiskExporterImplTest { - private SpanExporter wrapped; - private SignalDeserializer deserializer; - private Clock clock; - private FromDiskExporterImpl exporter; - private final List deserializedData = Collections.emptyList(); - @TempDir File rootDir; - private static final String STORAGE_FOLDER_NAME = SignalTypes.spans.name(); - - @BeforeEach - void setUp() throws IOException { - clock = createClockMock(); - - setUpSerializer(); - wrapped = mock(); - exporter = - FromDiskExporterImpl.builder( - TestData.getStorage(rootDir, SignalTypes.spans, clock)) - .setDeserializer(deserializer) - .setExportFunction(wrapped::export) - .build(); - } - - @Test - void whenExportingStoredBatch_withAvailableData_andSuccessfullyProcessed_returnTrue() - throws IOException { - when(wrapped.export(deserializedData)).thenReturn(CompletableResultCode.ofSuccess()); - - createDummyFile(); - when(clock.now()).thenReturn(MILLISECONDS.toNanos(1000L + MIN_FILE_AGE_FOR_READ_MILLIS)); - - assertThat(exporter.exportStoredBatch(1, TimeUnit.SECONDS)).isTrue(); - } - - @Test - void whenExportingStoredBatch_withAvailableData_andUnsuccessfullyProcessed_returnFalse() - throws IOException { - when(wrapped.export(deserializedData)).thenReturn(CompletableResultCode.ofSuccess()); - - createDummyFile(); - when(clock.now()).thenReturn(1000L + MIN_FILE_AGE_FOR_READ_MILLIS); - - assertThat(exporter.exportStoredBatch(1, TimeUnit.SECONDS)).isFalse(); - } - - @Test - void whenExportingStoredBatch_withNoAvailableData_returnFalse() throws IOException { - assertThat(exporter.exportStoredBatch(1, TimeUnit.SECONDS)).isFalse(); - } - - @Test - void verifyStorageFolderIsCreated() { - assertThat(new File(rootDir, STORAGE_FOLDER_NAME).exists()).isTrue(); - } - - @Test - void whenDeserializationFails_returnFalse() throws IOException { - doThrow(DeserializationException.class).when(deserializer).deserialize(any()); - - assertThat(exporter.exportStoredBatch(1, TimeUnit.SECONDS)).isFalse(); - } - - private void createDummyFile() throws IOException { - File file = new File(rootDir, STORAGE_FOLDER_NAME + "/" + 1000L); - Files.write(file.toPath(), singletonList("First line")); - } - - private void setUpSerializer() throws DeserializationException { - deserializer = mock(); - when(deserializer.deserialize(any())).thenReturn(deserializedData); - } - - private static Clock createClockMock() { - Clock mock = mock(); - when(mock.now()).thenReturn(MILLISECONDS.toNanos(1000L)); - return mock; - } -} diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/IntegrationTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/IntegrationTest.java index ee4deb57a..d30466430 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/IntegrationTest.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/IntegrationTest.java @@ -5,25 +5,25 @@ package io.opentelemetry.contrib.disk.buffering; -import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static java.lang.Thread.sleep; import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; +import static org.mockito.ArgumentMatchers.anyCollection; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoMoreInteractions; import io.opentelemetry.api.logs.Logger; import io.opentelemetry.api.metrics.Meter; import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.Tracer; -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterBuilder; -import io.opentelemetry.contrib.disk.buffering.internal.exporter.FromDiskExporterImpl; -import io.opentelemetry.contrib.disk.buffering.internal.exporter.ToDiskExporter; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; -import io.opentelemetry.sdk.common.Clock; -import io.opentelemetry.sdk.common.CompletableResultCode; +import io.opentelemetry.contrib.disk.buffering.exporters.LogRecordToDiskExporter; +import io.opentelemetry.contrib.disk.buffering.exporters.MetricToDiskExporter; +import io.opentelemetry.contrib.disk.buffering.exporters.SpanToDiskExporter; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.ExporterCallback; +import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileLogRecordStorage; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileMetricStorage; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileSpanStorage; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileStorageConfiguration; import io.opentelemetry.sdk.logs.SdkLoggerProvider; import io.opentelemetry.sdk.logs.data.LogRecordData; import io.opentelemetry.sdk.logs.export.LogRecordExporter; @@ -32,159 +32,124 @@ import io.opentelemetry.sdk.metrics.data.MetricData; import io.opentelemetry.sdk.metrics.export.MetricExporter; import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader; -import io.opentelemetry.sdk.testing.exporter.InMemoryLogRecordExporter; -import io.opentelemetry.sdk.testing.exporter.InMemoryMetricExporter; -import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter; import io.opentelemetry.sdk.trace.SdkTracerProvider; import io.opentelemetry.sdk.trace.data.SpanData; import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor; import io.opentelemetry.sdk.trace.export.SpanExporter; import java.io.File; import java.io.IOException; -import java.util.Collection; -import java.util.concurrent.TimeUnit; -import java.util.function.Function; -import java.util.function.Supplier; -import org.jetbrains.annotations.NotNull; +import java.util.ArrayList; +import java.util.List; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; import org.junit.jupiter.api.io.TempDir; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +@ExtendWith(MockitoExtension.class) class IntegrationTest { - private InMemorySpanExporter memorySpanExporter; private Tracer tracer; - private InMemoryMetricExporter memoryMetricExporter; private SdkMeterProvider meterProvider; private Meter meter; - private InMemoryLogRecordExporter memoryLogRecordExporter; private Logger logger; - private Clock clock; - @TempDir File rootDir; - private static final long INITIAL_TIME_IN_MILLIS = 1000; - private static final long NOW_NANOS = MILLISECONDS.toNanos(INITIAL_TIME_IN_MILLIS); - private StorageConfiguration storageConfig; - private Storage spanStorage; + private SignalStorage.Span spanStorage; + private SignalStorage.LogRecord logStorage; + private SignalStorage.Metric metricStorage; + private SpanToDiskExporter spanToDiskExporter; + private MetricToDiskExporter metricToDiskExporter; + private LogRecordToDiskExporter logToDiskExporter; + @Mock private ExporterCallback spanCallback; + @Mock private ExporterCallback logCallback; + @Mock private ExporterCallback metricCallback; + @TempDir private File rootDir; + private static final long DELAY_BEFORE_READING_MILLIS = 500; @BeforeEach - void setUp() throws IOException { - clock = mock(); - storageConfig = StorageConfiguration.getDefault(rootDir); - spanStorage = - Storage.builder(SignalTypes.spans) - .setStorageConfiguration(storageConfig) - .setStorageClock(clock) + void setUp() { + FileStorageConfiguration storageConfig = + FileStorageConfiguration.builder() + .setMaxFileAgeForWriteMillis(DELAY_BEFORE_READING_MILLIS - 1) + .setMinFileAgeForReadMillis(DELAY_BEFORE_READING_MILLIS) .build(); - when(clock.now()).thenReturn(NOW_NANOS); - // Setting up spans - memorySpanExporter = InMemorySpanExporter.create(); - ToDiskExporter toDiskSpanExporter = - buildToDiskExporter(SignalSerializer.ofSpans(), memorySpanExporter::export); - SpanToDiskExporter spanToDiskExporter = new SpanToDiskExporter(toDiskSpanExporter); + spanStorage = FileSpanStorage.create(new File(rootDir, "spans"), storageConfig); + spanToDiskExporter = + SpanToDiskExporter.builder(spanStorage).setExporterCallback(spanCallback).build(); tracer = createTracerProvider(spanToDiskExporter).get("SpanInstrumentationScope"); // Setting up metrics - memoryMetricExporter = InMemoryMetricExporter.create(); - ToDiskExporter toDiskMetricExporter = - buildToDiskExporter(SignalSerializer.ofMetrics(), memoryMetricExporter::export); - MetricToDiskExporter metricToDiskExporter = - new MetricToDiskExporter(toDiskMetricExporter, memoryMetricExporter); + metricStorage = FileMetricStorage.create(new File(rootDir, "metrics"), storageConfig); + metricToDiskExporter = + MetricToDiskExporter.builder(metricStorage).setExporterCallback(metricCallback).build(); meterProvider = createMeterProvider(metricToDiskExporter); meter = meterProvider.get("MetricInstrumentationScope"); // Setting up logs - memoryLogRecordExporter = InMemoryLogRecordExporter.create(); - ToDiskExporter toDiskLogExporter = - buildToDiskExporter(SignalSerializer.ofLogs(), memoryLogRecordExporter::export); - LogRecordToDiskExporter logToDiskExporter = new LogRecordToDiskExporter(toDiskLogExporter); + logStorage = FileLogRecordStorage.create(new File(rootDir, "logs"), storageConfig); + logToDiskExporter = + LogRecordToDiskExporter.builder(logStorage).setExporterCallback(logCallback).build(); logger = createLoggerProvider(logToDiskExporter).get("LogInstrumentationScope"); } @AfterEach void tearDown() throws IOException { + // Closing span exporter + spanToDiskExporter.shutdown(); + verify(spanCallback).onShutdown(); + verifyNoMoreInteractions(spanCallback); + + // Closing log exporter + logToDiskExporter.shutdown(); + verify(logCallback).onShutdown(); + verifyNoMoreInteractions(spanCallback); + + // Closing metric exporter + metricToDiskExporter.shutdown(); + verify(metricCallback).onShutdown(); + verifyNoMoreInteractions(spanCallback); + + // Closing storages spanStorage.close(); - } - - @NotNull - private ToDiskExporter buildToDiskExporter( - SignalSerializer serializer, Function, CompletableResultCode> exporter) { - return ToDiskExporter.builder(spanStorage) - .setSerializer(serializer) - .setExportFunction(exporter) - .build(); - } - - @NotNull - private static FromDiskExporterImpl buildFromDiskExporter( - FromDiskExporterBuilder builder, - Function, CompletableResultCode> exportFunction, - SignalDeserializer deserializer) - throws IOException { - return builder.setExportFunction(exportFunction).setDeserializer(deserializer).build(); + logStorage.close(); + metricStorage.close(); } @Test - void verifySpansIntegration() throws IOException { + void verifyIntegration() throws InterruptedException { + // Creating span Span span = tracer.spanBuilder("Span name").startSpan(); span.end(); - FromDiskExporterImpl fromDiskExporter = - buildFromDiskExporter( - FromDiskExporterImpl.builder(spanStorage), - memorySpanExporter::export, - SignalDeserializer.ofSpans()); - assertExporter(fromDiskExporter, () -> memorySpanExporter.getFinishedSpanItems().size()); - } + verify(spanCallback).onExportSuccess(anyCollection()); + verifyNoMoreInteractions(spanCallback); - @Test - void verifyMetricsIntegration() throws IOException { - meter.counterBuilder("Counter").build().add(2); - meterProvider.forceFlush(); - - FromDiskExporterImpl fromDiskExporter = - buildFromDiskExporter( - FromDiskExporterImpl.builder(spanStorage), - memoryMetricExporter::export, - SignalDeserializer.ofMetrics()); - assertExporter(fromDiskExporter, () -> memoryMetricExporter.getFinishedMetricItems().size()); - } + // Creating log + logger.logRecordBuilder().setBody("Log body").emit(); + verify(logCallback).onExportSuccess(anyCollection()); + verifyNoMoreInteractions(spanCallback); - @Test - void verifyLogRecordsIntegration() throws IOException { - logger.logRecordBuilder().setBody("I'm a log!").emit(); - - FromDiskExporterImpl fromDiskExporter = - buildFromDiskExporter( - FromDiskExporterImpl.builder(spanStorage), - memoryLogRecordExporter::export, - SignalDeserializer.ofLogs()); - assertExporter( - fromDiskExporter, () -> memoryLogRecordExporter.getFinishedLogRecordItems().size()); - } - - private void assertExporter(FromDiskExporterImpl exporter, Supplier finishedItems) - throws IOException { - // Verify no data has been received in the original exporter until this point. - assertThat(finishedItems.get()).isEqualTo(0); - - // Go to the future when we can read the stored items. - fastForwardTimeByMillis(storageConfig.getMinFileAgeForReadMillis()); - - // Read and send stored data. - assertThat(exporter.exportStoredBatch(1, TimeUnit.SECONDS)).isTrue(); - - // Now the data must have been delegated to the original exporter. - assertThat(finishedItems.get()).isEqualTo(1); - - // Bonus: Try to read again, no more data should be available. - assertThat(exporter.exportStoredBatch(1, TimeUnit.SECONDS)).isFalse(); - assertThat(finishedItems.get()).isEqualTo(1); - } - - @SuppressWarnings("DirectInvocationOnMock") - private void fastForwardTimeByMillis(long milliseconds) { - when(clock.now()).thenReturn(NOW_NANOS + MILLISECONDS.toNanos(milliseconds)); + // Creating metric + meter.counterBuilder("counter").build().add(1); + meterProvider.forceFlush(); + verify(metricCallback).onExportSuccess(anyCollection()); + verifyNoMoreInteractions(spanCallback); + + // Waiting for read time + sleep(DELAY_BEFORE_READING_MILLIS); + + // Read + List storedSpans = new ArrayList<>(); + List storedLogs = new ArrayList<>(); + List storedMetrics = new ArrayList<>(); + spanStorage.forEach(storedSpans::addAll); + logStorage.forEach(storedLogs::addAll); + metricStorage.forEach(storedMetrics::addAll); + + assertThat(storedSpans).hasSize(1); + assertThat(storedLogs).hasSize(1); + assertThat(storedMetrics).hasSize(1); } private static SdkTracerProvider createTracerProvider(SpanExporter exporter) { diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporterTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporterTest.java deleted file mode 100644 index 6409cf067..000000000 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/LogRecordToDiskExporterTest.java +++ /dev/null @@ -1,64 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.Mockito.doThrow; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.verify; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.ToDiskExporter; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.logs.data.LogRecordData; -import java.io.IOException; -import java.util.Arrays; -import java.util.List; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.Mock; -import org.mockito.junit.jupiter.MockitoExtension; - -@ExtendWith(MockitoExtension.class) -class LogRecordToDiskExporterTest { - - @Mock private ToDiskExporter delegate; - - @Test - void delegateShutdown_success() throws IOException { - LogRecordToDiskExporter testClass = new LogRecordToDiskExporter(delegate); - CompletableResultCode result = testClass.shutdown(); - assertThat(result.isSuccess()).isTrue(); - verify(delegate).shutdown(); - } - - @Test - void delegateShutdown_fail() throws IOException { - doThrow(new IOException("boom")).when(delegate).shutdown(); - LogRecordToDiskExporter testClass = new LogRecordToDiskExporter(delegate); - CompletableResultCode result = testClass.shutdown(); - assertThat(result.isSuccess()).isFalse(); - verify(delegate).shutdown(); - } - - @Test - void delegateExport() { - LogRecordData log1 = mock(); - LogRecordData log2 = mock(); - List logRecords = Arrays.asList(log1, log2); - - LogRecordToDiskExporter testClass = new LogRecordToDiskExporter(delegate); - testClass.export(logRecords); - - verify(delegate).export(logRecords); - } - - @Test - void flushReturnsSuccess() { - LogRecordToDiskExporter testClass = new LogRecordToDiskExporter(delegate); - CompletableResultCode result = testClass.flush(); - assertThat(result.isSuccess()).isTrue(); - } -} diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporterTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporterTest.java deleted file mode 100644 index 9ba84f67c..000000000 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/MetricToDiskExporterTest.java +++ /dev/null @@ -1,77 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import static io.opentelemetry.sdk.metrics.data.AggregationTemporality.CUMULATIVE; -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.Mockito.doThrow; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.verify; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.ToDiskExporter; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.metrics.InstrumentType; -import io.opentelemetry.sdk.metrics.data.AggregationTemporality; -import io.opentelemetry.sdk.metrics.data.MetricData; -import java.io.IOException; -import java.util.Arrays; -import java.util.List; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.Mock; -import org.mockito.junit.jupiter.MockitoExtension; - -@ExtendWith(MockitoExtension.class) -class MetricToDiskExporterTest { - - @Mock private ToDiskExporter delegate; - - @Test - void delegateShutdown_success() throws IOException { - MetricToDiskExporter testClass = - new MetricToDiskExporter(delegate, MetricToDiskExporterTest::temporalityFn); - CompletableResultCode result = testClass.shutdown(); - assertThat(result.isSuccess()).isTrue(); - verify(delegate).shutdown(); - } - - private static AggregationTemporality temporalityFn(InstrumentType instrumentType) { - return CUMULATIVE; - } - - @Test - void delegateShutdown_fail() throws IOException { - doThrow(new IOException("boom")).when(delegate).shutdown(); - MetricToDiskExporter testClass = - new MetricToDiskExporter(delegate, MetricToDiskExporterTest::temporalityFn); - CompletableResultCode result = testClass.shutdown(); - assertThat(result.isSuccess()).isFalse(); - verify(delegate).shutdown(); - } - - @Test - void delegateExport() { - MetricData metric1 = mock(); - MetricData metric2 = mock(); - List metrics = Arrays.asList(metric1, metric2); - - MetricToDiskExporter testClass = - new MetricToDiskExporter(delegate, MetricToDiskExporterTest::temporalityFn); - - testClass.export(metrics); - - verify(delegate).export(metrics); - } - - @Test - void flushReturnsSuccess() { - MetricToDiskExporter testClass = - new MetricToDiskExporter(delegate, MetricToDiskExporterTest::temporalityFn); - - CompletableResultCode result = testClass.flush(); - assertThat(result.isSuccess()).isTrue(); - } -} diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/SpanFromDiskExporterTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/SpanFromDiskExporterTest.java deleted file mode 100644 index 2ea0d2b8a..000000000 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/SpanFromDiskExporterTest.java +++ /dev/null @@ -1,145 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.ArgumentMatchers.eq; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.when; - -import io.opentelemetry.api.common.AttributeKey; -import io.opentelemetry.api.common.Attributes; -import io.opentelemetry.api.trace.SpanContext; -import io.opentelemetry.api.trace.SpanKind; -import io.opentelemetry.api.trace.StatusCode; -import io.opentelemetry.api.trace.TraceFlags; -import io.opentelemetry.api.trace.TraceState; -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.spans.models.SpanDataImpl; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; -import io.opentelemetry.contrib.disk.buffering.testutils.TestData; -import io.opentelemetry.sdk.common.Clock; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.resources.Resource; -import io.opentelemetry.sdk.trace.data.SpanData; -import io.opentelemetry.sdk.trace.data.StatusData; -import io.opentelemetry.sdk.trace.export.SpanExporter; -import java.io.File; -import java.util.Arrays; -import java.util.Collection; -import java.util.Collections; -import java.util.List; -import java.util.concurrent.TimeUnit; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; -import org.mockito.ArgumentCaptor; - -class SpanFromDiskExporterTest { - - @TempDir File tempDir; - - @SuppressWarnings("unchecked") - @Test - void fromDisk() throws Exception { - Clock clock = mock(Clock.class); - long start = TimeUnit.MILLISECONDS.toNanos(System.currentTimeMillis()); - when(clock.now()).thenReturn(start); - Storage storage = - Storage.builder(SignalTypes.spans) - .setStorageConfiguration(StorageConfiguration.builder().setRootDir(tempDir).build()) - .setStorageClock(clock) - .build(); - - List spans = writeSomeSpans(storage); - - when(clock.now()).thenReturn(start + TimeUnit.SECONDS.toNanos(60)); - - SpanExporter exporter = mock(); - ArgumentCaptor> capture = ArgumentCaptor.forClass(Collection.class); - when(exporter.export(capture.capture())).thenReturn(CompletableResultCode.ofSuccess()); - - SpanFromDiskExporter testClass = SpanFromDiskExporter.create(exporter, storage); - boolean result = testClass.exportStoredBatch(30, TimeUnit.SECONDS); - assertThat(result).isTrue(); - List exportedSpans = (List) capture.getValue(); - - long now = spans.get(0).getStartEpochNanos(); - SpanData expected1 = makeSpan1(TraceFlags.getSampled(), now); - SpanData expected2 = makeSpan2(TraceFlags.getSampled(), now); - - assertThat(exportedSpans.get(0)).isEqualTo(expected1); - assertThat(exportedSpans.get(1)).isEqualTo(expected2); - assertThat(exportedSpans).containsExactly(expected1, expected2); - - verify(exporter).export(eq(Arrays.asList(expected1, expected2))); - } - - private static List writeSomeSpans(Storage storage) throws Exception { - long now = System.currentTimeMillis() * 1_000_000; - SpanData span1 = makeSpan1(TraceFlags.getDefault(), now); - SpanData span2 = makeSpan2(TraceFlags.getSampled(), now); - List spans = Arrays.asList(span1, span2); - - storage.write(SignalSerializer.ofSpans().initialize(spans)); - storage.flush(); - return spans; - } - - private static SpanData makeSpan1(TraceFlags parentSpanContextFlags, long now) { - Attributes attributes = Attributes.of(AttributeKey.stringKey("foo"), "bar"); - SpanContext parentContext = TestData.makeContext(parentSpanContextFlags, TestData.SPAN_ID); - return SpanDataImpl.builder() - .setName("span1") - .setSpanContext( - SpanContext.create( - TestData.TRACE_ID, - TestData.SPAN_ID, - TraceFlags.getDefault(), - TraceState.getDefault())) - .setParentSpanContext(parentContext) - .setInstrumentationScopeInfo(TestData.INSTRUMENTATION_SCOPE_INFO_FULL) - .setStatus(StatusData.create(StatusCode.OK, "whatever")) - .setAttributes(attributes) - .setKind(SpanKind.SERVER) - .setStartEpochNanos(now) - .setEndEpochNanos(now + 50_000_000) - .setTotalRecordedEvents(0) - .setTotalRecordedLinks(0) - .setTotalAttributeCount(attributes.size()) - .setLinks(Collections.emptyList()) - .setEvents(Collections.emptyList()) - .setResource(Resource.getDefault()) - .build(); - } - - private static SpanData makeSpan2(TraceFlags parentSpanContextFlags, long now) { - Attributes attributes = Attributes.of(AttributeKey.stringKey("bar"), "baz"); - String spanId = "aaaaaaaaa12312312"; - SpanContext parentContext = TestData.makeContext(parentSpanContextFlags, spanId); - return SpanDataImpl.builder() - .setName("span2") - .setSpanContext( - SpanContext.create( - TestData.TRACE_ID, spanId, TraceFlags.getSampled(), TraceState.getDefault())) - .setParentSpanContext(parentContext) - .setInstrumentationScopeInfo(TestData.INSTRUMENTATION_SCOPE_INFO_FULL) - .setStatus(StatusData.create(StatusCode.OK, "excellent")) - .setAttributes(attributes) - .setKind(SpanKind.CLIENT) - .setStartEpochNanos(now + 12) - .setEndEpochNanos(now + 12 + 40_000_000) - .setTotalRecordedEvents(0) - .setTotalRecordedLinks(0) - .setTotalAttributeCount(attributes.size()) - .setLinks(Collections.emptyList()) - .setEvents(Collections.emptyList()) - .setResource(Resource.getDefault()) - .build(); - } -} diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporterTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporterTest.java deleted file mode 100644 index 96dcfcaa9..000000000 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/SpanToDiskExporterTest.java +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.Mockito.doThrow; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.verify; - -import io.opentelemetry.contrib.disk.buffering.internal.exporter.ToDiskExporter; -import io.opentelemetry.sdk.common.CompletableResultCode; -import io.opentelemetry.sdk.trace.data.SpanData; -import java.io.IOException; -import java.util.Arrays; -import java.util.List; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.Mock; -import org.mockito.junit.jupiter.MockitoExtension; - -@ExtendWith(MockitoExtension.class) -class SpanToDiskExporterTest { - @Mock private ToDiskExporter delegate; - - @Test - void delegateShutdown_success() throws IOException { - SpanToDiskExporter testClass = new SpanToDiskExporter(delegate); - CompletableResultCode result = testClass.shutdown(); - assertThat(result.isSuccess()).isTrue(); - verify(delegate).shutdown(); - } - - @Test - void delegateShutdown_fail() throws IOException { - doThrow(new IOException("boom")).when(delegate).shutdown(); - SpanToDiskExporter testClass = new SpanToDiskExporter(delegate); - CompletableResultCode result = testClass.shutdown(); - assertThat(result.isSuccess()).isFalse(); - verify(delegate).shutdown(); - } - - @Test - void delegateExport() { - SpanData span1 = mock(); - SpanData span2 = mock(); - List spans = Arrays.asList(span1, span2); - - SpanToDiskExporter testClass = new SpanToDiskExporter(delegate); - testClass.export(spans); - - verify(delegate).export(spans); - } - - @Test - void flushReturnsSuccess() { - SpanToDiskExporter testClass = new SpanToDiskExporter(delegate); - CompletableResultCode result = testClass.flush(); - assertThat(result.isSuccess()).isTrue(); - } -} diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporterTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporterTest.java deleted file mode 100644 index 0a98061ac..000000000 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/exporter/ToDiskExporterTest.java +++ /dev/null @@ -1,90 +0,0 @@ -/* - * Copyright The OpenTelemetry Authors - * SPDX-License-Identifier: Apache-2.0 - */ - -package io.opentelemetry.contrib.disk.buffering.internal.exporter; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.mockito.ArgumentMatchers.any; -import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.when; - -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.Storage; -import io.opentelemetry.sdk.common.CompletableResultCode; -import java.io.IOException; -import java.util.Arrays; -import java.util.Collection; -import java.util.List; -import java.util.concurrent.atomic.AtomicReference; -import java.util.function.Function; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.Mock; -import org.mockito.junit.jupiter.MockitoExtension; - -@ExtendWith(MockitoExtension.class) -class ToDiskExporterTest { - - private final List records = Arrays.asList("one", "two", "three"); - - @Mock private SignalSerializer serializer; - - @Mock private Storage storage; - private ToDiskExporter toDiskExporter; - private Function, CompletableResultCode> exportFn; - private Collection exportedFnSeen; - private AtomicReference exportFnResultToReturn; - - @BeforeEach - void setup() { - exportedFnSeen = null; - exportFnResultToReturn = new AtomicReference<>(null); - exportFn = - (Collection x) -> { - exportedFnSeen = x; - return exportFnResultToReturn.get(); - }; - toDiskExporter = new ToDiskExporter<>(serializer, exportFn, storage); - } - - @Test - void whenWritingSucceedsOnExport_returnSuccessfulResultCode() throws Exception { - when(storage.write(any())).thenReturn(true); - CompletableResultCode completableResultCode = toDiskExporter.export(records); - assertThat(completableResultCode.isSuccess()).isTrue(); - verify(storage).write(any()); - assertThat(exportedFnSeen).isNull(); - } - - @Test - void whenWritingFailsOnExport_doExportRightAway() throws Exception { - when(storage.write(any())).thenReturn(false); - exportFnResultToReturn.set(CompletableResultCode.ofSuccess()); - - CompletableResultCode completableResultCode = toDiskExporter.export(records); - - assertThat(completableResultCode.isSuccess()).isTrue(); - assertThat(exportedFnSeen).isEqualTo(records); - } - - @Test - void whenExceptionInWrite_doExportRightAway() throws Exception { - when(storage.write(any())).thenThrow(new IOException("boom")); - exportFnResultToReturn.set(CompletableResultCode.ofFailure()); - - CompletableResultCode completableResultCode = toDiskExporter.export(records); - - assertThat(completableResultCode.isSuccess()).isFalse(); - assertThat(exportedFnSeen).isEqualTo(records); - } - - @Test - void shutdownClosesStorage() throws Exception { - toDiskExporter.export(records); - toDiskExporter.shutdown(); - verify(storage).close(); - } -} diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/exporters/SignalStorageExporterTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/exporters/SignalStorageExporterTest.java similarity index 78% rename from disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/exporters/SignalStorageExporterTest.java rename to disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/exporters/SignalStorageExporterTest.java index 6f7db99ef..30bfe310a 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/exporters/SignalStorageExporterTest.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/exporters/SignalStorageExporterTest.java @@ -3,7 +3,7 @@ * SPDX-License-Identifier: Apache-2.0 */ -package io.opentelemetry.contrib.disk.buffering.exporters; +package io.opentelemetry.contrib.disk.buffering.internal.exporters; import static org.assertj.core.api.Assertions.assertThat; import static org.mockito.ArgumentMatchers.anyCollection; @@ -13,7 +13,7 @@ import static org.mockito.Mockito.verifyNoMoreInteractions; import static org.mockito.Mockito.when; -import io.opentelemetry.contrib.disk.buffering.SignalType; +import io.opentelemetry.contrib.disk.buffering.exporters.callback.ExporterCallback; import io.opentelemetry.contrib.disk.buffering.storage.SignalStorage; import io.opentelemetry.contrib.disk.buffering.storage.result.WriteResult; import io.opentelemetry.sdk.common.CompletableResultCode; @@ -32,32 +32,34 @@ import org.mockito.Mock; import org.mockito.junit.jupiter.MockitoExtension; +@SuppressWarnings("unchecked") @ExtendWith(MockitoExtension.class) class SignalStorageExporterTest { - @Mock private ExporterCallback callback; + @Mock private ExporterCallback callback; @Test void verifyExportToStorage_success() { SignalStorage.Span storage = new TestSpanStorage(); - SignalType signalType = SignalType.SPAN; SignalStorageExporter storageExporter = - new SignalStorageExporter<>(storage, callback, Duration.ofSeconds(1), signalType); + new SignalStorageExporter<>(storage, callback, Duration.ofSeconds(1)); SpanData item1 = mock(); SpanData item2 = mock(); SpanData item3 = mock(); - CompletableResultCode resultCode = storageExporter.exportToStorage(Arrays.asList(item1, item2)); + List items = Arrays.asList(item1, item2); + CompletableResultCode resultCode = storageExporter.exportToStorage(items); assertThat(resultCode.isSuccess()).isTrue(); - verify(callback).onExportSuccess(signalType); + verify(callback).onExportSuccess(items); verifyNoMoreInteractions(callback); // Adding more items clearInvocations(callback); - resultCode = storageExporter.exportToStorage(Collections.singletonList(item3)); + List items2 = Collections.singletonList(item3); + resultCode = storageExporter.exportToStorage(items2); assertThat(resultCode.isSuccess()).isTrue(); - verify(callback).onExportSuccess(signalType); + verify(callback).onExportSuccess(items2); verifyNoMoreInteractions(callback); // Checking items @@ -68,38 +70,36 @@ void verifyExportToStorage_success() { assertThat(storedItems).containsExactly(item1, item2, item3); } - @SuppressWarnings("ThrowableNotThrown") @Test void verifyExportToStorage_failure() { SignalStorage.Span storage = mock(); - SignalType signalType = SignalType.SPAN; SignalStorageExporter storageExporter = - new SignalStorageExporter<>(storage, callback, Duration.ofSeconds(1), signalType); + new SignalStorageExporter<>(storage, callback, Duration.ofSeconds(1)); SpanData item1 = mock(); // Without exception when(storage.write(anyCollection())) - .thenReturn(CompletableFuture.completedFuture(WriteResult.create(false, null))); + .thenReturn(CompletableFuture.completedFuture(WriteResult.error(null))); - CompletableResultCode resultCode = - storageExporter.exportToStorage(Collections.singletonList(item1)); + List items = Collections.singletonList(item1); + CompletableResultCode resultCode = storageExporter.exportToStorage(items); assertThat(resultCode.isSuccess()).isFalse(); assertThat(resultCode.getFailureThrowable()).isNull(); - verify(callback).onExportError(signalType, null); + verify(callback).onExportError(items, null); verifyNoMoreInteractions(callback); // With exception clearInvocations(callback); Exception exception = new Exception(); when(storage.write(anyCollection())) - .thenReturn(CompletableFuture.completedFuture(WriteResult.create(false, exception))); + .thenReturn(CompletableFuture.completedFuture(WriteResult.error(exception))); - resultCode = storageExporter.exportToStorage(Collections.singletonList(item1)); + resultCode = storageExporter.exportToStorage(items); assertThat(resultCode.isSuccess()).isFalse(); assertThat(resultCode.getFailureThrowable()).isEqualTo(exception); - verify(callback).onExportError(signalType, exception); + verify(callback).onExportError(items, exception); verifyNoMoreInteractions(callback); } @@ -129,7 +129,7 @@ public Iterator> iterator() { @Nonnull private static CompletableFuture getSuccessfulFuture() { - return CompletableFuture.completedFuture(WriteResult.create(true, null)); + return CompletableFuture.completedFuture(WriteResult.successful()); } } } diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManagerTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManagerTest.java index d8a95a9cc..044e7be9b 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManagerTest.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/FolderManagerTest.java @@ -35,7 +35,7 @@ class FolderManagerTest { @BeforeEach void setUp() { clock = mock(); - folderManager = new FolderManager(rootDir, TestData.getConfiguration(rootDir), clock); + folderManager = new FolderManager(rootDir, TestData.getConfiguration(), clock); } @AfterEach @@ -51,6 +51,19 @@ void createWritableFile_withTimeMillisAsName() throws IOException { assertThat(file.getFile().getName()).isEqualTo("1000"); } + @Test + void clearFiles() throws IOException { + when(clock.now()).thenReturn(MILLISECONDS.toNanos(1000L)); + + // Creating file + folderManager.createWritableFile(); + assertThat(rootDir.list()).containsExactly("1000"); + + // Clear + folderManager.clear(); + assertThat(rootDir.list()).isEmpty(); + } + @Test void createWritableFile_andRemoveOldestOne_whenTheAvailableFolderSpaceIsNotEnough() throws IOException { diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageTest.java index b33387a8c..611506b1b 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageTest.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/StorageTest.java @@ -5,47 +5,51 @@ package io.opentelemetry.contrib.disk.buffering.internal.storage; -import static io.opentelemetry.contrib.disk.buffering.internal.storage.responses.ReadableResult.TRY_LATER; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.FIRST_LOG_RECORD; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.MAX_FILE_AGE_FOR_READ_MILLIS; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.MAX_FILE_AGE_FOR_WRITE_MILLIS; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.MIN_FILE_AGE_FOR_READ_MILLIS; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.SECOND_LOG_RECORD; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.THIRD_LOG_RECORD; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.getConfiguration; import static org.assertj.core.api.Assertions.assertThat; -import static org.assertj.core.api.Assertions.assertThatThrownBy; -import static org.mockito.ArgumentMatchers.any; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.times; -import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.verifyNoInteractions; -import static org.mockito.Mockito.when; - -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.ByteArraySerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.ReadableFile; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.WritableFile; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.ProcessResult; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.fail; + +import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.ReadableResult; -import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.WritableResult; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; +import io.opentelemetry.sdk.common.Clock; +import io.opentelemetry.sdk.logs.data.LogRecordData; import java.io.File; import java.io.IOException; -import java.util.function.Function; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; -@SuppressWarnings("unchecked") class StorageTest { + @TempDir private File destinationDir; private FolderManager folderManager; - private Storage storage; - private Function processing; - private ReadableFile readableFile; - private WritableFile writableFile; + private Storage storage; + private SignalSerializer serializer; + private AtomicLong currentTimeMillis; + private static final SignalDeserializer DESERIALIZER = SignalDeserializer.ofLogs(); @BeforeEach - void setUp() throws IOException { - folderManager = mock(); - readableFile = mock(); - writableFile = createWritableFile(); - processing = mock(); - when(readableFile.readAndProcess(processing)).thenReturn(ReadableResult.SUCCEEDED); - storage = new Storage(folderManager, true); + void setUp() { + currentTimeMillis = new AtomicLong(0); + serializer = SignalSerializer.ofLogs(); + folderManager = FolderManager.create(destinationDir, getConfiguration(), new TestClock()); + storage = new Storage<>(folderManager); } @AfterEach @@ -54,203 +58,167 @@ void tearDown() throws IOException { } @Test - void whenReadingAndProcessingSuccessfully_returnSuccess() throws IOException { - when(folderManager.getReadableFile()).thenReturn(readableFile); - - assertThat(storage.readAndProcess(processing)).isEqualTo(ReadableResult.SUCCEEDED); + void writeAndRead() throws IOException { + assertThat(write(Arrays.asList(FIRST_LOG_RECORD, SECOND_LOG_RECORD))).isTrue(); + assertThat(write(Collections.singletonList(THIRD_LOG_RECORD))).isTrue(); + assertThat(destinationDir.list()).hasSize(1); + forwardToReadTime(); - verify(readableFile).readAndProcess(processing); - } - - @Test - void whenReadableFileProcessingFails_returnTryLater() throws IOException { - when(folderManager.getReadableFile()).thenReturn(readableFile); - when(readableFile.readAndProcess(processing)).thenReturn(TRY_LATER); + ReadableResult readResult = storage.readNext(DESERIALIZER); + assertNotNull(readResult); + assertThat(readResult.getContent()).containsExactly(FIRST_LOG_RECORD, SECOND_LOG_RECORD); + assertThat(destinationDir.list()).hasSize(1); - assertThat(storage.readAndProcess(processing)).isEqualTo(TRY_LATER); - - verify(readableFile).readAndProcess(processing); - } - - @Test - void whenReadingMultipleTimes_reuseReader() throws IOException { - ReadableFile anotherReadable = mock(); - when(folderManager.getReadableFile()).thenReturn(readableFile).thenReturn(anotherReadable); + // Delete result and read again + readResult.delete(); + readResult.close(); + ReadableResult readResult2 = storage.readNext(DESERIALIZER); + assertNotNull(readResult2); + assertThat(readResult2.getContent()).containsExactly(THIRD_LOG_RECORD); + assertThat(destinationDir.list()).hasSize(1); - assertThat(storage.readAndProcess(processing)).isEqualTo(ReadableResult.SUCCEEDED); - assertThat(storage.readAndProcess(processing)).isEqualTo(ReadableResult.SUCCEEDED); + // Read again without closing previous result + try { + storage.readNext(DESERIALIZER); + fail(); + } catch (IllegalStateException e) { + assertThat(e) + .hasMessage("You must close any previous ReadableResult before requesting a new one"); + } - verify(readableFile, times(2)).readAndProcess(processing); - verify(folderManager, times(1)).getReadableFile(); - verifyNoInteractions(anotherReadable); + // Read again when no more data is available (delete file) + readResult2.close(); + assertNull(storage.readNext(DESERIALIZER)); + assertThat(destinationDir.list()).isEmpty(); } @Test - void whenWritingMultipleTimes_reuseWriter() throws IOException { - ByteArraySerializer data = new ByteArraySerializer(new byte[1]); - WritableFile anotherWriter = createWritableFile(); - when(folderManager.createWritableFile()).thenReturn(writableFile).thenReturn(anotherWriter); - - storage.write(data); - storage.write(data); - - verify(writableFile, times(2)).append(data); - verify(folderManager, times(1)).createWritableFile(); - verifyNoInteractions(anotherWriter); - } - - @Test - void whenAttemptingToReadAfterClosed_returnFailed() throws IOException { + void interactionAfterClosed() throws IOException { + assertThat(write(Arrays.asList(FIRST_LOG_RECORD, SECOND_LOG_RECORD))).isTrue(); storage.close(); - assertThat(storage.readAndProcess(processing)).isEqualTo(ReadableResult.FAILED); - } + assertThat(destinationDir.list()).hasSize(1); + forwardToReadTime(); - @Test - void whenAttemptingToWriteAfterClosed_returnFalse() throws IOException { - storage.close(); - assertThat(storage.write(new ByteArraySerializer(new byte[1]))).isFalse(); - } + // Reading + assertNull(storage.readNext(DESERIALIZER)); - @Test - void whenNoFileAvailableForReading_returnFailed() throws IOException { - assertThat(storage.readAndProcess(processing)).isEqualTo(ReadableResult.FAILED); + // Writing + assertThat(write(Collections.singletonList(THIRD_LOG_RECORD))).isFalse(); } @Test void whenTheReadTimeExpires_lookForNewFileToRead() throws IOException { - when(folderManager.getReadableFile()).thenReturn(readableFile).thenReturn(null); - when(readableFile.readAndProcess(processing)).thenReturn(ReadableResult.FAILED); + long firstFileWriteTime = 1000; + long secondFileWriteTime = firstFileWriteTime + MAX_FILE_AGE_FOR_WRITE_MILLIS + 1; + currentTimeMillis.set(firstFileWriteTime); + assertThat(write(Arrays.asList(FIRST_LOG_RECORD, SECOND_LOG_RECORD))).isTrue(); - storage.readAndProcess(processing); + // Forward past first file write time + currentTimeMillis.set(secondFileWriteTime); + assertThat(write(Collections.singletonList(THIRD_LOG_RECORD))).isTrue(); + assertThat(destinationDir.list()) + .containsExactlyInAnyOrder( + String.valueOf(firstFileWriteTime), String.valueOf(secondFileWriteTime)); - verify(folderManager, times(2)).getReadableFile(); - } + // Forward past first time read + currentTimeMillis.set(firstFileWriteTime + MAX_FILE_AGE_FOR_READ_MILLIS + 1); - @Test - void whenNoMoreLinesToRead_lookForNewFileToRead() throws IOException { - when(folderManager.getReadableFile()).thenReturn(readableFile).thenReturn(null); - when(readableFile.readAndProcess(processing)).thenReturn(ReadableResult.FAILED); + // Read + ReadableResult result = storage.readNext(DESERIALIZER); + assertNotNull(result); + assertThat(result.getContent()).containsExactly(THIRD_LOG_RECORD); + assertThat(destinationDir.list()) + .containsExactlyInAnyOrder( + String.valueOf(firstFileWriteTime), String.valueOf(secondFileWriteTime)); - storage.readAndProcess(processing); - - verify(folderManager, times(2)).getReadableFile(); + // Purge expired files on write + currentTimeMillis.set(50000); + assertThat(write(Collections.singletonList(FIRST_LOG_RECORD))).isTrue(); + assertThat(destinationDir.list()).containsExactly("50000"); } @Test - void whenResourceClosed_lookForNewFileToRead() throws IOException { - when(folderManager.getReadableFile()).thenReturn(readableFile).thenReturn(null); - when(readableFile.readAndProcess(processing)).thenReturn(ReadableResult.FAILED); - - storage.readAndProcess(processing); - - verify(folderManager, times(2)).getReadableFile(); - } - - @Test - void whenEveryNewFileFoundCannotBeRead_returnContentNotAvailable() throws IOException { - when(folderManager.getReadableFile()).thenReturn(readableFile); - when(readableFile.readAndProcess(processing)).thenReturn(ReadableResult.FAILED); - - assertThat(storage.readAndProcess(processing)).isEqualTo(ReadableResult.FAILED); - - verify(folderManager, times(3)).getReadableFile(); - } - - @Test - void appendDataToFile() throws IOException { - when(folderManager.createWritableFile()).thenReturn(writableFile); - ByteArraySerializer data = new ByteArraySerializer(new byte[1]); - - storage.write(data); - - verify(writableFile).append(data); - } - - @Test - void whenWritingTimeoutHappens_retryWithNewFile() throws IOException { - ByteArraySerializer data = new ByteArraySerializer(new byte[1]); - WritableFile workingWritableFile = createWritableFile(); - when(folderManager.createWritableFile()) - .thenReturn(writableFile) - .thenReturn(workingWritableFile); - when(writableFile.append(data)).thenReturn(WritableResult.FAILED); - - storage.write(data); - - verify(folderManager, times(2)).createWritableFile(); - } - - @Test - void whenThereIsNoSpaceAvailableForWriting_retryWithNewFile() throws IOException { - ByteArraySerializer data = new ByteArraySerializer(new byte[1]); - WritableFile workingWritableFile = createWritableFile(); - when(folderManager.createWritableFile()) - .thenReturn(writableFile) - .thenReturn(workingWritableFile); - when(writableFile.append(data)).thenReturn(WritableResult.FAILED); - - storage.write(data); - - verify(folderManager, times(2)).createWritableFile(); - } - - @Test - void whenWritingResourceIsClosed_retryWithNewFile() throws IOException { - ByteArraySerializer data = new ByteArraySerializer(new byte[1]); - WritableFile workingWritableFile = createWritableFile(); - when(folderManager.createWritableFile()) - .thenReturn(writableFile) - .thenReturn(workingWritableFile); - when(writableFile.append(data)).thenReturn(WritableResult.FAILED); - - storage.write(data); - - verify(folderManager, times(2)).createWritableFile(); - } - - @Test - void whenEveryAttemptToWriteFails_returnFalse() throws IOException { - ByteArraySerializer data = new ByteArraySerializer(new byte[1]); - when(folderManager.createWritableFile()).thenReturn(writableFile); - when(writableFile.append(data)).thenReturn(WritableResult.FAILED); - - assertThat(storage.write(data)).isFalse(); - - verify(folderManager, times(3)).createWritableFile(); - } - - @Test - void whenClosing_closeWriterAndReaderIfNotNull() throws IOException { - when(folderManager.createWritableFile()).thenReturn(writableFile); - when(folderManager.getReadableFile()).thenReturn(readableFile); - storage.write(new ByteArraySerializer(new byte[1])); - storage.readAndProcess(processing); - - storage.close(); - - verify(writableFile).close(); - verify(readableFile).close(); - } - - @Test - void whenMinFileReadIsNotGraterThanMaxFileWrite_throwException() { - StorageConfiguration invalidConfig = - StorageConfiguration.builder() - .setMaxFileAgeForWriteMillis(2) - .setMinFileAgeForReadMillis(1) - .setRootDir(new File(".")) - .build(); - - assertThatThrownBy( - () -> Storage.builder(SignalTypes.logs).setStorageConfiguration(invalidConfig)) - .isInstanceOf(IllegalArgumentException.class) - .hasMessage( - "The configured max file age for writing must be lower than the configured min file age for reading"); - } - - private static WritableFile createWritableFile() throws IOException { - WritableFile mock = mock(); - when(mock.append(any())).thenReturn(WritableResult.SUCCEEDED); - return mock; + void whenNoMoreLinesToRead_lookForNewFileToRead() throws IOException { + long firstFileWriteTime = 1000; + long secondFileWriteTime = firstFileWriteTime + MAX_FILE_AGE_FOR_WRITE_MILLIS + 1; + currentTimeMillis.set(firstFileWriteTime); + assertThat(write(Arrays.asList(FIRST_LOG_RECORD, SECOND_LOG_RECORD))).isTrue(); + + // Forward past first file write time + currentTimeMillis.set(secondFileWriteTime); + assertThat(write(Collections.singletonList(THIRD_LOG_RECORD))).isTrue(); + assertThat(destinationDir.list()) + .containsExactlyInAnyOrder( + String.valueOf(firstFileWriteTime), String.valueOf(secondFileWriteTime)); + + // Forward to all files read time + currentTimeMillis.set(secondFileWriteTime + MIN_FILE_AGE_FOR_READ_MILLIS); + + // Read + ReadableResult result = storage.readNext(DESERIALIZER); + assertNotNull(result); + assertThat(result.getContent()).containsExactly(FIRST_LOG_RECORD, SECOND_LOG_RECORD); + assertThat(destinationDir.list()) + .containsExactlyInAnyOrder( + String.valueOf(firstFileWriteTime), String.valueOf(secondFileWriteTime)); + result.delete(); + result.close(); + + // Read again + ReadableResult result2 = storage.readNext(DESERIALIZER); + assertNotNull(result2); + assertThat(result2.getContent()).containsExactly(THIRD_LOG_RECORD); + assertThat(destinationDir.list()).containsExactly(String.valueOf(secondFileWriteTime)); + result2.close(); + } + + @Test + void deleteFilesWithCorruptedData() throws IOException { + // Add files with invalid data + Files.write( + new File(destinationDir, "1000").toPath(), "random data".getBytes(StandardCharsets.UTF_8)); + Files.write( + new File(destinationDir, "2000").toPath(), "random data".getBytes(StandardCharsets.UTF_8)); + Files.write( + new File(destinationDir, "3000").toPath(), "random data".getBytes(StandardCharsets.UTF_8)); + Files.write( + new File(destinationDir, "4000").toPath(), "random data".getBytes(StandardCharsets.UTF_8)); + + // Set time ready to read all files + currentTimeMillis.set(4000 + MIN_FILE_AGE_FOR_READ_MILLIS); + + // Read + assertNull(storage.readNext(DESERIALIZER)); + assertThat(destinationDir.list()).containsExactly("4000"); // it tries 3 times max per call. + } + + private void forwardToReadTime() { + forwardCurrentTimeByMillis(MIN_FILE_AGE_FOR_READ_MILLIS); + } + + private void forwardCurrentTimeByMillis(long millis) { + currentTimeMillis.set(currentTimeMillis.get() + millis); + } + + private boolean write(Collection items) throws IOException { + serializer.initialize(items); + try { + return storage.write(serializer); + } finally { + serializer.reset(); + } + } + + private class TestClock implements Clock { + + @Override + public long now() { + return TimeUnit.MILLISECONDS.toNanos(currentTimeMillis.get()); + } + + @Override + public long nanoTime() { + return 0; + } } } diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/TestData.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/TestData.java index 75d86726e..3b51125b6 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/TestData.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/TestData.java @@ -5,31 +5,73 @@ package io.opentelemetry.contrib.disk.buffering.internal.storage; -import io.opentelemetry.contrib.disk.buffering.config.StorageConfiguration; -import io.opentelemetry.contrib.disk.buffering.internal.utils.SignalTypes; -import io.opentelemetry.sdk.common.Clock; -import java.io.File; -import java.io.IOException; +import io.opentelemetry.api.common.Value; +import io.opentelemetry.api.logs.Severity; +import io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.logs.models.LogRecordDataImpl; +import io.opentelemetry.contrib.disk.buffering.storage.impl.FileStorageConfiguration; +import io.opentelemetry.sdk.logs.data.LogRecordData; public final class TestData { + public static final LogRecordData FIRST_LOG_RECORD = + LogRecordDataImpl.builder() + .setResource(io.opentelemetry.contrib.disk.buffering.testutils.TestData.RESOURCE_FULL) + .setSpanContext(io.opentelemetry.contrib.disk.buffering.testutils.TestData.SPAN_CONTEXT) + .setInstrumentationScopeInfo( + io.opentelemetry.contrib.disk.buffering.testutils.TestData + .INSTRUMENTATION_SCOPE_INFO_FULL) + .setAttributes(io.opentelemetry.contrib.disk.buffering.testutils.TestData.ATTRIBUTES) + .setBodyValue(Value.of("First log body")) + .setSeverity(Severity.DEBUG) + .setSeverityText("Log severity text") + .setTimestampEpochNanos(100L) + .setObservedTimestampEpochNanos(200L) + .setTotalAttributeCount(3) + .setEventName("") + .build(); + + public static final LogRecordData SECOND_LOG_RECORD = + LogRecordDataImpl.builder() + .setResource(io.opentelemetry.contrib.disk.buffering.testutils.TestData.RESOURCE_FULL) + .setSpanContext(io.opentelemetry.contrib.disk.buffering.testutils.TestData.SPAN_CONTEXT) + .setInstrumentationScopeInfo( + io.opentelemetry.contrib.disk.buffering.testutils.TestData + .INSTRUMENTATION_SCOPE_INFO_FULL) + .setAttributes(io.opentelemetry.contrib.disk.buffering.testutils.TestData.ATTRIBUTES) + .setBodyValue(Value.of("Second log body")) + .setSeverity(Severity.DEBUG) + .setSeverityText("Log severity text") + .setTimestampEpochNanos(100L) + .setObservedTimestampEpochNanos(200L) + .setTotalAttributeCount(3) + .setEventName("event") + .build(); + + public static final LogRecordData THIRD_LOG_RECORD = + LogRecordDataImpl.builder() + .setResource(io.opentelemetry.contrib.disk.buffering.testutils.TestData.RESOURCE_FULL) + .setSpanContext(io.opentelemetry.contrib.disk.buffering.testutils.TestData.SPAN_CONTEXT) + .setInstrumentationScopeInfo( + io.opentelemetry.contrib.disk.buffering.testutils.TestData + .INSTRUMENTATION_SCOPE_INFO_FULL) + .setAttributes(io.opentelemetry.contrib.disk.buffering.testutils.TestData.ATTRIBUTES) + .setBodyValue(Value.of("Third log body")) + .setSeverity(Severity.DEBUG) + .setSeverityText("Log severity text") + .setTimestampEpochNanos(100L) + .setObservedTimestampEpochNanos(200L) + .setTotalAttributeCount(3) + .setEventName("") + .build(); + public static final long MAX_FILE_AGE_FOR_WRITE_MILLIS = 1000; public static final long MIN_FILE_AGE_FOR_READ_MILLIS = MAX_FILE_AGE_FOR_WRITE_MILLIS + 500; public static final long MAX_FILE_AGE_FOR_READ_MILLIS = 10_000; - public static final int MAX_FILE_SIZE = 100; - public static final int MAX_FOLDER_SIZE = 300; - - public static Storage getStorage(File rootDir, SignalTypes types, Clock clock) - throws IOException { - return Storage.builder(types) - .setStorageConfiguration(getConfiguration(rootDir)) - .setStorageClock(clock) - .build(); - } + public static final int MAX_FILE_SIZE = 2000; + public static final int MAX_FOLDER_SIZE = 6000; - public static StorageConfiguration getConfiguration(File rootDir) { - return StorageConfiguration.builder() - .setRootDir(rootDir) + public static FileStorageConfiguration getConfiguration() { + return FileStorageConfiguration.builder() .setMaxFileAgeForWriteMillis(MAX_FILE_AGE_FOR_WRITE_MILLIS) .setMinFileAgeForReadMillis(MIN_FILE_AGE_FOR_READ_MILLIS) .setMaxFileAgeForReadMillis(MAX_FILE_AGE_FOR_READ_MILLIS) diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFileTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFileTest.java index 8fdc1d41e..791d80faa 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFileTest.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/ReadableFileTest.java @@ -5,23 +5,20 @@ package io.opentelemetry.contrib.disk.buffering.internal.storage.files; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.FIRST_LOG_RECORD; import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.MAX_FILE_AGE_FOR_READ_MILLIS; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.SECOND_LOG_RECORD; +import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.THIRD_LOG_RECORD; import static io.opentelemetry.contrib.disk.buffering.internal.storage.TestData.getConfiguration; import static java.util.concurrent.TimeUnit.MILLISECONDS; import static org.assertj.core.api.Assertions.assertThat; -import static org.assertj.core.api.Assertions.fail; +import static org.junit.jupiter.api.Assertions.fail; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; -import io.opentelemetry.api.common.Value; -import io.opentelemetry.api.logs.Severity; import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.DeserializationException; import io.opentelemetry.contrib.disk.buffering.internal.serialization.deserializers.SignalDeserializer; -import io.opentelemetry.contrib.disk.buffering.internal.serialization.mapping.logs.models.LogRecordDataImpl; import io.opentelemetry.contrib.disk.buffering.internal.serialization.serializers.SignalSerializer; -import io.opentelemetry.contrib.disk.buffering.internal.storage.files.reader.ProcessResult; -import io.opentelemetry.contrib.disk.buffering.internal.storage.responses.ReadableResult; -import io.opentelemetry.contrib.disk.buffering.testutils.TestData; import io.opentelemetry.sdk.common.Clock; import io.opentelemetry.sdk.logs.data.LogRecordData; import java.io.File; @@ -45,57 +42,13 @@ class ReadableFileTest { private static final long CREATED_TIME_MILLIS = 1000L; private static final SignalSerializer SERIALIZER = SignalSerializer.ofLogs(); private static final SignalDeserializer DESERIALIZER = SignalDeserializer.ofLogs(); - private static final LogRecordData FIRST_LOG_RECORD = - LogRecordDataImpl.builder() - .setResource(TestData.RESOURCE_FULL) - .setSpanContext(TestData.SPAN_CONTEXT) - .setInstrumentationScopeInfo(TestData.INSTRUMENTATION_SCOPE_INFO_FULL) - .setAttributes(TestData.ATTRIBUTES) - .setBodyValue(Value.of("First log body")) - .setSeverity(Severity.DEBUG) - .setSeverityText("Log severity text") - .setTimestampEpochNanos(100L) - .setObservedTimestampEpochNanos(200L) - .setTotalAttributeCount(3) - .setEventName("") - .build(); - - private static final LogRecordData SECOND_LOG_RECORD = - LogRecordDataImpl.builder() - .setResource(TestData.RESOURCE_FULL) - .setSpanContext(TestData.SPAN_CONTEXT) - .setInstrumentationScopeInfo(TestData.INSTRUMENTATION_SCOPE_INFO_FULL) - .setAttributes(TestData.ATTRIBUTES) - .setBodyValue(Value.of("Second log body")) - .setSeverity(Severity.DEBUG) - .setSeverityText("Log severity text") - .setTimestampEpochNanos(100L) - .setObservedTimestampEpochNanos(200L) - .setTotalAttributeCount(3) - .setEventName("event") - .build(); - - private static final LogRecordData THIRD_LOG_RECORD = - LogRecordDataImpl.builder() - .setResource(TestData.RESOURCE_FULL) - .setSpanContext(TestData.SPAN_CONTEXT) - .setInstrumentationScopeInfo(TestData.INSTRUMENTATION_SCOPE_INFO_FULL) - .setAttributes(TestData.ATTRIBUTES) - .setBodyValue(Value.of("Third log body")) - .setSeverity(Severity.DEBUG) - .setSeverityText("Log severity text") - .setTimestampEpochNanos(100L) - .setObservedTimestampEpochNanos(200L) - .setTotalAttributeCount(3) - .setEventName("") - .build(); @BeforeEach void setUp() throws IOException { source = new File(dir, "sourceFile"); addFileContents(source); clock = mock(); - readableFile = new ReadableFile(source, CREATED_TIME_MILLIS, clock, getConfiguration(dir)); + readableFile = new ReadableFile(source, CREATED_TIME_MILLIS, clock, getConfiguration()); } @AfterEach @@ -115,58 +68,15 @@ private static void addFileContents(File source) throws IOException { } @Test - void readSingleItemAndRemoveIt() throws IOException { - readableFile.readAndProcess( - bytes -> { - assertThat(deserialize(bytes)).isEqualTo(FIRST_LOG_RECORD); - return ProcessResult.SUCCEEDED; - }); + void readAndRemoveItems() throws IOException { + assertThat(FIRST_LOG_RECORD).isEqualTo(deserialize(readableFile.readNext())); + readableFile.removeTopItem(); List logs = getRemainingDataAndClose(readableFile); - assertThat(logs.size()).isEqualTo(2); - assertThat(logs.get(0)).isEqualTo(SECOND_LOG_RECORD); - assertThat(logs.get(1)).isEqualTo(THIRD_LOG_RECORD); - } - - @Test - void whenProcessingSucceeds_returnSuccessStatus() throws IOException { - assertThat(readableFile.readAndProcess(bytes -> ProcessResult.SUCCEEDED)) - .isEqualTo(ReadableResult.SUCCEEDED); - } - - @Test - void whenProcessingFails_returnTryLaterStatus() throws IOException { - assertThat(readableFile.readAndProcess(bytes -> ProcessResult.TRY_LATER)) - .isEqualTo(ReadableResult.TRY_LATER); - } - - @Test - void readMultipleLinesAndRemoveThem() throws IOException { - readableFile.readAndProcess( - bytes -> { - assertDeserializedData(FIRST_LOG_RECORD, bytes); - return ProcessResult.SUCCEEDED; - }); - readableFile.readAndProcess( - bytes -> { - assertDeserializedData(SECOND_LOG_RECORD, bytes); - return ProcessResult.SUCCEEDED; - }); - - List logs = getRemainingDataAndClose(readableFile); - - assertThat(logs.size()).isEqualTo(1); - assertThat(logs.get(0)).isEqualTo(THIRD_LOG_RECORD); - } - - @Test - void whenConsumerReturnsFalse_doNotRemoveLineFromSource() throws IOException { - readableFile.readAndProcess(bytes -> ProcessResult.TRY_LATER); - - List logs = getRemainingDataAndClose(readableFile); - - assertThat(logs.size()).isEqualTo(3); + assertThat(2).isEqualTo(logs.size()); + assertThat(SECOND_LOG_RECORD).isEqualTo(logs.get(0)); + assertThat(THIRD_LOG_RECORD).isEqualTo(logs.get(1)); } @Test @@ -177,15 +87,6 @@ void whenReadingLastLine_deleteOriginalFile_and_close() throws IOException { assertThat(readableFile.isClosed()).isTrue(); } - @Test - void whenTheFileContentIsInvalid_deleteOriginalFile_and_close() throws IOException { - assertThat(readableFile.readAndProcess(bytes -> ProcessResult.CONTENT_INVALID)) - .isEqualTo(ReadableResult.FAILED); - - assertThat(source.exists()).isFalse(); - assertThat(readableFile.isClosed()).isTrue(); - } - @Test void whenNoMoreLinesAvailableToRead_deleteOriginalFile_close_and_returnNoContentStatus() throws IOException { @@ -195,58 +96,38 @@ void whenNoMoreLinesAvailableToRead_deleteOriginalFile_close_and_returnNoContent } ReadableFile emptyReadableFile = - new ReadableFile(emptyFile, CREATED_TIME_MILLIS, clock, getConfiguration(dir)); + new ReadableFile(emptyFile, CREATED_TIME_MILLIS, clock, getConfiguration()); - assertThat(emptyReadableFile.readAndProcess(bytes -> ProcessResult.SUCCEEDED)) - .isEqualTo(ReadableResult.FAILED); + assertThat(emptyReadableFile.readNext()).isNull(); assertThat(emptyReadableFile.isClosed()).isTrue(); assertThat(emptyFile.exists()).isFalse(); } @Test - void - whenReadingAfterTheConfiguredReadingTimeExpired_deleteOriginalFile_close_and_returnFileExpiredException() - throws IOException { - readableFile.readAndProcess(bytes -> ProcessResult.SUCCEEDED); + void whenReadingAfterTheConfiguredReadingTimeExpired_close() throws IOException { when(clock.now()) .thenReturn(MILLISECONDS.toNanos(CREATED_TIME_MILLIS + MAX_FILE_AGE_FOR_READ_MILLIS)); - assertThat(readableFile.readAndProcess(bytes -> ProcessResult.SUCCEEDED)) - .isEqualTo(ReadableResult.FAILED); - + assertThat(readableFile.readNext()).isNull(); assertThat(readableFile.isClosed()).isTrue(); } @Test - void whenReadingAfterClosed_returnFailedStatus() throws IOException { - readableFile.readAndProcess(bytes -> ProcessResult.SUCCEEDED); + void whenReadingAfterClosed_returnNull() throws IOException { readableFile.close(); - assertThat(readableFile.readAndProcess(bytes -> ProcessResult.SUCCEEDED)) - .isEqualTo(ReadableResult.FAILED); - } - - private static void assertDeserializedData(LogRecordData expected, byte[] bytes) { - try { - List deserialized = DESERIALIZER.deserialize(bytes); - assertThat(deserialized.get(0)).isEqualTo(expected); - } catch (DeserializationException e) { - throw new RuntimeException(e); - } + assertThat(readableFile.readNext()).isNull(); } private static List getRemainingDataAndClose(ReadableFile readableFile) throws IOException { List result = new ArrayList<>(); - ReadableResult readableResult = ReadableResult.SUCCEEDED; - while (readableResult == ReadableResult.SUCCEEDED) { - readableResult = - readableFile.readAndProcess( - bytes -> { - result.add(deserialize(bytes)); - return ProcessResult.SUCCEEDED; - }); + byte[] bytes = readableFile.readNext(); + while (bytes != null) { + result.add(deserialize(bytes)); + readableFile.removeTopItem(); + bytes = readableFile.readNext(); } readableFile.close(); diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFileTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFileTest.java index 91ec94f09..2f3d408d6 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFileTest.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/internal/storage/files/WritableFileTest.java @@ -43,7 +43,7 @@ void setUp() throws IOException { new WritableFile( new File(rootDir, String.valueOf(CREATED_TIME_MILLIS)), CREATED_TIME_MILLIS, - TestData.getConfiguration(rootDir), + TestData.getConfiguration(), clock); } diff --git a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/testutils/BaseSignalSerializerTest.java b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/testutils/BaseSignalSerializerTest.java index 69186f812..d113d9f22 100644 --- a/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/testutils/BaseSignalSerializerTest.java +++ b/disk-buffering/src/test/java/io/opentelemetry/contrib/disk/buffering/testutils/BaseSignalSerializerTest.java @@ -37,7 +37,7 @@ protected byte[] serialize(SIGNAL_SDK_ITEM... items) { protected List deserialize(byte[] source) { try (ByteArrayInputStream in = new ByteArrayInputStream(source)) { StreamReader streamReader = DelimitedProtoStreamReader.Factory.getInstance().create(in); - return getDeserializer().deserialize(Objects.requireNonNull(streamReader.readNext()).content); + return getDeserializer().deserialize(Objects.requireNonNull(streamReader.readNext())); } catch (IOException e) { throw new RuntimeException(e); }