get_seed fallback

Author: samcwiley

CHANGELOG.md

@@ -9,9 +9,15 @@ is roughly based on [Keep a Changelog], and this project tries to adheres to
 
 ### Changed
 
-- Combined `IterWithContext`, `ReaderWithContext`, and `WriterWithContext` into a single `WithContext` struct
-- The default method for `aligner` is now the 3-pass algorithm which has a more stable runtime and memory performance for large sequences. This may result in different optimal alignments
-- The profile for alignment is now built from the reference by default when the 1-pass algorithm is enabled. This may result in different optimal alignments
+- Combined `IterWithContext`, `ReaderWithContext`, and `WriterWithContext` into
+  a single `WithContext` struct
+- The default method for `aligner` is now the 3-pass algorithm which has a more
+  stable runtime and memory performance for large sequences. This may result in
+  different optimal alignments
+- The profile for alignment is now built from the reference by default when the
+  1-pass algorithm is enabled. This may result in different optimal alignments
+- `sampler` now uses `IRMA_SEED` as a fallback if no `--rng-seed` is provided,
+  allowing reproducibility in sampling runs
 
 ### Removed
 
@@ -22,8 +28,11 @@ is roughly based on [Keep a Changelog], and this project tries to adheres to
 
 ### Added
 
-- Added preliminary implementations for `FastX` views (within `irma-records` public API)
-- A `ValidatePaths` trait is now available in the library portion for ensuring no path is passed as both an input and an output, and that all output paths are distinct
+- Added preliminary implementations for `FastX` views (within `irma-records`
+  public API)
+- A `ValidatePaths` trait is now available in the library portion for ensuring
+  no path is passed as both an input and an output, and that all output paths
+  are distinct
 - `WriteRecord` is now compatible with `FastX`
 
 ### Changed

crates/irma-core-cli/processes/shared/mod.rs

@@ -23,7 +23,7 @@ pub(crate) trait PrintWarning {
     /// `stdout` or `stderr`.
     ///
     /// The message includes a timestamp, indentation based on the shell level
-    /// (`SHLVL` environmental variable), and the word `WARNING`.
+    /// (`SHLVL` environment variable), and the word `WARNING`.
     fn warn(&self, program: &str, message: &str, use_stderr: bool);
 }
 

crates/irma-core-cli/processes/standalone/sampler.rs

@@ -2,6 +2,7 @@
 
 use clap::Args;
 use irma_records::{
+    hashing::get_seed,
     io::{
         DispatchFastX, FastXReader, FastXType, InputOptions, OutputOptions, ReadFileZipInThread, RecordReaders,
         RecordWriters, SequenceWriter, ValidatePaths, WithContext, WriteFileZipStdout, WriteRecord,
@@ -48,7 +49,8 @@ pub struct SamplerArgs {
 
     #[arg(short = 's', long)]
     /// For reproducibility, provide an optional seed for the random number
-    /// generator
+    /// generator. If not provided, checks `IRMA_SEED` environment variable, and
+    /// finally falls back to a random seed
     pub rng_seed: Option<u64>,
 
     #[arg(short = 'v', long)]
@@ -402,6 +404,8 @@ enum SamplingTarget {
 fn parse_sampler_args(args: SamplerArgs) -> Result<(IOArgs, Xoshiro256StarStar, SamplingTarget, bool), std::io::Error> {
     let rng = if let Some(seed) = &args.rng_seed {
         Xoshiro256StarStar::seed_from_u64(*seed)
+    } else if let Some(seed) = get_seed() {
+        Xoshiro256StarStar::seed_from_u64(seed)
     } else {
         make_rng()
     };

crates/irma-records/hashing.rs

@@ -3,13 +3,50 @@ use std::env;
 
 const SEED_ENV_VAR: &str = "IRMA_SEED";
 
-fn get_seed() -> Option<u64> {
-    env::var(SEED_ENV_VAR).ok().map(|s| s.bytes().fold(0, |a, b| a ^ b) as u64)
+/// Attempts to parse the environment variable `IRMA_SEED` into a `u64` to use
+/// as a seed. If a seed exists but is unable to be parsed, this falls back to a
+/// hashing algorithm.
+pub fn get_seed() -> Option<u64> {
+    env::var(SEED_ENV_VAR)
+        .ok()
+        .map(|value| value.parse::<u64>().unwrap_or_else(|_| seed_from_string(&value)))
 }
 
+fn seed_from_string(value: &str) -> u64 {
+    let mut result = [0u8; 8];
+
+    let (chunks, remainder) = value.as_bytes().as_chunks::<8>();
+    // takes the seed in chunks of 8 bytes, and XORs the ith byte of all chunks
+    // against eachother
+    for chunk in chunks {
+        for (position, byte) in chunk.iter().enumerate() {
+            result[position] ^= byte;
+        }
+    }
+    // handle the bytes in the remainder
+    for (position, byte) in remainder.iter().enumerate() {
+        result[position] ^= byte;
+    }
+    // then folds the bytes back into a u64 by shifting each byte by 8*i, then
+    // ORing it against the accumulator
+    result
+        .into_iter()
+        .enumerate()
+        .fold(0u64, |seed, (position, byte)| seed | (u64::from(byte) << (8 * position)))
+}
+
+/// Creates a hasher based on the seed provided in the environment variable
+/// `IRMA_SEED`, or falls back to creating a random one.
 pub fn get_hasher() -> SeedableRandomState {
     match get_seed() {
         Some(seed) => SeedableRandomState::with_seed(seed, SharedSeed::global_fixed()),
         None => SeedableRandomState::random(),
     }
 }
+
+#[test]
+fn test_seed_from_string() {
+    // i worked this one out by hand
+    let string = "ACGTACGTTGCATGCA";
+    assert_eq!(seed_from_string(string), 1514339863296738325);
+}

docs/ALIGNER.md

@@ -6,7 +6,7 @@ IRMA-core's `aligner` provides an efficient and exact local sequence alignment r
 
 ## Multithreading
 
-`aligner` uses `rayon` to perform multithreading to enable higher throughput. To specify the number of threads, set the `RAYON_NUM_THREADS` environmental variable as described [in rayon](https://docs.rs/rayon/latest/rayon/fn.max_num_threads.html). Or, to limit to a single worker thread, pass `--single-thread` to `aligner`.
+`aligner` uses `rayon` to perform multithreading to enable higher throughput. To specify the number of threads, set the `RAYON_NUM_THREADS` environment variable as described [in rayon](https://docs.rs/rayon/latest/rayon/fn.max_num_threads.html). Or, to limit to a single worker thread, pass `--single-thread` to `aligner`.
 
 For benchmarking or scenarios where a single thread is always used, the `dev_no_rayon` feature can be enabled in IRMA-core to remove the use of channels. This feature may be removed in future releases, and so should not be relied upon except for testing.
 

docs/SAMPLER.md

@@ -4,20 +4,33 @@
 
 Next generation sequencers often produce an excess of reads, for reasons such as to create redundancy for enabling consensus generation, due to requiring a minimum amount of reagents per run, or other reasons. Running the full read set through a pipeline can be computationally expensive, so it may be desirable to transform the data into a smaller subset to save time.
 
-IRMA-core's sampler process provides efficient, random, and fully-representative downsampling (also referred to as subsampling), as well as some other useful functionality.
+IRMA-core's `sampler` process provides efficient, random, and fully-representative downsampling (also referred to as subsampling), as well as some other useful functionality.
 
 ## Downsampling Targets
 
-IRMA-core's sampler requires a target for downsampling. This can either be provided as `--subsample-target`, which is the exact number of reads to be in the subsampled output, or `--percent-target`, which is the percentage of the original amount of reads to be in the subsampled output.
+IRMA-core's `sampler` requires a target for downsampling. This can either be provided as `--subsample-target`, which is the exact number of reads to be in the subsampled output, or `--percent-target`, which is the percentage of the original amount of reads to be in the subsampled output.
 
 Percent targets must be provided as an integer [0-100] and may not provide an exact percentage downsampled in the cases of streamed or compressed input.
 
 - If a `--percent-target` of 100 is provided, no downsampling will occur. This could be useful for de-interleaving without downsampling.
 - If a `--subsample-target` is provided that is *greater* than the amount of sequences in the input, the process will succeed and give an output that is identical to the input, but provide a warning for the user.
 
+## Random number generation and seeds
+
+`sampler` uses the `Xoshiro256StarStar` random number generator, which requires
+a seed, which will be accessed in the following order:
+
+1. A seed may be provided via `--rng-seed` in a `u64` format
+2. If that `--rng-seed` is not provided, `sampler` will check the `IRMA_SEED`
+environment variable, and if it is set, attempt to parse it into a `u64`
+3. If `IRMA_SEED` is unable to be parsed, `sampler` will run the seed through a
+hashing function
+4. Lastly, if `--rng-seed` is not provided, and `IRMA_SEED` is not set, a random
+seed will be generated from the system
+
 ## Inputs and Outputs
 
-Sampler can downsample `FASTQ` and `FASTA` formats. Inputs are provided as positional arguments, with sampler accepting either a single file, or as a pair of paired-read files. The files may also be a stream (e.g., from a process substitution) or a `.gz` compressed file.
+`sampler` can downsample `FASTQ` and `FASTA` formats. Inputs are provided as positional arguments, with `sampler` accepting either a single file, or as a pair of paired-read files. The files may also be a stream (e.g., from a process substitution) or a `.gz` compressed file.
 
 For outputs, you can select one output file with `-o` (`--output`) or two output files with `-1` and `-2` (`--output` and `--output2`). If no output is provided, IRMA-core will output the subsampled data to `stdout`.