Merge branch 'main' into filtering

.github/workflows/test-build-push.yml

@@ -19,7 +19,7 @@ jobs:
           - label: Latest
             # we can't specify `tophu` for conda, not available on mac
             spec: >-
-              isce3
+              isce3-cpu
               dask
 
       fail-fast: true
@@ -51,7 +51,7 @@ jobs:
             "opera-utils>=0.4.1" \
             git+https://github.com/isce-framework/tophu@main \
             git+https://github.com/isce-framework/whirlwind@40defb38d2d6deca2819934788ebbc57e418e32d
-          python -m pip install git+https://github.com/scottstanie/spurt@use-mp-spawn
+          python -m pip install git+https://github.com/isce-framework/spurt@bd64ed4766a668fabdff4d312fcbb0d47344f632
           python -m pip install --no-deps .
       - name: Install test dependencies
         run: |

.pre-commit-config.yaml

@@ -26,13 +26,13 @@ repos:
         args: [--preview]
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.6.9
+    rev: v0.7.4
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix, --show-fixes]
 
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: "v1.11.2"
+    rev: "v1.13.0"
     hooks:
       - id: mypy
         additional_dependencies:

CHANGELOG.md

@@ -1,6 +1,54 @@
 # Changelog
 
-## [Unreleased](https://github.com/isce-framework/dolphin/compare/v0.28.0...main)
+# Changelog
+
+## [Unreleased](https://github.com/isce-framework/dolphin/compare/v0.30.0...main)
+
+## [0.30.0](https://github.com/isce-framework/dolphin/compare/v0.29.0...v0.30.0) - 2024-11-01
+
+### Added
+- Support for specifying output bounds as WKT
+- Ability to calculate similarity using nearest-3 interferograms
+- Add configuration option for single tile reoptimize functionality for snaphu-py. Turn off by default.
+- Support for layover shadow mask files to mask pixels during wrapped phase
+- Zero correlation threshold parameter for phase linking
+- Censored least squares solving capability for missing data
+- Support for reading from S3 using osgeo.gdal with `/vsis3` conversion
+
+### Changed
+- Updated dolphin timeseries CLI for new options, using L1 by default
+- Increased spurt `max_tiles` default to 49 for smaller MCF problems
+- Modified compressed_reference_idx with relative index fix
+- Replaced CSVs with `chi2.ppf` for GLRT test
+- Lowered `min_conncomp_frac` default to 0.001
+- Using `output_reference_idx` as default for creating compressed SLCs with `ALWAYS_FIRST`
+- Increased `buffer_pixels` when making OPERA CSLC mask
+- Refactored `repack_raster` to work in blocks for lower memory usage
+
+### Fixed
+- Fixed ministack output and compressed SLC indexing
+- Fixed `in_trim_slice` size for non-overlapping blocks
+- Fixed units passthrough during timeseries._redo_reference
+- Improved spurt subprocess handling to avoid fork issues
+- Fixed block_shape passthrough to create_similarities after phase linking
+
+## [0.29.0](https://github.com/isce-framework/dolphin/compare/v0.28.0...v0.29.0) - 2024-10-14
+
+### Added
+- Sample dolphin_config.yaml to the documentation
+- Stitched phase cosine similarity raster to outputs
+- Post-processing step to fill NaN gaps after running spurt
+- Outside input range options to get_nearest_date_idx
+
+### Changed
+- Set EVD input to be weighted by correlation
+- Using dataclasses instead of NamedTuple for displacement and stitched outputs
+- Passing through phase_linking.output_reference_idx to avoid reference resets
+
+### Fixed
+- Added logic to reset extra_reference_date after network unwrapping
+- Applied nan/0 mask to output of goldstein filter
+- Added references/url fixes from JOSS branch
 
 ## [0.28.0](https://github.com/isce-framework/dolphin/compare/v0.27.1...v0.28.0) - 2024-09-19
 

CONTRIBUTING.md

@@ -71,7 +71,7 @@ For bug fixes, a good practice is to write a test which fails with the current c
 ## Creating Documentation
 
 We use [MKDocs](https://www.mkdocs.org/) to generate the documentation.
-The reference documentation is generated from the code docstrings using [mkdocstrings](mkdocstrings.github.io/).
+The reference documentation is generated from the code docstrings using [mkdocstrings](https://mkdocstrings.github.io/python/)
 
 The dependencies for building and viewing the documentation locally can be installed:
 

README.md

@@ -2,6 +2,7 @@
 
 [![Pytest and build docker image](https://github.com/isce-framework/dolphin/actions/workflows/test-build-push.yml/badge.svg?branch=main)](https://github.com/isce-framework/dolphin/actions/workflows/test-build-push.yml)
 [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/isce-framework/dolphin/main.svg)](https://results.pre-commit.ci/latest/github/isce-framework/dolphin/main)
+[![Documentation Status][rtd-badge]][rtd-link]
 [![Conda-Forge][conda-badge]][conda-link]
 [![PyPI platforms][pypi-platforms]][pypi-link]
 [![GitHub Discussion][github-discussions-badge]][github-discussions-link]
@@ -13,6 +14,8 @@
 [pypi-platforms]:           https://img.shields.io/pypi/pyversions/dolphin
 [github-discussions-badge]: https://img.shields.io/static/v1?label=Discussions&message=Ask&color=blue&logo=github
 [github-discussions-link]:  https://github.com/isce-framework/dolphin/discussions
+[rtd-badge]:                https://readthedocs.org/projects/dolphin-insar/badge/?version=latest
+[rtd-link]:                 https://dolphin-insar.readthedocs.io/en/latest/?badge=latest
 <!-- prettier-ignore-end -->
 
 High resolution wrapped phase estimation for Interferometric Synthetic Aperture Radar (InSAR) using combined persistent scatterer (PS) and distributed scatterer (DS) processing.
@@ -21,21 +24,24 @@ High resolution wrapped phase estimation for Interferometric Synthetic Aperture
 
 ## Install
 
-`dolphin` is available on conda:
+`dolphin` may be installed via conda-forge:
 
 ```bash
 # if mamba is not already installed, see here: https://mamba.readthedocs.io/en/latest/
 mamba install -c conda-forge dolphin
 ```
 
-`dolphin` has the ability to unwrap interferograms using several options:
+It is also available via [`PyPI`](https://pypi.org/project/dolphin/) and may be `pip`-installed on some platforms, such as Google's Colab. However, certain dependencies (e.g. GDAL) are more easily set up through `conda`.
+
+`dolphin` has the ability to unwrap interferograms using several options, which can be toggled using the `unwrap_method` configuration option:
 
 1. [`snaphu-py`](https://github.com/isce-framework/snaphu-py), a lightweight Python bindings to [SNAPHU](https://web.stanford.edu/group/radar/softwareandlinks/sw/snaphu/). Available on both pip and conda.
 2. [`isce3`](https://github.com/isce-framework/isce3)'s python bindings to SNAPHU, PHASS, or ICU
-3. [`tophu`](https://github.com/isce-framework/tophu), a multi-scale unwrapper designed to unwrap large interferograms in parallel tiles at multiple resolution.
+3. [`spurt`](https://github.com/isce-framework/spurt), a 3D unwrapper, implementing the Extended Minimum Cost Flow (ECMF) algorithm
+4. [`tophu`](https://github.com/isce-framework/tophu), a multi-scale unwrapper designed to unwrap large interferograms in parallel tiles at multiple resolution.
 
-These may be installed via conda or (in the case of `snaphu-py`) pip.
 
+These may be installed via conda or (in the case of `snaphu-py`) pip.
 
 To install locally:
 
@@ -77,6 +83,8 @@ Example usage:
 
 ```bash
 dolphin config --slc-files /path/to/slcs/*tif
+# OR: to make a coarser output 4x as quickly:
+# dolphin config --slc-files /path/to/slcs/*tif --strides 2 2
 dolphin run dolphin_config.yaml
 ```
 
@@ -91,8 +99,6 @@ If the SLC files are spread over multiple files, you can either
 
 The full set of options is written to the configuration file; you can edit this file, or you can see which commonly tuned options by are changeable running `dolphin config --help`.
 
-See the [documentation](https://dolphin-insar.readthedocs.io/) for more details.
-
 ## Building and running via Docker
 
 `dolphin` can also be run using Docker. You can use the one built on [Github](https://github.com/isce-framework/dolphin/pkgs/container/dolphin), or build it locally using the script
@@ -101,7 +107,13 @@ See the [documentation](https://dolphin-insar.readthedocs.io/) for more details.
 ./docker/build-docker-image.sh
 ```
 
-See `./docker/build-docker-image.sh --help` for more building options.
+## Contributing
+
+We welcome many forms of contributing, including testing, bug reports, and documentation fixes. If you think you've found a problem, please let us know! You can raise an [issue](https://github.com/isce-framework/dolphin/issues) on the repository, where there are templates for Bug Reports and Feature Requests. If you have a general question of idea, feel free to raise it in the [Discussions](https://github.com/isce-framework/dolphin/discussions) page.
+
+For more detailed guidance on setting up a development environment, including how make and test changes to the code, see [Contributing to Dolphin](CONTRIBUTING.md).
+
+For more general Q&A, please use the [Discussions](https://github.com/isce-framework/dolphin/discussions) page.
 
 ## License
 

conda-env.yml

@@ -2,7 +2,7 @@ name: dolphin-env
 channels:
   - conda-forge
 dependencies:
-  - python>=3.8
+  - python>=3.9
   - pip>=21.3  # https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#editable-installation
   - git  # for pip install, due to setuptools_scm
   - gdal>=3.3

docs/getting-started.md

@@ -1,5 +1,5 @@
---8<-- "README.md:19:104"
+--8<-- "README.md:25:100"
 
-For a more complete tutorial, see our [tutorials page](tutorials.md)
+For a more complete tutorial, see our [notebooks](notebooks)
 
 For help with installing locally and contributing, see the [full developer setup instructions](developer-setup.md).

docs/index.md

@@ -6,6 +6,7 @@
 -->
 
 1. [Getting started](./getting-started.md)
+1. [Overview of processing modules](./overview.md)
 1. [Tutorials](tutorials.md)
 1. [Code Reference](reference/summary.md)
 1. [Changelog](changelog.md)

docs/overview.md

@@ -0,0 +1,16 @@
+
+# Overview of Dolphin
+
+Dolphin processes coregistered single-look complex (SLC) radar images into a time series of surface displacement. The software has an end-to-end surface displacement processing workflow, accessible through a command line tool, which calls core algorithms for PS/DS processing:
+
+- The `shp` subpackage estimates the SAR backscatter distribution to find neighborhoods of statistically homogeneous pixels (SHPs) using the generalized likelihood ratio test from @Parizzi2011AdaptiveInSARStack or the Kolmogorov-Smirnov test from @Ferretti2011NewAlgorithmProcessing.
+- The `phase_link` subpackage processes the complex SAR covariance matrix into a time series of wrapped phase using the CAESAR algorithm [@Fornaro2015CAESARApproachBased], the eigenvalue-based maximum likelihood estimator of interferometric phase (EMI) [@Ansari2018EfficientPhaseEstimation], or the combined phase linking (CPL) approach from @Mirzaee2023NonlinearPhaseLinking.
+- The `ps` module selects persistent scatterer pixels from the full-resolution SLCs to be integrated into the wrapped interferograms [@Ferretti2001PermanentScattersSAR].
+- The `unwrap` subpackage exposes multiple phase unwrapping algorithms, including the Statistical-cost, Network-flow Algorithm for Phase Unwrapping (SNAPHU) [@Chen2001TwodimensionalPhaseUnwrapping], the PHASS algorithm (available in the InSAR Scientific Computing Environment [@Rosen2018InSARScientificComputing]), and the Extended Minimum Cost Flow (EMCF) 3D phase unwrapping algorithm via the `spurt` library. Dolphin has pre- and post-processing options, including Goldstein filtering [@Goldstein1998RadarInterferogramFiltering] or interferogram masking and interpolation [@Chen2015PersistentScattererInterpolation].
+- The `timeseries` module contains basic functionality to invert an overdetermined network of unwrapped interferograms into a time series and estimate the average surface velocity. The outputs of Dolphin are also compatible with the Miami INsar Time-series software for users who are already comfortable with MintPy [@Yunjun2019SmallBaselineInSAR].
+
+To meet the computational demands of large-scale InSAR processing, Dolphin leverages Just-in-time (JIT) compilation, maintaining the readability of Python while matching the speed of compiled languages. The software's compute-intensive routines use the XLA compiler within JAX [@Bradbury2018JAXComposableTransformations] for efficient CPU or GPU processing. Users with compatible GPUs can see 5-20x speedups by simply installing additional packages. Dolphin manages memory efficiently through batch processing and multi-threaded I/O, allowing it to handle datasets larger than available memory while typically using a few gigabytes for most processing stages. These optimizations enable Dolphin to process hundreds of full-frame Sentinel-1 images with minimal configuration, making it well-suited for large-scale projects such as OPERA.
+
+![Overview of main workflow to generate surface displacement. Rectangular stacks indicate input or intermediate raster images. Arrows show the flow of data through the configurable submodules of Dolphin.](./figures/dolphin-modules.png)
+
+For more on running the `dolphin` command line tool, see the walkthroughs on the [Tutorials page](tutorials.md).

docs/references.bib

@@ -51,7 +51,6 @@ @article{Bekaert2021IntroducingOPERAProject
   urldate = {2024-05-15}
 }
 
-
 @article{Berardino2002NewAlgorithmSurface,
   title = {A {{New Algorithm}} for {{Surface Deformation Monitoring Based}} on {{Small Baseline Differential SAR Interferograms}}},
   author = {Berardino, Paolo and Fornaro, Dianfranco and Linari, Riccardo and Sansosti, Eugenio},
@@ -97,6 +96,7 @@ @article{Chen2001TwodimensionalPhaseUnwrapping
   urldate = {2024-05-16},
   langid = {english}
 }
+
 @article{Chen2012IonosphericArtifactsSimultaneous,
   title = {Ionospheric {{Artifacts}} in {{Simultaneous L-Band InSAR}} and {{GPS Observations}}},
   author = {Chen, Jingyi and Zebker, Howard A.},
@@ -125,14 +125,29 @@ @article{Chen2015PersistentScattererInterpolation
 }
 
 @article{Fattahi2019FRInGEFullResolutionInSAR,
-  title = {{{FRInGE}}; {{Full-Resolution InSAR}} Timeseries Using {{Generalized Eigenvectors}}},
-  author = {Fattahi, H. and Agram, P. S. and Tymofyeyeva, E. and Bekaert, D. P.},
-  year = {2019},
-  month = dec,
+  author = {{Fattahi}, H. and {Agram}, P.S. and {Tymofyeyeva}, E. and {Bekaert}, D.P.},
+  title = {{FRInGE; Full-Resolution InSAR timeseries using Generalized Eigenvectors}},
+  keywords = {1209 Tectonic deformation, GEODESY AND GRAVITY, 1211 Non-tectonic deformation, GEODESY AND GRAVITY, 1240 Satellite geodesy: results, GEODESY AND GRAVITY, 1241 Satellite geodesy: technical issues, GEODESY AND GRAVITY},
+  booktitle = {AGU Fall Meeting Abstracts},
+  year = 2019,
   volume = {2019},
+  month = dec,
+  eid = {G11B-0514},
   pages = {G11B-0514},
-  urldate = {2024-05-16},
-  keywords = {1209 Tectonic deformation,1211 Non-tectonic deformation,1240 Satellite geodesy: results,1241 Satellite geodesy: technical issues,GEODESY AND GRAVITY}
+  adsurl = {https://ui.adsabs.harvard.edu/abs/2019AGUFM.G11B0514F},
+  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}
+
+@article{Ferretti2001PermanentScattersSAR,
+  title = {Permanent {{Scatters}} in {{SAR Interferometry}}},
+  author = {Ferretti, Alessandro and Prati, Claudio and Rocca, Fabrio},
+  year = {2001},
+  journal = {IEEE Transactions on Geoscience and Remote Sensing},
+  volume = {39},
+  number = {1},
+  pages = {8--20},
+  issn = {01962892},
+  doi = {10.1109/36.898661}
 }
 
 @article{Ferretti2011NewAlgorithmProcessing,
@@ -304,6 +319,7 @@ @article{Yunjun2019SmallBaselineInSAR
   urldate = {2020-08-14},
   langid = {english}
 }
+
 @article{Zwieback2022CheapValidRegularizers,
   title = {Cheap, Valid Regularizers for Improved Interferometric Phase Linking},
   author = {Zwieback, S.},

docs/requirements.txt

@@ -1,7 +1,7 @@
 markdown
 mkdocs
 mkdocs-gen-files
-mkdocs-jupyter
+mkdocs-jupyter==0.25.0
 mkdocs-literate-nav
 mkdocs-material
 mkdocs-section-index

mkdocs.yml

@@ -56,6 +56,7 @@ watch:
 nav:
 - index.md
 - getting-started.md
+- overview.md
 - tutorials.md
 # - Tutorials:
 #   - Notebook page: notebooks/walkthrough-basic.ipynb

pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 name = "dolphin"
 description = "Workflows for generating surface displacement maps using InSAR"
 readme = { file = "README.md", content-type = "text/markdown" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
 
 classifiers = [
   "Development Status :: 2 - Pre-Alpha",