V2: Point cloud viewer with GeoParquet + DuckDB + deck.gl#21
Open
scottstanie wants to merge 60 commits intoopera-adt:mainfrom
Open
V2: Point cloud viewer with GeoParquet + DuckDB + deck.gl#21scottstanie wants to merge 60 commits intoopera-adt:mainfrom
scottstanie wants to merge 60 commits intoopera-adt:mainfrom
Conversation
Currently the colors are forgotten as you flip between datasets (e.g. switching to Magma when viewing temporal coherence overrides what you were using for the `timeseries` viewing
The _format_dates function was producing "%Y%m%d"-formatted strings
joined by underscores (e.g. "20250310_20250313"), which
chartjs-adapter-date-fns cannot parse. This caused all data points
to collapse to the same x position on the time series chart.
Changed to output ISO date strings ("2025-03-13") using the last date
from multi-date filenames, consistent with how Zarr mode already works.
https://claude.ai/code/session_01Gs7FnDKQu95PpxXRkpLCiB
When all COG files share the same first date (common for interferogram stacks), detect it and surface it to the frontend. The chart x-axis title shows "Time (ref: 2025-03-10)" so users know the reference date. https://claude.ai/code/session_01Gs7FnDKQu95PpxXRkpLCiB
The x_values check `any(d is None for d in dates)` didn't catch empty tuples from get_dates() on files without dates (e.g. velocity.tif). Changed to `any(not d for d in dates)` which handles both None and (). Added test_titiler.py with unit tests for _format_dates and integration tests for RasterGroup.x_values and .reference_date using test data. https://claude.ai/code/session_01Gs7FnDKQu95PpxXRkpLCiB
When multi-date files (unwrapped phase, coherence) have different reference dates, the secondary dates can repeat. Instead of using only the secondary date (which caused duplicate x-values and zigzag charts), use full "ref_secondary" date pair labels and render with a Chart.js category scale. - Remove unused _format_dates function and its tests - Detect duplicate secondary dates in x_values and fall back to date pairs - Add CategoryScale to Chart.js, auto-select scale type from label format - Clean up unused imports in test fixtures Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Design doc (BOWSER_V2_DESIGN.md) covering data model, architecture, UX, and implementation plan for raster + vector viewing - Pydantic manifest schema (bowser_manifest.json) for unified dataset config - Dolphin raster-to-GeoParquet converter with quality masking, spatial sorting (Morton curve), and proper row group layout - DuckDB-backed FastAPI router for point queries: bbox filtering, attribute filtering, single/multi-point time series, stats, Arrow IPC transport - CLI: `bowser convert dolphin` command, `--manifest` flag on `bowser run` - 11 passing tests covering converter and endpoint functionality Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- `bowser convert wide-parquet` command for sarlet-style point clouds where displacement dates are stored as wide columns (YYYYMMDD_YYYYMMDD) - Auto-detects date columns, separates static attrs from time series - Upcasts float16 → float32 for DuckDB compatibility - Tested against real 193k-point Mexico City dataset (16 dates) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- MapContainer: MapLibre GL JS replaces Leaflet, deck.gl ScatterplotLayer renders point clouds with WebGL, RdBu_r color scale, click-to-inspect - TimeSeriesChart: Plotly.js replaces Chart.js, dark theme, scattergl renderer for clicked point displacement time series - App: Auto-detects V2 point layer mode, fetches attributes for bounds - New usePointsApi hook: Arrow IPC decoding for point data, timeseries - AppContext: New state for activePointLayer, pointLayerBounds, clickedPoints - Raster tile layer support preserved in MapContainer for V1 compatibility - Removed Leaflet CDN dependency from index.html Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- PointControlsPanel: dark overlay with color-by attribute selector,
vmin/vmax inputs, point count, and mean display
- Dynamically populated from /points/{layer}/attributes endpoint
- Auto-sets symmetric vmin/vmax for velocity, full range for others
- MapContainer reads pointColorBy/pointVmin/pointVmax from global state
- App.tsx dispatches SET_POINT_ATTRIBUTES on init for the controls panel
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
ESRI satellite tiles max out around zoom 18-19. Google satellite tiles support up to zoom 21, which is needed when inspecting individual PS points at street level. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Basemap toggle (Satellite/OSM/Dark) in controls panel with reactive tile source switching. Attribute filter UI with add/remove expressions that chain via AND and pass through to DuckDB backend queries. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Shared colorscales module with 5 colormaps (rdbu_r, viridis, plasma, inferno, coolwarm). Colorbar gradient in controls panel shows current range. Colormap dropdown switches point colors in real-time. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Normal click selects a single point and shows its timeseries. Shift+click adds points to the selection for comparison. Selected points highlighted in yellow with larger radius. Chart shows a selection bar with per-point remove buttons and clear all. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Ctrl+click (or Cmd+click on Mac) sets a reference point shown in green. All displayed timeseries are differenced against the reference, with y-axis label changing to "Relative Displacement". Reference indicator in the chart bar with clear button. Green highlight + larger radius for the reference point on the map. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
DuckDB 1.5+ deprecated fetch_arrow_table(). Use .arrow().read_all() which returns a proper pyarrow.Table for IPC serialization. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
App now initializes BOTH raster datasets and point layers when available, instead of either/or. Layer visibility toggles let users show/hide raster and point layers independently. Point opacity slider controls deck.gl layer transparency. Backend /datasets endpoint returns empty dict gracefully in points-only mode. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Refactor load_data_sources() to load point layers from manifest AND raster data from BOWSER_STACK_DATA_FILE or BOWSER_DATASET_CONFIG_FILE when the manifest doesn't have raster layers. This enables running bowser with --manifest for points alongside an existing stack file for rasters. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
bowser generate-testdata -o DIR -n 50000 creates realistic clustered point distributions with velocity trends, temporal coherence, and amplitude dispersion. Outputs points.parquet + timeseries.parquet + bowser_manifest.json for immediate viewing. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
7 end-to-end tests that start a real Bowser server with synthetic data and drive a headless Chromium browser: - App loads with map canvas - Point controls panel visible with layer toggles - Colormap selector switches between options - Basemap switcher (satellite/OSM/dark) works - Stats overlay shows point count after data loads - Filter panel: add and clear attribute filters - Click on map triggers timeseries chart Uses pytest-playwright with session-scoped server fixture. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Client-side linear regression computes mm/yr rate and R² for each clicked point's timeseries. Trend button in chart header toggles dashed trend lines with rate shown in legend. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Backend GET /points/{layer}/export with format, bbox, filter, and
include_timeseries params. Frontend export buttons in controls panel
that open download in new tab with current filter applied. Three new
backend tests for CSV, GeoJSON, and Parquet export formats.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Date picker inputs in chart toolbar filter displayed timeseries client-side. Trend toggle shows dashed regression lines with mm/yr rate and R² in legend. Reset button clears date range. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
SVG-based mini histogram shows distribution of the current color-by attribute, colored by the active colormap. Histogram bins computed in MapContainer when point data loads, stored as lightweight bin counts in state. Updates when filter or color-by changes. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Replace Shapely Point loop with gpd.points_from_xy() (vectorized C) - Parallel raster reads with ThreadPoolExecutor for both static attributes and timeseries dates - Vectorize Morton key with numpy part1by1 bit-interleave instead of Python for-loop Should significantly reduce wall time for large datasets (10M+ points). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix OSM tile URL ({z}/{y}/{x} → {z}/{x}/{y})
- Remove V1 sidebar when point layer exists; fold raster controls
(dataset selector, time slider, opacity) into V2 controls panel
- Full-width map layout when points are active
- Add include-timeseries checkbox to export buttons
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Remove V1 ControlPanel sidebar entirely. The overlay panel now handles all modes: points-only, raster-only, or both. Raster controls (server colormap with colorbar image, vmin/vmax, opacity, time slider, show timeseries button) are nested under the raster layer toggle. Point sections hidden when no point layer is loaded. Full-width map layout in all modes. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Add response.ok guards to all point API hooks (prevents parsing HTML error pages as Arrow IPC in raster-only mode) - Fix filterAttr state-during-render issue with effectiveFilterAttr - Move chart date range + trend controls from absolute overlay to a dedicated toolbar row, eliminating overlap with Plotly legend Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Standalone script that starts a server with synthetic data and captures screenshots at key UI states: initial load, basemap switching, colormap change, filter applied, point click, trend lines, multi-select. Saves numbered PNGs to a timestamped directory for review. Usage: python tests/visual_test.py [--headed] [--n-points 50000] Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
View state (colorBy, vmin/vmax, colormap, basemap, filter, lat/lon/zoom) is serialized to URL search params and updated on every change via history.replaceState. On load, URL params override defaults, so shared links restore the exact same view. Map viewport syncs on moveend. Example: ?colorBy=velocity&vmin=-20&vmax=20&colormap=viridis&basemap=dark&lat=19.43&lon=-99.13&zoom=14 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Light theme: white background, serif font (Georgia), dark text, thin grid lines — suitable for academic papers. Toggle button in chart toolbar. Theme persists in URL state (?theme=light) for sharing. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Save PNG button in chart toolbar captures the map canvas and Plotly chart, composites them into a single image, and triggers a download. Uses dynamic import for Plotly.toImage to avoid bloating the main bundle. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Debounce URL viewport sync to prevent moveend from writing lat/lon/zoom before fitBounds runs (which caused fitBounds to be skipped) - Wait for map style load before adding raster tile source/layer (prevents silent failures from adding sources before style ready) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
New LosConfig and LosVectorConstant models in manifest. Supports two
LOS source types:
- constant: single ENU unit vector (e.g. {"east": 0.477, "north": -0.449, "up": 0.755})
- geotiff: 3-band GeoTIFF with spatially-varying E/N/U components
LOS config lives at manifest level (shared SAR geometry for all layers).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
New /gps router with endpoints:
- GET /gps/los_info — returns LOS config
- GET /gps/stations?bbox= — list GPS stations in viewport (via geepers)
- GET /gps/stations/{id}/timeseries — ENU→LOS projected timeseries
Supports constant LOS vector and spatially-varying GeoTIFF modes.
Returns both LOS-projected and raw ENU components for frontend toggle.
5 backend tests with mocked geepers (no network calls).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The tile URL path was /{endpoint}/WebMercatorQuad/tiles/{z}/{x}/{y}.png
but the titiler route is /{endpoint}/tiles/WebMercatorQuad/{z}/{x}/{y}.png.
This was a V2 regression that made all raster tiles 404.
Also change default map center from Mexico City to (0,0) so it's
obvious when fitBounds doesn't fire.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
In raster-only mode: - Click map to add a time series point (colored marker + pixel values fetched from /point endpoint, displayed in chart) - Draggable red reference marker for spatial referencing - Auto-fetch pixel timeseries for new points - "Show time series" button toggles chart visibility Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Reference marker now fetches pixel values on drag and dispatches SET_REF_VALUES for datasets with spatial referencing. This triggers raster tile re-rendering relative to the reference point (shift algorithm). Also fetches initial ref values on marker creation. Time series point markers are now draggable — on drag, position updates and cached data clears, triggering automatic re-fetch at the new location. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
When a dataset uses spatial referencing (uses_spatial_ref: true), the chart now subtracts state.refValues from the raw pixel values. This makes the time series relative to the reference marker position, consistent with how the raster tiles are rendered via the shift algorithm. Chart updates reactively when refValues change. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Export fixes: - Add preserveDrawingBuffer to MapLibre for canvas capture - Split into PNG (map+chart combined) and SVG (chart only, vector) - Better export proportions and 2x DPI scaling Chart improvements: - Light theme font: Palatino Linotype (nicer for publications) - Larger font size in light mode (13px vs 11px) Reference subtraction: - Fetch ref values for ALL datasets (not just uses_spatial_ref) - Always subtract refValues in chart when available - Both ref marker drag and TS point drag trigger reactive updates Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
useUrlStateSync was creating URLSearchParams from scratch, overwriting lat/lon/zoom written by the map moveend handler. Now reads existing params first and merges, so viewport and app state coexist in the URL. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
GPS types (GpsStation, GpsTimeseriesEntry, GpsComponent) and state in AppContext. useGpsApi hook. GPS section in controls panel with checkbox toggle, station count, LOS/E/N/U component buttons. Only shown when LOS config exists. App checks /gps/los_info on init. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Green ScatterplotLayer for GPS stations, fetched on moveend when GPS toggle enabled. Click station to select and fetch timeseries. Uses deckLayersRef for coordinated layer composition across effects. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
GPS trace rendered as dashed green line with diamond markers, visually distinct from InSAR traces. Respects LOS/E/N/U component selector. GPS trend line included when Trend toggled. GPS station tag in selection bar with dismiss button. Date range filter applies to GPS. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
bowser convert dolphin now auto-detects los_enu.json or los_enu*.tif
in the dolphin directory and writes LOS config into the manifest.
No manual configuration needed.
For raster mode, added --los flag to bowser run:
bowser run --los path/to/los_enu.json
bowser run --los '{"east": 0.477, "north": -0.449, "up": 0.755}'
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The only scipy usage was stats.linregress in calculate_trend(). Replaced with manual normal equations using numpy — same math, one fewer dependency. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- GPS stations: use row['id'] column (4-char station code) instead of DataFrame numeric index. Fixes "GPS: 0" display. Updated test mock to match real geepers GeoDataFrame format. - Raster pixel query: cast complex values to real before JSON serialization. Fixes TypeError with interferogram phase data. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
GPS: use df['date'] column instead of df.index for dates. Raster: restore localStorage persistence for colormap/vmin/vmax. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
GPS station clicks now suppress the raster click handler via a ref flag, preventing unwanted time series points when clicking GPS markers. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Scoping document for a standalone GPS station exploration and curation tool, proposed as a geepers subcommand. Covers the problem (raw GPS data needs preprocessing before InSAR comparison), the solution (interactive web app for browse/curate/reference/export), where it should live, technical architecture, and MVP scope. The output would be processed GeoParquet files loadable directly as bowser point layers — keeping bowser focused on viewing. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bowser V2: Point Cloud + Raster Unified Viewer
What's new
/gpsrouter with geepers integration, ENU→LOS projection (constant or GeoTIFF), 5 backend testsbowser convert dolphin(3x faster with parallelized I/O),bowser convert wide-parquet,bowser generate-testdataCommits (36)
Foundation:
Interaction:
5. Basemap switcher (satellite/OSM/dark) + attribute filter panel
6. Colorbar legend + colormap selector (rdbu_r, viridis, plasma, inferno, coolwarm)
7. Multi-point selection (shift+click) with bulk timeseries
8. Reference point subtraction (ctrl+click) for point cloud mode
Raster + vector unification:
9. Unified init — loads both raster and point layers simultaneously
10. Consolidated single controls panel (removed V1 sidebar)
11. Raster tile URL fix, style load guard, reference marker + TS click restored
12. Reference marker fetches pixel values, TS markers draggable, chart subtracts ref
Analysis & export:
13. Trend line toggle (client-side linear regression)
14. Export endpoint + UI (CSV/GeoJSON/GeoParquet with optional timeseries)
15. Date range filter for timeseries
16. Attribute histogram (SVG mini-chart in controls panel)
Shareability:
17. URL state encoding (viewport, colormap, filter — shareable links)
18. Dark/light chart theme toggle (Palatino serif for publication)
19. PNG export (map+chart combined, 2x DPI) + SVG export (vector)
GPS overlay (backend):
20. LOS config schema in manifest (constant JSON or 3-band GeoTIFF)
21. GPS router with geepers integration + ENU→LOS projection + 5 tests
Testing & tools:
22. Playwright E2E browser tests (7 tests)
23. Visual screenshot test script
24.
bowser generate-testdataCLI command25. Parallelized dolphin converter (3x speedup)
Test results
How to test
🤖 Generated with Claude Code