Skip to content

Commit 53c92ed

Browse files
gaelsogaelso
committed
updated AI context
1 parent c2f2b92 commit 53c92ed

2 files changed

Lines changed: 330 additions & 6287 deletions

File tree

‎data-raw/ai-context.md‎

Lines changed: 330 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,330 @@
1+
# `arenalytics.dev` — AI Agent Context
2+
3+
> Last updated: 2026-04-16
4+
> Purpose: Technical briefing for an AI agent acting as expert R / Shiny developer continuing work on this package.
5+
6+
---
7+
8+
## What the Package Does
9+
10+
`arenalytics.dev` is an R package that ships a Shiny dashboard for analysing ecological/forestry survey data produced by **OpenForis Arena** — a field data collection and processing platform used mainly in national forest inventories (NFIs).
11+
12+
Workflow:
13+
1. A user runs a processing chain in Arena, which outputs a ZIP file of semi-aggregated OLAP data.
14+
2. This package reads that ZIP, displays data insights (dimension frequency tables, measure summaries), and computes design-based statistical estimates (survey means per ha and totals) using `srvyr`.
15+
16+
---
17+
18+
## Repository Structure
19+
20+
```
21+
arenalytics.dev/
22+
├── R/
23+
│ ├── shiny_run_arenalytics_dev.R # App entry point (exported function)
24+
│ ├── fct_checkzip.R # Validate ZIP before reading
25+
│ ├── fct_readzip2.R # Read and parse the ZIP (main loader)
26+
│ ├── fct_varinfo.R # Extract & classify column metadata per entity
27+
│ ├── fct_arenalyse.R # Survey estimation (means + totals)
28+
│ ├── fct_mean.R # Small utility for value box rendering
29+
│ ├── fct_readzip.R # Old flat-return loader (superseded, kept for reference)
30+
│ ├── mod_tool_UI2.R # ACTIVE tool module UI
31+
│ ├── mod_tool_server2.R # ACTIVE tool module server
32+
│ ├── mod_tool_UI.R / server.R # Older versions (kept for reference, NOT wired in)
33+
│ ├── mod_home_UI/server.R # Landing page module
34+
│ ├── mod_about_UI/server.R # About page + demo file download
35+
│ ├── utils.R # fct_get_dim_meta() — fully commented out, superseded
36+
│ └── utils-tr.R # i18n key registry (.tr_keys())
37+
├── inst/
38+
│ ├── extdata/
39+
│ │ ├── OLAP_Shiny_demo.zip # Demo dataset (uses OLAP_ prefix, for dev/testing only)
40+
│ │ └── OLAP_Shiny_demo_broken.zip
41+
│ └── assets/ # CSS, JS, logo, favicon, translations.json
42+
├── tests/
43+
├── app.R # Thin wrapper: pkgload::load_all() + run
44+
└── DESCRIPTION
45+
```
46+
47+
Only `mod_tool_UI2` / `mod_tool_server2` are wired into the live app. The `*UI.R` / `*server.R` (no suffix `2`) are legacy and can be ignored.
48+
49+
---
50+
51+
## The ZIP Data Format
52+
53+
The input is a ZIP produced by an Arena processing chain.
54+
55+
| File | Format | Description |
56+
|---|---|---|
57+
| `MAU_<entity>.csv` | CSV | Wide-format OLAP table per entity (e.g. `MAU_tree`, `MAU_bamboo`) |
58+
| `chain_summary.json` | JSON | Survey metadata: sampling strategy, base unit, clustering entity, stratum attribute, result variable definitions, selected language |
59+
| `SchemaSummary.csv` | CSV | Data dictionary: all input dimension columns with types, labels, category names |
60+
| `ReportDimensions.csv` | CSV | Which dimensions are available for reporting per entity |
61+
| `categories.rds` | RDS | Named list of category tables; each maps `code → label` + per-language columns (e.g. `label_en`) |
62+
| `taxonomies.rds` | RDS | Taxon lookup tables for species dimensions |
63+
64+
**Entity prefix**: production ZIPs use `MAU_` prefix; the bundled demo uses `OLAP_`. The prefix is stored in `.ep <- "MAU_"` at the top of `mod_tool_server2` and threaded through the entire pipeline via the `.entity_prefix` argument.
65+
66+
---
67+
68+
## Core Functions
69+
70+
### `fct_checkzip(.path, .entity_prefix)`
71+
72+
Validates ZIP structure without reading data. Returns `list(all_ok = TRUE/FALSE, missing = character())`. Gates the "Read data" button in the UI.
73+
74+
---
75+
76+
### `fct_readzip2(.path, .pb_session, .pb_id, .entity_prefix = "MAU_")`
77+
78+
Main data loader. Reads every file inside the ZIP individually in `tryCatch`, emitting timestamped `message()` calls captured by the Shiny progress console div via `withCallingHandlers`.
79+
80+
**Name normalisation**: file names are lowercased and camelCase-split, then the entity prefix is re-capitalised:
81+
```r
82+
file_names <- tolower(gsub("([a-z0-9])([A-Z])", "\\1_\\2", file_names))
83+
file_names <- stringr::str_replace_all(file_names, tolower(.entity_prefix), .entity_prefix)
84+
# e.g. "MAU_tree.csv" → "mau_tree" → "MAU_tree"
85+
```
86+
87+
**Returns**:
88+
```r
89+
list(
90+
data = list( # flat named list keyed by normalised file names
91+
chain_summary = <list>, # parsed JSON
92+
schema_summary = <data.frame>,
93+
report_dimensions = <data.frame>,
94+
categories = <list of data.frames>,
95+
taxonomies = <list>,
96+
MAU_tree = <data.frame>, # OLAP entity table
97+
MAU_bamboo = <data.frame>,
98+
...
99+
),
100+
errors = character(0), # named vector: file → error message
101+
var_meta = list( # pre-computed fct_varinfo() result per entity
102+
tree = <tibble>,
103+
bamboo = <tibble>,
104+
...
105+
)
106+
)
107+
```
108+
109+
> **Note**: if any file read errors occurred, `var_meta` is `NULL`.
110+
111+
---
112+
113+
### `fct_varinfo(.zip, .entity, .entity_prefix = "MAU_")`
114+
115+
Called with the **inner flat data** (`rv$inputs$data`). Builds a tibble describing every column of the entity OLAP table by merging `schema_summary` (input dims) with `chain_summary$resultVariables` (computed dims + measures), then filtering to columns that actually appear in the OLAP table.
116+
117+
**Returns a tibble with one row per column:**
118+
119+
| Column | Description |
120+
|---|---|
121+
| `name` | Column name in the OLAP table |
122+
| `label` | Human-readable label (language-aware) |
123+
| `report_type` | `"dimension"` or `"measure"` (NA for weight/internal cols) |
124+
| `type` | `"code"`, `"numeric"`, etc. |
125+
| `categoryName` | Key into `categories` list for code→label lookup |
126+
| `parentEntity` | Entity the column belongs to |
127+
| `dimension_baseunit` | `TRUE` = base-unit level; `FALSE` = sub-unit dim (e.g. tree species is sub-unit of plot) |
128+
| `stratum` | `TRUE` if this column is the stratification variable |
129+
| `categoryType` | `"F"` flat or `"H"` hierarchical (square brackets in `categoryName`) |
130+
131+
Also called by `fct_readzip2` to pre-populate `var_meta` at load time, and by `fct_arenalyse` at analysis time.
132+
133+
---
134+
135+
### `fct_arenalyse(.zip, .entity, .dim)`
136+
137+
Survey estimation engine. Called with the **inner flat data** (`rv$inputs$data`).
138+
139+
**Steps:**
140+
1. Calls `fct_varinfo` internally to get column metadata and dynamically detect the entity prefix.
141+
2. Reads sampling design from `chain_summary$samplingStrategy`: simple SRS (1–2), stratified SRS (3–4), or cluster.
142+
3. Classifies user-selected dims as base-unit vs sub-unit.
143+
4. Expands base-unit × sub-unit combinations with `tidyr::complete()` when sub-unit dims are included.
144+
5. Builds a `srvyr` survey design with `ids` (cluster UUID if clustered), `strata` (stratum column if stratified), `weights` (`exp_factor_`).
145+
6. Computes `survey_mean()` with SE + CI for each measure via `purrr::map` (one measure at a time).
146+
7. Joins expansion areas (`exp_factor_` sum per base-unit dim combination) via a `JOIN_COL` key.
147+
8. Derives totals by multiplying means × area.
148+
9. Cleans column names (strips `srvyr` `_1_` artefact suffix), floors negative CI lower bounds at zero.
149+
150+
**Returns** `list(MEANS = <tibble>, TOTALS = <tibble>)`.
151+
152+
Output columns: dimension columns, `JOIN_COL` (pipe-separated base-unit dim values), `<measure>`, `<measure>_se`, `<measure>_low`, `<measure>_upp`, `area`, `base_unit_count`, `item_count`, optionally `cluster_count`.
153+
154+
> **Important**: all measure columns in the raw OLAP table are stored as **character**. `fct_arenalyse` casts them to numeric internally. The server also calls `as.numeric()` when computing insight summaries.
155+
156+
---
157+
158+
## Shiny App Architecture
159+
160+
### Entry Point
161+
162+
`shiny_run_arenalytics_dev()` in `R/shiny_run_arenalytics_dev.R`. UI and server are defined inside this single function (golem-inspired inline structure). `app.R` at repo root calls `pkgload::load_all()` then this function, enabling RStudio's Run App button.
163+
164+
### Top-Level Layout
165+
166+
`bslib::page_navbar()` with Bootstrap 5 / Bootswatch **Yeti** theme. Three nav panels: **Home**, **Tool**, **About**. A `shinyWidgets::pickerInput` language selector (EN/FR/ES) in the navbar drives `shiny.i18n` translations.
167+
168+
### Reactive Values (`rv`)
169+
170+
A nested `reactiveValues` structure passed to every module server:
171+
172+
```r
173+
rv <- reactiveValues(
174+
inputs = reactiveValues(),
175+
# After fct_readzip2(): rv$inputs$data = inner flat data list
176+
# rv$inputs$var_meta = pre-computed fct_varinfo per entity
177+
# rv$inputs$errors = read error vector
178+
# rv$inputs$data_ok = TRUE/FALSE
179+
# rv$inputs$path_zip = uploaded file path
180+
181+
insights = reactiveValues(),
182+
# rv$insights$entities_named = setNames(entities, entities_labs)
183+
# rv$insights$bu_choices = setNames(name, label) for base-unit dims
184+
# rv$insights$sub_choices = setNames(name, label) for sub-unit dims
185+
# rv$insights$meas_choices = setNames(name, label) for measures
186+
# rv$insights$entity_table = rv$inputs$data[[paste0(.ep, entity)]]
187+
188+
analysis = reactiveValues(),
189+
# rv$analysis$dim_meta = fct_varinfo result for selected entity
190+
# rv$analysis$strat_label = label of stratum column (or NULL)
191+
# rv$analysis$measures_meta = dim_meta filtered to report_type == "measure"
192+
# rv$analysis$result = list(MEANS, TOTALS) after label replacement
193+
# rv$analysis$dims = character vector of selected dim column names
194+
# rv$analysis$entity = selected entity name (e.g. "tree")
195+
196+
ct = reactiveValues(), # crosstalk test state (demo only, dead code)
197+
actions = reactiveValues() # cross-module navigation: to_tool, to_about
198+
)
199+
```
200+
201+
### Custom JS Handlers
202+
203+
Two scripts in `inst/assets/`:
204+
- `js_activate_tab.js` — Shiny message handler `"activate-tab"`: programmatically switches the active tab (`session$sendCustomMessage("activate-tab", list(id=ns("tool_tabs"), value="tab_analysis"))`).
205+
- `js_handlers.js` — `"scroll_top"` handler.
206+
207+
---
208+
209+
## Tool Module — Detailed
210+
211+
### UI (`mod_tool_UI2`)
212+
213+
`navset_card_tab` with a 300px `sidebar` and two main panels.
214+
215+
**Sidebar accordions:**
216+
217+
- **Acc1 "Load ZIP file"** (`ac1`): `fileInput` → validation message → "Read data" `actionButton`.
218+
- ~~Acc2~~ — removed; entity/variable selection moved into the Insights panel.
219+
- **Acc3 "Run analysis"** (`ac3`):
220+
- `uiOutput("analysis_entity")` — `selectInput` populated server-side from entity list
221+
- `uiOutput("analysis_dims")` — two `checkboxGroupButtons(individual=TRUE, size="sm")` (base-unit dims then `hr()` then sub-unit dims), rendered server-side
222+
- `uiOutput("analysis_strat_text")` — italic `text-info` note when a stratum is auto-included
223+
- `uiOutput("analysis_too_many_dims")` — italic `text-warning` note when > 4 dims selected
224+
- `actionButton("btn_run_analysis")` — disabled until ≥ 1 dim checked
225+
226+
**Panel: Insights** (`tab_insights`):
227+
228+
Three states managed by `shinyjs::show/hide`: initial message → progress (progressBar + live console div) → data insights.
229+
230+
Data insights layout:
231+
- `selectInput("insight_sel_entity")` for entity selection
232+
- Three `card()` elements using `layout_columns(col_widths=c(6,6))`:
233+
- **Base-unit dims**: left = `checkboxGroupButtons("insight_bu_sel", individual=TRUE)`, right = `uiOutput("insight_bu_out")`
234+
- **Sub-unit dims**: same pattern with `"insight_sub_sel"` / `"insight_sub_out"`
235+
- **Measures**: same pattern with `"insight_meas_sel"` / `"insight_meas_out"`
236+
237+
**Panel: Analysis** (`tab_analysis`):
238+
239+
- Plot controls card: `selectInput` for x-axis dim, measure, fill, facet + error bar `checkboxInput`; dynamic `virtualSelectInput` filters per dimension (`uiOutput("analysis_extra_filters")`)
240+
- Two `plotOutput`s: "Means (per ha)" and "Totals"
241+
242+
---
243+
244+
### Server (`mod_tool_server2`)
245+
246+
**Key constant at the top:**
247+
```r
248+
.ep <- "MAU_" # entity table prefix in the ZIP
249+
```
250+
251+
**Dev/test setup** (lines 16–27 in the server, commented out):
252+
```r
253+
# rv$inputs <- fct_readzip2(.path = "...", .entity_prefix = .ep)
254+
# input <- list(analysis_sel_entity = "tree", ...)
255+
# result <- fct_arenalyse(.zip = rv$inputs$data, .entity = "tree", .dim = c(...))
256+
```
257+
Run these lines in the R console to set up `rv` and `input` for interactive debugging without launching the app.
258+
259+
**Data loading sequence:**
260+
1. `observeEvent(input$load_zip)` → `fct_checkzip()` → toggle messages/button
261+
2. `observeEvent(input$btn_read_data)` → `fct_readzip2()` captured with `withCallingHandlers` → `rv$inputs <- <result>` → set `rv$inputs$data_ok`
262+
3. `observeEvent(input$btn_data_insights)` → show insights panel
263+
4. `observe({ req(rv$inputs$data) })` → populate `rv$insights$entities_named` from `names(rv$inputs$data) |> str_subset(.ep) |> str_remove(.ep)`
264+
265+
**Analysis sequence:**
266+
1. `observeEvent(input$analysis_sel_entity)` → `rv$analysis$dim_meta <- rv$inputs$var_meta[[entity]]`, detect stratum label
267+
2. `output$analysis_dims` (renderUI) → two `checkboxGroupButtons` from `dim_meta`
268+
3. `observe` → toggle run button when `isTruthy(input$analysis_bu_dims) || isTruthy(input$analysis_sub_dims)`
269+
4. `observeEvent(input$btn_run_analysis)`:
270+
- `dims_sel <- c(input$analysis_bu_dims, input$analysis_sub_dims)`
271+
- `fct_arenalyse(.zip = rv$inputs$data, .entity = ..., .dim = dims_sel)`
272+
- **Apply `replace_dim_labels()`** to both `result$MEANS` and `result$TOTALS` before storing
273+
- Store in `rv$analysis$result`, `rv$analysis$dims`, `rv$analysis$entity`
274+
275+
**Local helper functions defined inside `moduleServer`:**
276+
277+
- **`replace_dim_labels(df, dim_meta, categories, lang)`**: replaces dimension codes with human-readable labels. Iterates dimension columns present in `df` via `purrr::reduce`; looks up `categoryName` from `dim_meta`, finds the matching table in `categories`, maps `code → label_<lang>` (falls back to `label`). Applied to `MEANS` and `TOTALS` right after `fct_arenalyse()` returns.
278+
279+
- **`make_dim_summary(sel, choices, tbl)`**: renders a `tags$pre()` with `table()` output per selected dimension. Uses `options(width = 60)` temporarily to force paired label/value block wrapping (console style).
280+
281+
- **`make_meas_summary(sel, choices, tbl)`**: renders a `tags$pre()` with `summary(as.numeric(...))` per selected measure.
282+
283+
- **`make_bar_plot(df, x_dim, measure, fill_col, facet_col, show_errbar, dim_meta, measures_meta, extra_filter_vals, comma_y)`**: shared ggplot2 bar chart builder. Handles optional fill (dodged bars), optional facet, optional CI error bars, dimension filters, and comma-formatted y-axis for totals.
284+
285+
**Insight summary outputs** (`output$insight_bu_out`, `output$insight_sub_out`, `output$insight_meas_out`): each is `renderUI` gated on `req(rv$insights$<choices>, rv$insights$entity_table)`.
286+
287+
**Analysis plot selectors**: `observeEvent(rv$analysis$result)` updates `plot_dim`, `plot_measure`, `plot_fill`, `plot_facet` selectors and reveals the results div.
288+
289+
**Dimension filters** (`output$analysis_extra_filters`): one `virtualSelectInput` per dim in `rv$analysis$dims`, defaulting all values selected (no filter applied). Filter values are label strings post `replace_dim_labels()`.
290+
291+
---
292+
293+
## Key Architectural Notes and Known Issues
294+
295+
1. **`rv$inputs` vs `rv$inputs$data`**: After loading, `rv$inputs` is the full `fct_readzip2()` return value (`list(data, errors, var_meta)`). So `rv$inputs$data` is the inner flat list with `MAU_tree` etc. Always access entity tables as `rv$inputs$data[[paste0(.ep, entity)]]` and metadata as `rv$inputs$var_meta[[entity]]`.
296+
297+
2. **`fct_arenalyse` data path**: called with `.zip = rv$inputs$data` (the flat inner list). It dynamically detects the entity prefix via `stringr::str_subset(names(.zip), .entity)` — this is why it works regardless of prefix.
298+
299+
3. **Measures stored as character**: all measure columns in OLAP tables are character. Cast to numeric before any arithmetic (`as.numeric()`). `fct_arenalyse` handles this internally; `make_meas_summary` does it explicitly.
300+
301+
4. **Dual module files**: `mod_tool_UI.R` / `mod_tool_server.R` are legacy, not wired in. `mod_tool_UI2.R` / `mod_tool_server2.R` are the active versions.
302+
303+
5. **`fct_get_dim_meta`** in `utils.R` is fully commented out — superseded by `fct_varinfo`.
304+
305+
6. **i18n**: `shiny.i18n` drives UI text via `i18n$t(.tr$key)`. Keys centralised in `utils-tr.R → .tr_keys()`. Translation JSON at `inst/assets/translations.json`. Only UI-level text is translated; server-generated strings are mostly English.
306+
307+
7. **`%||%` operator**: used for NULL fallbacks (e.g. `lang <- rv$inputs$data$chain_summary$selectedLanguage %||% "en"`). Comes from `rlang`, imported via `@importFrom rlang .data` in the NAMESPACE.
308+
309+
8. **Demo ZIP vs production ZIP**: `OLAP_Shiny_demo.zip` uses `OLAP_` prefix (lowercased to `olap_` then not re-capitalised since `.ep = "MAU_"`). This means `var_meta` ends up empty when testing with the demo ZIP in an actual running app (the server sets `.ep <- "MAU_"`). Use the test setup lines in the server comments, pointing to a real MAU ZIP, for full end-to-end testing.
310+
311+
9. **Dead code — crosstalk panel**: the old UI had an `ac4` accordion and a crosstalk nav panel (d3scatter linked plots). These have been removed from the `UI2`/`server2` active files. Some crosstalk output code may remain commented out in the server.
312+
313+
---
314+
315+
## Dependencies (key)
316+
317+
| Package | Role |
318+
|---|---|
319+
| `bslib` | Bootstrap 5 layout, theming, cards, sidebars |
320+
| `shinyWidgets` | `checkboxGroupButtons`, `virtualSelectInput`, `progressBar`, `pickerInput`, `sendSweetAlert` |
321+
| `shinyjs` | `show/hide/toggle`, `enable/disable`, DOM manipulation |
322+
| `srvyr` | Survey-weighted estimation (`as_survey_design`, `survey_mean`) |
323+
| `ggplot2` | All plots in the analysis panel |
324+
| `purrr` | `map`, `reduce`, `list_c` throughout |
325+
| `dplyr` / `tidyr` | Data wrangling in all functions |
326+
| `stringr` | String manipulation (entity prefix handling, name normalisation) |
327+
| `jsonlite` | Parse `chain_summary.json` |
328+
| `zip` | List and extract ZIP archive contents |
329+
| `shiny.i18n` | EN/FR/ES translations |
330+
| `scales` | Comma-formatted y-axis labels for totals plots |

0 commit comments

Comments
 (0)