Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
16 changes: 8 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
# Config in the root pyproject.toml ([tool.ruff]). VERSION PINNED ON
# PURPOSE: `uvx ruff` floats to the latest release, which silently
# changes the gate. Bump deliberately, with the resulting diff reviewed.
run: uvx ruff@0.16.0 check packages integrations
run: uvx ruff@0.16.0 check libs policies robots

workspace:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -68,7 +68,7 @@ jobs:
echo "grabette rev: $REV"
curl -fsSL "https://raw.githubusercontent.com/pollen-robotics/grabette/$REV/packages/gripette/proto/gripper.proto" \
-o "$RUNNER_TEMP/gripper.proto"
diff -u "$RUNNER_TEMP/gripper.proto" integrations/openarm/openarm_gripette_simu/proto/gripper.proto
diff -u "$RUNNER_TEMP/gripper.proto" robots/openarm/openarm_gripette_simu/proto/gripper.proto

tests:
runs-on: ubuntu-latest
Expand All @@ -83,7 +83,7 @@ jobs:
# lane installs neither lerobot nor torch. None of these tests need them.
run: |
uv sync --python 3.11 --package grabette-dataset-tools --extra test
uv run --no-sync --package grabette-dataset-tools pytest packages/grabette-dataset-tools/tests -q
uv run --no-sync --package grabette-dataset-tools pytest libs/grabette-dataset-tools/tests -q

lerobot:
# Guards the lerobot boundary so a dependency bump can't silently regress it.
Expand All @@ -97,7 +97,7 @@ jobs:
# no GPU. The lerobot spec is read from the dataset-tools pyproject so the
# tested version tracks the pin automatically.
run: |
SPEC=$(grep -oE "lerobot\[[a-z,]*\]==[0-9.]+" packages/grabette-dataset-tools/pyproject.toml | head -1)
SPEC=$(grep -oE "lerobot\[[a-z,]*\]==[0-9.]+" libs/grabette-dataset-tools/pyproject.toml | head -1)
echo "Testing against: $SPEC"
uv venv .venv-lerobot --python 3.12
VIRTUAL_ENV=.venv-lerobot uv pip install --torch-backend=cpu "$SPEC" pytest pandas scipy
Expand All @@ -108,14 +108,14 @@ jobs:
# `gpu` marker and the package's own addopts deselects it.
run: |
VIRTUAL_ENV=.venv-lerobot uv pip install matplotlib opencv-python-headless
VIRTUAL_ENV=.venv-lerobot uv pip install --no-deps ./packages/grabette-attention
.venv-lerobot/bin/python -m pytest packages/grabette-attention/tests -q
VIRTUAL_ENV=.venv-lerobot uv pip install --no-deps ./libs/grabette-attention
.venv-lerobot/bin/python -m pytest libs/grabette-attention/tests -q

- name: Chunk-relative processor tests
# Two of its test files import lerobot at module scope. This package
# supplies the chunk-relative action transform that the pi0.5
# checkpoints' preprocessors reference by registry name, so a regression
# here breaks checkpoint loading rather than just a metric.
run: |
PYTHONPATH=packages/grabette-chunkrel .venv-lerobot/bin/python -m pytest \
packages/grabette-chunkrel/tests -q
PYTHONPATH=libs/grabette-chunkrel .venv-lerobot/bin/python -m pytest \
libs/grabette-chunkrel/tests -q
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# grabette-policy

Policy learning for [GRABETTE](https://github.com/pollen-robotics/grabette):
training integrations, the action representations and diagnostics they use,
training recipes, the action representations and diagnostics they use,
and the OpenArm-Gripette robot they are evaluated and deployed on.

Datasets come from the GRABETTE recording rig
Expand All @@ -11,22 +11,22 @@ LeRobot datasets by the SLAM pipeline

| Path | What |
|---|---|
| [`packages/grabette-dataset-tools`](packages/grabette-dataset-tools) | Grasp projection, chunk-relative stats, raw-capture normalization, publish gate |
| [`packages/grabette-chunkrel`](packages/grabette-chunkrel) | Chunk-relative action representation + its LeRobot processor steps |
| [`packages/grabette-attention`](packages/grabette-attention) | Offline attention maps and view-ablation diagnostics |
| [`integrations/DiffusionPolicy`](integrations/DiffusionPolicy) | Diffusion Policy data prep and training (own `uv.lock`) |
| [`integrations/Pi05`](integrations/Pi05) | Pi0.5 fine-tuning (own `uv.lock`) |
| [`integrations/openarm`](integrations/openarm) | OpenArm-Gripette: model, MuJoCo sim, real-arm driver, teleop bridge |
| [`libs/grabette-dataset-tools`](libs/grabette-dataset-tools) | Grasp projection, chunk-relative stats, raw-capture normalization, publish gate |
| [`libs/grabette-chunkrel`](libs/grabette-chunkrel) | Chunk-relative action representation + its LeRobot processor steps |
| [`libs/grabette-attention`](libs/grabette-attention) | Offline attention maps and view-ablation diagnostics |
| [`policies/DiffusionPolicy`](policies/DiffusionPolicy) | Diffusion Policy data prep and training (own `uv.lock`) |
| [`policies/Pi05`](policies/Pi05) | Pi0.5 fine-tuning (own `uv.lock`) |
| [`robots/openarm`](robots/openarm) | OpenArm-Gripette: model, MuJoCo sim, real-arm driver, teleop bridge |

## Development

Requires [uv](https://docs.astral.sh/uv/). `packages/*` and
`integrations/openarm/*` form one uv workspace with a single lock; DiffusionPolicy
Requires [uv](https://docs.astral.sh/uv/). `libs/*` and
`robots/openarm/*` form one uv workspace with a single lock; DiffusionPolicy
and Pi05 are standalone projects with their own lock.

```bash
uv sync --all-packages # the workspace
cd integrations/DiffusionPolicy && uv sync # a standalone integration
cd policies/DiffusionPolicy && uv sync # a standalone policy project
```

The OpenArm meshes are in Git LFS: install [git-lfs](https://git-lfs.com) before cloning.
Expand Down
10 changes: 5 additions & 5 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ for attribution and license compliance.
### OpenArm & openarm_can — Apache-2.0 (Enactic, Inc.)
<https://github.com/enactic/openarm> · <https://github.com/enactic/openarm_can>

- The robot description in `integrations/openarm/openarm_gripette_model` is
- The robot description in `robots/openarm/openarm_gripette_model` is
derived from the OpenArm arm model, adapted to carry the Gripette end-effector.
- `openarm_can` is a dependency of `integrations/openarm/openarm_gripette`.
- `integrations/openarm/openarm_gripette/examples/calibrate_arm_no_gripper.py` is
- `openarm_can` is a dependency of `robots/openarm/openarm_gripette`.
- `robots/openarm/openarm_gripette/examples/calibrate_arm_no_gripper.py` is
adapted from OpenArm's zero-position calibration procedure.

### LeRobot — Apache-2.0 (Hugging Face, Inc.)
<https://github.com/huggingface/lerobot>

- Used as a dependency for LeRobot-format dataset generation and policy training.
- `integrations/openarm/openarm_gripette/openarm_gripette/openarm7_follower.py`
- `robots/openarm/openarm_gripette/openarm_gripette/openarm7_follower.py`
and `config_openarm7_follower.py` subclass LeRobot's `OpenArmFollower` and
retain the original Hugging Face copyright headers.

Expand All @@ -32,6 +32,6 @@ Used to generate the OpenArm-Gripette URDF / MuJoCo model from Onshape CAD.
### 6-D rotation representation — Zhou et al., CVPR 2019
*"On the Continuity of Rotation Representations in Neural Networks."*

`integrations/DiffusionPolicy/rotation.py` is a Pollen Robotics implementation of
`policies/DiffusionPolicy/rotation.py` is a Pollen Robotics implementation of
this representation (Apache-2.0), vendored to avoid depending on a fork.

Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Attention maps and view ablation: first measurements

What the tool in `packages/grabette-attention` actually found. Companion to
What the tool in `libs/grabette-attention` actually found. Companion to
`saliency_review.md`, which is the literature review and the
argument for the tool's design; this file is the data. Every script and CSV
cited here is in `refs/`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ Legend: **[V]** = verified by reading source; **[I]** = inferred (standard behav
- All four venvs that matter carry **byte-identical** `modeling_pi05.py` (md5 `a5758a20…`) and
`modeling_diffusion.py` (md5 `0272d025…`) [V]:
- `…/worktrees/attention-maps/.venv` and `…/worktrees/relative-actions/.venv` -> lerobot **0.6.0** (PyPI)
- `…/worktrees/relative-actions/integrations/Pi05/.venv` -> lerobot **0.6.1** from git commit `e40b58a8` (`direct_url.json`)
- `…/integrations/DiffusionPolicy/.venv` -> lerobot 0.6.0
- `…/worktrees/relative-actions/policies/Pi05/.venv` -> lerobot **0.6.1** from git commit `e40b58a8` (`direct_url.json`)
- `…/policies/DiffusionPolicy/.venv` -> lerobot 0.6.0
- (`/home/steve/Project/Repo/GRABETTE/openarm_gripette_simu/.venv` is stale lerobot 0.5.1 with a *different* file; ignore.)
- Readable source copy used for line numbers below (identical to what is installed):
`LR = /home/steve/.cache/uv/git-v0/checkouts/b2400a7a62d6a7cf/e40b58a8/src/lerobot/policies`
- Stack in the Pi05 venv: `transformers 5.5.4`, `torch 2.11.0+cu128` [V]. transformers paths below are
`TF = …/relative-actions/integrations/Pi05/.venv/lib/python3.12/site-packages/transformers`.
`TF = …/relative-actions/policies/Pi05/.venv/lib/python3.12/site-packages/transformers`.

---

Expand Down Expand Up @@ -187,7 +187,7 @@ h = expert_attn[i].register_forward_hook(lambda mod, inp, out: store(i, out[1]))
- Inference: `predict_action_chunk` and `select_action` are `@torch.no_grad()` (l.102, 123); `generate_actions` (l.307) and `conditional_sample` (l.233) are **not** — call
`policy.diffusion.generate_actions(batch)` under grad for Grad-CAM (batch must already be time-stacked `(B, S, …)` with `OBS_IMAGES` key; see `predict_action_chunk` l.113-119 for how it builds it offline).

### 4.3 Grabette config (`integrations/DiffusionPolicy/train.py:473-520`) [V]
### 4.3 Grabette config (`policies/DiffusionPolicy/train.py:473-520`) [V]
`DiffusionConfig(n_obs_steps=args.n_obs_steps (default 2, l.281-283), horizon=16, n_action_steps=args.n_action_steps (default 8), vision_backbone="resnet18",
resize_shape=(236, 236), crop_ratio=0.95 # -> crop_shape (224, 224) via configuration_diffusion.py:191-196, crop_is_random=not args.no_random_crop,
pretrained_backbone_weights=None, use_group_norm=True, use_separate_rgb_encoder_per_camera=False, spatial_softmax_num_keypoints=32,
Expand All @@ -204,12 +204,12 @@ Cameras: `--cameras` default `["observation.images.cam0"]` (l.309); input featur

## 5. Repo scripts

### 5.1 `integrations/openarm/openarm_gripette_simu/examples/evaluate.py` (2373 lines) [V]
### 5.1 `robots/openarm/openarm_gripette_simu/examples/evaluate.py` (2373 lines) [V]
- `--dump_obs DIR` (l.509-512): "Directory to dump the EXACT observations fed to the policy (obs_XXXXX.png + state.jsonl, one subdir per episode). Use with --num_episodes 1 for a train/deploy distribution check (ood_check.py)."
- Sync path (l.1234-1244): **per control step**, `cv2.imwrite(dump_dir / f"obs_{step:05d}.png", cv2.cvtColor(camera_image, cv2.COLOR_RGB2BGR), [IMWRITE_PNG_COMPRESSION, 1])` and one JSON line
`{"step": step, "state": [floats]}` appended to `state.jsonl`. `camera_image` is **RGB HWC uint8 full-res (960x720 live), pre-normalisation, pre-letterbox**.
- Async path (l.1804-1828): same PNG/state.jsonl keyed by `executor.sent_count`, plus `traj.jsonl` lines `{"t", "tick", "meas", "meas_r6d", "cmd", "cmd_r6d", "grip_cmd", "grip_obs"}`.
- Per-episode subdir: `f"{args.dump_obs}/ep{ep:03d}"` (l.2268, 2296). No action/chunk is dumped in sync mode. Consumer: `integrations/DiffusionPolicy/ood_check.py` (`--images DIR`, docstring l.1-31).
- Per-episode subdir: `f"{args.dump_obs}/ep{ep:03d}"` (l.2268, 2296). No action/chunk is dumped in sync mode. Consumer: `policies/DiffusionPolicy/ood_check.py` (`--images DIR`, docstring l.1-31).
- Policy loading `_load_policy_any` (l.37-56): `cfg = PreTrainedConfig.from_pretrained(ckpt); cfg.compile_model = False; policy = get_policy_class(cfg.type).from_pretrained(ckpt, config=cfg); if cfg.type in ("pi05","pi0"): policy = policy.to(dtype=torch.float32)`.
Then `policy.to(device); policy.eval(); make_pre_post_processors(policy.config, pretrained_path=ckpt, preprocessor_overrides={"device_processor": {"device": str(device)}})` (l.2064-2074).
- Batch construction (l.1246-1259): `image_tensor = torch.from_numpy(camera_image).float() / 255.0; .permute(2,0,1)`; batch = `{"observation.state": (1, D), "observation.images.cam0": (1,3,H,W), "task": task}`, then `batch = preprocessor(batch)` (l.1271) and `policy.select_action(batch)` under `torch.no_grad()` (l.1274-1276, 1391-1393).
Expand All @@ -225,7 +225,7 @@ Cameras: `--cameras` default `["observation.images.cam0"]` (l.309); input featur
`serve.py:6` "one msgpack frame {"actions": ..., "server_timing": {...}}"; args `--checkpoint --host --port --api_key --device --dtype {float32,bfloat16} --transport {websocket,iroh}` (l.123-136).
-> **Attention cannot be obtained remotely today**; it would need a server-side hook + an extra reply key (e.g. `"attention": {layer: (8,50,prefix_len)}`), and the wire protocol string `PROTOCOL = "ficelle/1"` (l.43) bumped. Local checkpoint path is the natural place for the tool.

### 5.3 `integrations/Pi05/smoke_generation.py` (447 lines) [V]
### 5.3 `policies/Pi05/smoke_generation.py` (447 lines) [V]
- Offline load (l.210-219): `cfg = PreTrainedConfig.from_pretrained(ckpt); cfg.device = "cpu"; cfg.compile_model = False; policy = get_policy_class(args.policy_type).from_pretrained(ckpt, config=cfg);
policy = policy.to(dtype=torch.float32 if args.fp32 else torch.bfloat16).eval(); policy = policy.to(device); policy.config.device = device; pre, post = make_pre_post_processors(policy.config, args.checkpoint)`.
`--fp32` help (l.182-186): "the pi05 port has a bf16 dtype clash in its flow path … fp32 is 16.6GB of WEIGHTS … use a >=24GB card".
Expand All @@ -236,11 +236,11 @@ Cameras: `--cameras` default `["observation.images.cam0"]` (l.309); input featur
- `probe_task_sensitivity.py` is the remote (ficelle) twin: same idea over `open_client` (PEP-723 header pins lerobot `@e40b58a8`).

### 5.4 Existing visualisation conventions to follow [V]
- `integrations/DiffusionPolicy/offline_eval.py:165-200`: matplotlib `Agg`, `fig.savefig(out_dir / f"offline_eval_ep{ep:03d}.png", dpi=110)`, graceful "matplotlib not installed — skipping plots". Batch building at l.126-131 mirrors smoke_generation.
- `integrations/DiffusionPolicy/ood_check.py:45-60`: `FeatureExtractor` hooks `policy.diffusion.rgb_encoder` directly with the checkpoint preprocessor (`self.encoder = policy.diffusion.rgb_encoder`) — precedent for reaching into `policy.diffusion.*` from a tool.
- `integrations/DiffusionPolicy/vision_check.py` (docstring l.1-45): "Does the policy actually USE the image?" — image-swap / pixel-shift sensitivity probes; the attention tool is the natural complement (it should be listed next to it in the DP README table).
- `policies/DiffusionPolicy/offline_eval.py:165-200`: matplotlib `Agg`, `fig.savefig(out_dir / f"offline_eval_ep{ep:03d}.png", dpi=110)`, graceful "matplotlib not installed — skipping plots". Batch building at l.126-131 mirrors smoke_generation.
- `policies/DiffusionPolicy/ood_check.py:45-60`: `FeatureExtractor` hooks `policy.diffusion.rgb_encoder` directly with the checkpoint preprocessor (`self.encoder = policy.diffusion.rgb_encoder`) — precedent for reaching into `policy.diffusion.*` from a tool.
- `policies/DiffusionPolicy/vision_check.py` (docstring l.1-45): "Does the policy actually USE the image?" — image-swap / pixel-shift sensitivity probes; the attention tool is the natural complement (it should be listed next to it in the DP README table).
- `grabette-postprocess/scripts/visualize/visualize_rgbd_trajectory.py` in the grabette-slam repo: rerun (`rr.init(app_id, spawn=True)`, `rr.set_time("time", timestamp=…)`, `rr.log("camera_feed", rr.Image(frame_disp))` after `cv2.resize(frame, (640, 400))`, `rr.SeriesLines`/`rr.Scalars`). If you want a rerun front-end, this is the house style (entity paths `world/...`, `camera_feed`, `imu/...`).
- `integrations/openarm/openarm_gripette_simu/examples/replay_dataset.py:55, 124`: `to_bgr()` helper "LeRobot video frame -> HWC uint8 BGR for cv2.imshow"; `openarm_gripette/examples/view_camera.py`: matplotlib QtAgg live view because lerobot pulls headless OpenCV (l.3-35).
- `robots/openarm/openarm_gripette_simu/examples/replay_dataset.py:55, 124`: `to_bgr()` helper "LeRobot video frame -> HWC uint8 BGR for cv2.imshow"; `openarm_gripette/examples/view_camera.py`: matplotlib QtAgg live view because lerobot pulls headless OpenCV (l.3-35).
- `cv2.applyColorMap`/`cv2.addWeighted` are **not used anywhere** in the repo yet (grep) — no existing heat-map overlay to match.

### 5.5 Existing mentions of "attention" / "saliency" / "grad-cam" / "heatmap" [V]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""Offline attention-map and view-ablation diagnostics for Grabette policies.

Read `packages/grabette-attention/docs/saliency_review.md` before drawing conclusions from the
Read `libs/grabette-attention/docs/saliency_review.md` before drawing conclusions from the
maps: on pi0.5 an interventional score is measurably more faithful than
attention, so the maps are hypothesis generators and the ablation numbers are
the evidence.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""Write attention overlays as PNGs and the numbers as a text summary.

Headless by construction: matplotlib is switched to Agg before pyplot is
imported, matching `integrations/DiffusionPolicy/offline_eval.py`. Frames are
imported, matching `policies/DiffusionPolicy/offline_eval.py`. Frames are
RGB in memory throughout; nothing here writes BGR.
"""

Expand All @@ -17,7 +17,7 @@
"NOTE: pi0.5 spreads attention broadly with low peaks, and action "
"fine-tuning makes it more diffuse. A broad map is NORMAL and is not "
"evidence of anything. The ablation millimetres are the interventional "
"measurement; the map is a hypothesis. See packages/grabette-attention/docs/saliency_review.md."
"measurement; the map is a hypothesis. See libs/grabette-attention/docs/saliency_review.md."
)

# Attention grids routinely carry one or two "register" cells -- high-norm
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""Load a pi0.5 checkpoint the way the rest of the repo does.

Copied deliberately from `integrations/Pi05/smoke_generation.py` rather than
Copied deliberately from `policies/Pi05/smoke_generation.py` rather than
invented: CPU config first, `compile_model` off so forward hooks are not
swallowed by a graph, fp32 because the pi05 port has a bf16 clash in its flow
path, and camera keys taken from the CHECKPOINT rather than the dataset.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
image, re-run, and measure how far the commanded chunk moves. The result is a
causal map in millimetres, not an attention map -- which matters because on
this policy attention mass does not track causal effect (see
packages/grabette-attention/docs/findings.md), and the attention map's dominant component is
libs/grabette-attention/docs/findings.md), and the attention map's dominant component is
locked to the gripper rather than to the scene.

The occlusion grid divides the SOURCE image into blocks. The attention grid has
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ class DatasetSource:
"""Reads held-out frames from a LeRobot dataset.

Follows the offline loading already used by
`integrations/Pi05/smoke_generation.py`: one episode at a time, frames
`policies/Pi05/smoke_generation.py`: one episode at a time, frames
indexed directly. Camera keys come from the CHECKPOINT, not the dataset, so
a dataset missing a view is handled by the adapter's masking rather than by
guessing here.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ dependencies = ["numpy"]

[project.optional-dependencies]
# The PNG front end. matplotlib only; the repo already uses the Agg backend for
# offline plots (integrations/DiffusionPolicy/offline_eval.py).
# offline plots (policies/DiffusionPolicy/offline_eval.py).
png = ["matplotlib"]
# The second front end, added after the PNG one.
rerun = ["rerun-sdk"]
Expand Down
Loading
Loading