Camera Control Component with websocket h264 stream provider by stmudie · Pull Request #20 · AustralianSynchrotron/sci-comp-ui

stmudie · 2026-03-05T23:03:49Z

This has turned into a lot of code and needs review. I'm expecting to need to make changes but need others to start looking at it.

…ge resize.

…ging.

NoxHarmonium

Looks pretty good, just some refactoring suggestions.
How can I test it? Can I spin up the h264-websocket-stream and it will access it via localhost?

NoxHarmonium · 2026-03-15T22:49:02Z

src/ui/experimental/camera-control/camera-control.tsx

+    startX: number;
+    startY: number;
+    width: number;
+    height: number;


Since this type definition is duplicated on line onZoom it might be good to extract to a type alias.

Are you happy with what I have done in d997fa4

NoxHarmonium · 2026-03-15T22:54:30Z

src/ui/experimental/camera-control/camera-control.tsx

-    if (typeof source === "object" && "video" in source && source.video) {
-      return { w: source.video.videoWidth, h: source.video.videoHeight };
+    if (!image) return { w: 0, h: 0 };
+    if (typeof image === "object" && "video" in image && image.video) {


If image can only be { video: HTMLVideoElement | null; frameId: number } | ImageBitmap | null if we rearrange the guards like this can we avoid this check?

// Rules out `null` if (!image) return { w: 0, h: 0 }; // Rules out `ImageBitmap` if (image instanceof ImageBitmap) { return { w: image.width, h: image.height }; } // This must be `{ video: HTMLVideoElement | null; frameId: number }` return { w: image.video.videoWidth, h: image.video.videoHeight };

I think that last statement return { w: 0, h: 0 }; is effectively dead code since it can never not be one of those types. If it can be, then I guess the types are wrong.

Alternatively, we could turn typeof image === "object" && "video" in image && image.video into a type guard function that can be reused in other parts of this file.

I think rearranging things that way works, but the type checker isn't happy:
'image.video' is possibly 'null'.ts(18047)

const image: { video: HTMLVideoElement | null; frameId: number; } Which is how I ended up with that weird guard statement. I will try the solution mentioned in the latter comment.

Done in 7d2c944

NoxHarmonium · 2026-03-15T22:57:17Z

src/ui/experimental/camera-control/camera-control.tsx

+    lastY: number;
+  } | null>(null);
+  const [spaceHeld, setSpaceHeld] = useState<boolean>(false);
+  const [cursorDisplay, setCursorDisplay] = useState<string>("crosshair");
  const [frameCount, setFrameCount] = useState<number>(0);
  const [startTime, setStartTime] = useState<number>(performance.now());

  // Helper to get dimensions safely
  const getDimensions = () => {


If this uses the return type of other hooks (e.g. image comes from useContext) should it be wrapped in useCallback with a dependency list?

Please correct me if I'm wrong, but I don't think the fact that image comes from useContext necessarily implies that we should use useCallback. It is more if we want to memoize this function if it is causing un-necessary re-renders. The only dependency in this function is image - so we don't gain anything by using useCallback with image in the dependency list.

NoxHarmonium · 2026-03-15T22:57:57Z

src/ui/experimental/camera-control/camera-control.tsx

+
+      // Report the size back to the provider
+      reportSize(Math.floor(width), Math.floor(height));
+    }, 100);


This 100 might be better as a constant that describes what it does

Made the default 100 ms in the debounce function instead.

NoxHarmonium · 2026-03-15T23:01:15Z

src/ui/experimental/camera-control/camera-control.tsx

-        ctx.drawImage(source.video, 0, 0);
-      } else if (source instanceof ImageBitmap) {
-        ctx.drawImage(source, 0, 0);
+      if (typeof image === "object" && "video" in image && image.video) {


As mentioned above, this check is in multiple places so maybe could be a type guard function:

const isVideo = (image: unknown): image is { video: HTMLVideoElement | null; frameId: number } => image !== null && typeof image === "object" && "video" in image && image.video

or bet yet I guess { video: HTMLVideoElement | null; frameId: number } could have a type alias defined in image-context.tsx

type VideoFrame = { video: HTMLVideoElement | null; frameId: number };

const isVideo = (image: unknown): image is VideoFrame => image !== null && typeof image === "object" && "video" in image && image.video

Done in 7d2c944

NoxHarmonium · 2026-03-15T23:14:44Z

src/docs/routes/experimental/camera-control-h264.tsx

 function CameraControlWebsocketH264Demo() {
  const [mousePos, setMousePos] = useState<{ x: number, y: number, intensity: number } | null>(null);
  const [clickPos, setClickPos] = useState<{ x: number, y: number, intensity: number } | null>(null);
+  const api = useMemo(() => h264FetchApi("localhost:9999"), []);


How will this work in the docs if you don't have something running at localhost?

Yeah it won't. I wonder if there is a way to fake it for the docs.

Maybe we can implement a mock context that just emits some test images in a loop or something. If we mock out the API, we would still need a mock websocket that emits h264 I guess which sounds hard.

NoxHarmonium · 2026-03-15T23:19:52Z

src/ui/experimental/camera-control/websocket-h264-provider.tsx

+                configuredRef.current = false; // force reconfigure
+                void tryConfigure();
+              } else {
+                /**/


This else block is redundant

NoxHarmonium · 2026-03-15T23:21:53Z

src/ui/experimental/camera-control/websocket-h264-provider.tsx

 }

-export const WebsocketH264Provider: React.FC<WebsocketH264ProviderProps> = ({ children, wsUrl }) => {
-
+export const WebsocketH264Provider: React.FC<WebsocketH264ProviderProps> = ({


Might be good to have just a short doc explaining that this should be used with the CameraControl component

NoxHarmonium · 2026-03-15T23:22:36Z

src/ui/experimental/camera-control/camera-control.tsx

+    width: number;
+    height: number;
+  }) => void;
+  sizeFollowsImage?: boolean;
 }



Might be good if this had a short doc explaining what this component is

Done in e413dc8

src/ui/experimental/camera-control/camera-control.tsx

stmudie · 2026-03-16T02:13:23Z

Looks pretty good, just some refactoring suggestions. How can I test it? Can I spin up the h264-websocket-stream and it will access it via localhost?

Yes. Or if you use the Eiger stream I think you need to use localhost/test

stmudie added 9 commits February 13, 2026 15:20

Resolution, cropping, dragging for H264

37b92a4

Getting zoom and cropping working correctly.

e1e261d

Cleaning up using refs.

7713cde

Formatted. Fixed references/state so that ws isn't reconnected on ima…

9bf86bb

…ge resize.

sessionId injetion and cleanup with refs

832ed5b

Working version before changing to using padding passed via API.

7c13ece

Using fixed crop resonse

baf0a77

Space button for panning.

02ed30b

Use metadata from stream instead of rest to get current crop for drag…

bfbd363

…ging.

stmudie requested a review from a team March 5, 2026 23:03

stmudie assigned NoxHarmonium Mar 5, 2026

NoxHarmonium requested changes Mar 15, 2026

View reviewed changes

stmudie added 6 commits March 31, 2026 12:40

Move camera control and related components/providers to a sub directory.

a124d5c

Add type definition for reused bounding box.

d997fa4

Create default resolution constant

aab712d

Added docstrings and small cleanup

e413dc8

Change guarding for video frames by create isVideo

7d2c944

Add default delay to debounce function and use it.

aa33784

Conversation

stmudie commented Mar 5, 2026

Uh oh!

NoxHarmonium left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NoxHarmonium Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stmudie Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stmudie commented Mar 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

NoxHarmonium Mar 15, 2026 •

edited

Loading

stmudie Mar 31, 2026 •

edited

Loading