Skip to content

Conversation

@onyedikachi-david
Copy link


Fixes #760
/claim #760

What kind of change does this PR introduce?

Feature

Summary

This PR introduces the CursorReplayStrategy feature, which allows the AI model to suggest coordinates for a red dot on a screenshot representing the intended action target location. This new strategy provides the ability to visualize and self-correct initial suggestions, improving replay accuracy. It also includes an update to the screenshot capturing functionality to support capturing all monitors.

Checklist

  • My code follows the style guidelines of OpenAdapt
  • I have performed a self-review of my code
  • If applicable, I have added tests to prove my fix is functional/effective
  • I have linted my code locally prior to submission
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
  • New and existing unit tests pass locally with my changes

How can your code be run and tested?

  1. Test the CursorReplayStrategy by running the AI model and verifying the red dot is correctly suggested and visualized on the screenshot.
  2. Use the self-correction mechanism to adjust the red dot position and verify the corrections are applied accurately.

Other information

No additional context needed.


@abrichr abrichr mentioned this pull request Jul 5, 2024
7 tasks
@abrichr
Copy link
Member

abrichr commented Jul 5, 2024

Thank you @onyedikachi-david !

Can you please update your PR description with images containing the screenshot with the red dot? Ideally multiple over time as the model self corrects.

@abrichr
Copy link
Member

abrichr commented Jan 17, 2026

Closing as part of the architecture transition. The legacy codebase has been moved to legacy/ and frozen at v0.46.0.

If this change is still needed, please:

  1. Open a PR against the relevant modular package repository
  2. Or re-open this PR targeting the legacy/ directory if it's a critical legacy fix

See PR #960 for details on the new meta-package architecture.

@abrichr abrichr closed this Jan 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement model cursor for visual feedback

2 participants