Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consensus simple merging #8953

Open
wants to merge 143 commits into
base: develop
Choose a base branch
from
Open

Conversation

zhiltsov-max
Copy link
Contributor

@zhiltsov-max zhiltsov-max commented Jan 16, 2025

Motivation and context

Depends on #8939, #9026
A part of #8434

  • Added support for consensus task and consensus job merging (API and UI)
  • Added simple consensus settings
  • Added server tests
  • Added new consensus RQ queue and worker
  • Updated skeleton comparisons: hidden points now also contribute to the skeleton similarity. Only visibility is taken into account for invisible points

Limitations:

  • Merging is supported for all annotations except 2d and 3d cuboids. 3d tasks are not supported
  • Annotation groups are not supported (each annotation is considered separate in a group)
  • Polygons and masks are not interchangeable (each type is compared only with the same type)

How has this been tested?

Checklist

  • I submit my changes into the develop branch
  • I have created a changelog fragment
  • I have updated the documentation accordingly
  • I have added tests to cover my changes
  • I have linked related issues (see GitHub docs)
  • I have increased versions of npm packages if it is necessary
    (cvat-canvas,
    cvat-core,
    cvat-data and
    cvat-ui)

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.

@zhiltsov-max zhiltsov-max marked this pull request as ready for review February 3, 2025 16:11
cvat-core/src/api-implementation.ts Outdated Show resolved Hide resolved
cvat-core/src/api-implementation.ts Outdated Show resolved Hide resolved
cvat-ui/src/actions/consensus-actions.ts Outdated Show resolved Hide resolved
cvat-core/src/server-proxy.ts Show resolved Hide resolved
cvat-ui/src/components/task-page/top-bar.tsx Show resolved Hide resolved
cvat-ui/src/reducers/consensus-reducer.ts Outdated Show resolved Hide resolved
cvat-ui/src/reducers/consensus-reducer.ts Outdated Show resolved Hide resolved
cvat-ui/src/reducers/index.ts Outdated Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need another worker for this? Can't an existing worker, like the quality report worker, handle this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we add a worker almost for any new activity, so this PR does the same. Probably, it can be optimized.

schema:
type: string
description: |
The consensus merge request id. Can be specified to check operation status.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be using the request API for status checking.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, the reason is that Requests API v2 is not implemented yet. It's hard to integrate anything except import or export into the same server code. Supporting other use cases is one of the key goals of v2.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you at least move the status check to a different endpoint? Creating a request and checking its status are different operations, and should not be mixed in a single endpoint.

Copy link
Contributor Author

@zhiltsov-max zhiltsov-max Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will create one more API style. The one implemented is used for quality reports, job validations, and analytics.

cvat/apps/consensus/models.py Outdated Show resolved Hide resolved
cvat/apps/engine/models.py Outdated Show resolved Hide resolved
cvat/apps/engine/models.py Outdated Show resolved Hide resolved
schema:
type: string
description: |
The consensus merge request id. Can be specified to check operation status.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you at least move the status check to a different endpoint? Creating a request and checking its status are different operations, and should not be mixed in a single endpoint.

cvat/schema.yml Outdated
type: number
format: double
description: |
Minimum required percent of sources having an annotation for it to be accepted
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend using fractions in the API (0-1) rather than percentages. Percentage is a display format; for calculations, fractions usually make more sense.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a number in the range [0;1], actually. This description is a little bit more UI friendly, but it's possible to change the word and replace it in UI settings instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean... the description is for the API, not the UI. So it should describe what the API accepts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, updated

cvat/apps/consensus/signals.py Show resolved Hide resolved
utils.is_sandbox
user := input.auth.user
qobject := [
{"task__owner_id": user.id},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason to allow listing quality settings without a task_id filter? It seems like you could make that filter mandatory, and then just make sure that the user has the appropriate permissions for the task. That way, you won't have to write more complex access rules.

Copy link
Contributor Author

@zhiltsov-max zhiltsov-max Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's hard to say, it's just a regular list endpoint for a model. Maybe someone needs to list settings in a project or in an org.

cvat/apps/consensus/serializers.py Outdated Show resolved Hide resolved
cvat/apps/consensus/serializers.py Outdated Show resolved Hide resolved
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants