Skip to content

Commit

Permalink
Fix annotation accuracy display in the UI (#7652)
Browse files Browse the repository at this point in the history
<!-- Raise an issue to propose your change
(https://github.com/opencv/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution
guide](https://opencv.github.io/cvat/docs/contributing/). -->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
<!-- Why is this change required? What problem does it solve? If it
fixes an open
issue, please link to the issue here. Describe your changes in detail,
add
screenshots. -->
The current formula the UI uses to calculate accuracy is incorrect. The
denominator is supposed to be the total sample count, but ds_count +
gt_count - valid_count is not the same value. To illustrate:

      ****      ***.        *...         ****
      ****      ***.        .*..         ****
      ****      ***.        ..*.         ****
      ....      ***.        ....         ***.

    ds_count  gt_count  valid_count  total_count (expected)

Each value is the sum of the values in the confusion matrix marked by
`*`. The current formula essentially adds values that are off the
diagonal twice.

Instead of trying to derive the total count from other values, just
transmit the correct value from the server and use it in the UI.

### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
Manual testing.

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have created a changelog fragment <!-- see top comment in
CHANGELOG.md -->
- ~~[ ] I have updated the documentation accordingly~~
- ~~[ ] I have added tests to cover my changes~~
- ~~[ ] I have linked related issues (see [GitHub docs](

https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))~~
- ~~[ ] I have increased versions of npm packages if it is necessary

([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning),

[cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning),

[cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning)
and

[cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))~~

### License

- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the
project.
  Feel free to contact the maintainers if that's a concern.
  • Loading branch information
SpecLad authored Mar 21, 2024
1 parent fc54c47 commit 7e2f9d3
Show file tree
Hide file tree
Showing 6 changed files with 16 additions and 2 deletions.
4 changes: 4 additions & 0 deletions changelog.d/20240321_153828_roman_fix_accuracy_display.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
### Fixed

- Fixed accuracy being displayed incorrectly on the task analytics page
(<https://github.com/opencv/cvat/pull/7652>)
3 changes: 1 addition & 2 deletions cvat-core/src/quality-report.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,7 @@ export default class QualityReport {
validCount: this.#summary.valid_count,
dsCount: this.#summary.ds_count,
gtCount: this.#summary.gt_count,
accuracy: (this.#summary.valid_count /
(this.#summary.ds_count + this.#summary.gt_count - this.#summary.valid_count)) * 100,
accuracy: (this.#summary.valid_count / this.#summary.total_count) * 100,
precision: (this.#summary.valid_count / this.#summary.gt_count) * 100,
recall: (this.#summary.valid_count / this.#summary.ds_count) * 100,
conflictsByType: {
Expand Down
1 change: 1 addition & 0 deletions cvat-core/src/server-response-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,7 @@ export interface SerializedQualityReportData {
valid_count: number;
ds_count: number;
gt_count: number;
total_count: number;
error_count: number;
warning_count: number;
conflicts_by_type: {
Expand Down
1 change: 1 addition & 0 deletions cvat/apps/quality_control/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ class QualityReportSummarySerializer(serializers.Serializer):
valid_count = serializers.IntegerField(source="annotations.valid_count")
ds_count = serializers.IntegerField(source="annotations.ds_count")
gt_count = serializers.IntegerField(source="annotations.gt_count")
total_count = serializers.IntegerField(source="annotations.total_count")


class QualityReportSerializer(serializers.ModelSerializer):
Expand Down
3 changes: 3 additions & 0 deletions cvat/schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9140,6 +9140,8 @@ components:
type: integer
gt_count:
type: integer
total_count:
type: integer
required:
- conflict_count
- conflicts_by_type
Expand All @@ -9148,6 +9150,7 @@ components:
- frame_count
- frame_share
- gt_count
- total_count
- valid_count
- warning_count
QualityReportTarget:
Expand Down
6 changes: 6 additions & 0 deletions tests/python/shared/assets/quality_reports.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
"frame_count": 3,
"frame_share": 0.2727272727272727,
"gt_count": 33,
"total_count": 43,
"valid_count": 21,
"warning_count": 15
},
Expand Down Expand Up @@ -56,6 +57,7 @@
"frame_count": 3,
"frame_share": 0.2727272727272727,
"gt_count": 33,
"total_count": 43,
"valid_count": 21,
"warning_count": 15
},
Expand Down Expand Up @@ -86,6 +88,7 @@
"frame_count": 3,
"frame_share": 0.2727272727272727,
"gt_count": 35,
"total_count": 46,
"valid_count": 22,
"warning_count": 16
},
Expand Down Expand Up @@ -116,6 +119,7 @@
"frame_count": 3,
"frame_share": 0.2727272727272727,
"gt_count": 35,
"total_count": 46,
"valid_count": 22,
"warning_count": 16
},
Expand Down Expand Up @@ -146,6 +150,7 @@
"frame_count": 3,
"frame_share": 0.2727272727272727,
"gt_count": 35,
"total_count": 46,
"valid_count": 22,
"warning_count": 16
},
Expand Down Expand Up @@ -176,6 +181,7 @@
"frame_count": 3,
"frame_share": 0.2727272727272727,
"gt_count": 35,
"total_count": 46,
"valid_count": 22,
"warning_count": 16
},
Expand Down

0 comments on commit 7e2f9d3

Please sign in to comment.