Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: New TableFormer model weights #1100

Merged
merged 3 commits into from
Mar 11, 2025
Merged

feat: New TableFormer model weights #1100

merged 3 commits into from
Mar 11, 2025

Conversation

cau-git
Copy link
Contributor

@cau-git cau-git commented Mar 3, 2025

  • New Tableformer weights that work better for tables that have lots of text
  • This PR sets "Accurate" tableformer mode as default mode
  • Re-generated tests
  • Evaluations show improvement on every evaluation dataset!

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

Evaluation of FinTabNet

FinTabNet table_structure TEDS_struct-with-text:
mean=0.90 median=0.92 std=0.09
vs mean=0.89 median=0.91 std=0.10

image

FinTabNet table_structure TEDS_struct-only:
mean=0.90 median=0.92 std=0.08
vs mean=0.90 median=0.92 std=0.09

image


Evaluation of PubTabNet

PubTabNet table_structure TEDS_struct-with-text:
mean=0.85 median=0.90 std=0.17
vs mean=0.80 median=0.83 std=0.13

image

PubTabNet table_structure TEDS_struct-only:
mean=0.86 median=0.90 std=0.15
vs mean=0.81 median=0.84 std=0.12

image


Evaluation of Pub1M

Pub1M table_structure TEDS_struct-with-text:
mean=0.82 median=0.87 std=0.18
vs mean=0.82 median=0.86 std=0.17

image

Pub1M table_structure TEDS_struct-only:
mean=0.84 median=0.88 std=0.17
vs mean=0.84 median=0.87 std=0.16

image

Copy link

mergify bot commented Mar 3, 2025

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

Wonderful, this rule succeeded.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

maxmnemonic
maxmnemonic previously approved these changes Mar 4, 2025
Copy link
Contributor

@maxmnemonic maxmnemonic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests don't break with new TF weights, so no need to update them!
(Edit: it's because tests run FAST configuration, not ACCURATE, I'm changing that...)

@maxmnemonic
Copy link
Contributor

Switched TF model to Accurate by default (in API and CLI), and re-generated tests

cau-git and others added 3 commits March 10, 2025 16:23
@cau-git cau-git marked this pull request as ready for review March 10, 2025 15:41
@maxmnemonic maxmnemonic changed the title feat: New tableformer model weights [WIP] feat: New TableFormer model weights Mar 10, 2025
@dolfim-ibm
Copy link
Contributor

@maxmnemonic after release, let's push the performance details in the docs.

@cau-git cau-git merged commit eb97357 into main Mar 11, 2025
10 checks passed
@cau-git cau-git deleted the dev/new-tf-weights branch March 11, 2025 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants