Skip to content

Conversation

@bayo-ibm
Copy link
Contributor

@bayo-ibm bayo-ibm commented Jul 22, 2025

Description of the change

The block-by-block model evaluation does not work with granite-3.x models. This is because the eval_llm_1GPU does not take into consideration the peculiarities associated with granite-3.x-models.

The quant/ptq.py was updated to reflect the current naming conventions for the various blocks in granite-3.x models.
The eval_llm_1GPU function was updated to correctly calculate the logits for granite-3. x models.

Related issues or PRs

#155

How to verify the PR

Was the PR tested

The fix was validated by performing FP8 DQ-SQ using the granite-3.0-8B-Instruct/granite-3.3-8B-Instruct models. The evaluation of the quantized and the unquantized models using eval_llm_1GPU achieved the same results as using the "evaluator.evaluate" method

  • I have added >=1 unit test(s) for every new method I have added (if that coverage is difficult, please briefly explain the reason)
  • I have ensured all unit tests pass

Checklist for passing CI/CD:

  • All commits are signed showing "Signed-off-by: Name <[email protected]>" with git commit -signoff or equivalent
  • PR title and commit messages adhere to Conventional Commits
  • Contribution is formatted with tox -e fix
  • Contribution passes linting with tox -e lint
  • Contribution passes spellcheck with tox -e spellcheck
  • Contribution passes all unit tests with tox -e unit

Note: CI/CD performs unit tests on multiple versions of Python from a fresh install. There may be differences with your local environment and the test environment.

@bayo-ibm bayo-ibm changed the title Enabling block-by-block evaluation for granite-3.x-models Fix: enabling block-by-block evaluation for granite-3.x-models Jul 22, 2025
@bayo-ibm bayo-ibm changed the title Fix: enabling block-by-block evaluation for granite-3.x-models fix: enabling block-by-block evaluation for granite-3.x-models Jul 22, 2025
@github-actions github-actions bot added the fix label Jul 22, 2025
@bayo-ibm bayo-ibm force-pushed the eval_1gpu branch 2 times, most recently from 8a68e2c to 6cdbd4c Compare July 22, 2025 03:34
Copy link
Collaborator

@andrea-fasoli andrea-fasoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worked with Bayo to troubleshoot this Granite evaluation issue. The solution looks functional.

@chichun-charlie-liu
Copy link
Collaborator

pylint issue regarding triton kernel will be fixed by #166 .

@chichun-charlie-liu chichun-charlie-liu merged commit 78214ae into foundation-model-stack:main Jul 24, 2025
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants