Refactor inference.py for LLM and RoBERTa support#34

Draft

andrea-fasoli wants to merge 30 commits intofoundation-model-stack:mainfrom

andrea-fasoli:refactor_inference

Commits on Apr 23, 2025

Refactor argument parsing
andrea-fasoli
committed
Refactor model setup
andrea-fasoli
committed
Refactor setup of quantization (addons, linear_config)
andrea-fasoli
committed
Refactor LLM handling
andrea-fasoli
committed
Refactor RoBERTa handling
andrea-fasoli
committed
Refactor Direct Quantization (wip)
andrea-fasoli
committed
Refactor inference entry point for LLM and RoBERTa
andrea-fasoli
committed
Refactor AIU setup (relocate env vars setup)
andrea-fasoli
committed

Commits on Jun 19, 2025

Fix tokenizer; add some dec/enc args validation
andrea-fasoli
committed
Update AIU env var
andrea-fasoli
committed
Minor args update
andrea-fasoli
committed
Relocate print_model_params function
andrea-fasoli
committed
Gate transformers import
andrea-fasoli
committed
Bring recent updates to inference.py into run_decoder.py
andrea-fasoli
committed
Add new sendnn compile arg
andrea-fasoli
committed
Remove unified inference.py
andrea-fasoli
committed
Small fixes
andrea-fasoli
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor inference.py for LLM and RoBERTa support#34

Refactor inference.py for LLM and RoBERTa support#34
andrea-fasoli wants to merge 30 commits intofoundation-model-stack:mainfrom
andrea-fasoli:refactor_inference

Commits on Apr 23, 2025

Refactor argument parsing

Refactor model setup

Refactor setup of quantization (addons, linear_config)

Refactor LLM handling

Refactor RoBERTa handling

Refactor Direct Quantization (wip)

Refactor inference entry point for LLM and RoBERTa

Refactor AIU setup (relocate env vars setup)

Commits on Jun 18, 2025

Remove deprecated local_size

Remove env vars already set in e2e_stable image

Group and update parser arguments

Rename enc/dec utils

Gating some AIU settings

Split inference into decoder/encoder scripts (wip)

Commits on Jun 19, 2025

Fix tokenizer; add some dec/enc args validation

Update AIU env var

Minor args update

Relocate print_model_params function

Gate transformers import

Bring recent updates to inference.py into run_decoder.py

Add new sendnn compile arg

Remove unified inference.py

Small fixes

Commits on Jun 20, 2025

Remove deprecated torch dynamo config option

Update type hints

Update skip compile message

Adjust extra_generation_kwargs handling

Remove INT8 DQ

Update import of ids_for_prompt and fix some formatting

Minor changes

Refactor inference.py for LLM and RoBERTa support#34andrea-fasoli wants to merge 30 commits intofoundation-model-stack:mainfoundation-model-stack/aiu-fms-testing-utils:mainfrom andrea-fasoli:refactor_inferenceandrea-fasoli/aiu-fms-testing-utils:refactor_inferenceCopy head branch name to clipboard

Commits

Commits on Apr 23, 2025

Commits on Jun 18, 2025

Commits on Jun 19, 2025

Commits on Jun 20, 2025

Refactor inference.py for LLM and RoBERTa support#34
andrea-fasoli wants to merge 30 commits intofoundation-model-stack:mainfrom
andrea-fasoli:refactor_inference