Fix the crash in default ATTENTION path for Gaudi backend (#3235)

sywangyi · web-flow · commit 43b1b07fb963 · 2025-05-20T14:02:32.000+02:00
Signed-off-by: Wang, Yi A &lt;yi.a.wang@intel.com&gt;
diff --git a/backends/gaudi/server/text_generation_server/tgi_service.py b/backends/gaudi/server/text_generation_server/tgi_service.py
@@ -31,6 +31,7 @@ def main(args):
         trust_remote_code=args.trust_remote_code,
         uds_path=args.uds_path,
         max_input_tokens=args.max_input_tokens,
+        kv_cache_dtype="auto",
     )
 
 

Original file line number	Diff line number	Diff line change
`@@ -31,6 +31,7 @@ def main(args):`
`31`	`31`	`trust_remote_code=args.trust_remote_code,`
`32`	`32`	`uds_path=args.uds_path,`
`33`	`33`	`max_input_tokens=args.max_input_tokens,`
	`34`	`+ kv_cache_dtype="auto",`
`34`	`35`	`)`
`35`	`36`
`36`	`37`