Skip to content

Commit eccf758

Browse files
JakeStevensfacebook-github-bot
authored andcommitted
Add Whisper HF export (encoder, cross_kv, static-cache decoder) (#23901)
Summary: Add `examples/audio/export_hf_whisper.py` to export `WhisperForConditionalGeneration` directly through Transformers into one `.ptn` with three methods: `encoder` maps mel features to hidden states, `cross_kv` projects per-layer cross-attention keys then values, and `decoder` consumes token IDs, explicit cache positions, and those projections to produce logits. Self-attention uses zero-initialized mutable static-cache buffers. Positions and the causal mask derive from `cache_position`, so the runner needs no attention-mask input. `--dump_dir` writes runner inputs and an eager greedy transcript with a fresh decoder, preserving the exported cache buffers. An unspecified forced language token is resolved from the example features through Transformers language detection, retaining subsequent forced-token positions. Sparse forced prefixes are rejected. Generation is capped at `max_target_positions` and skips the unused decoder call after the last requested token. The export format is independent of the runtime engine. The exporter and the existing LLM example share lowering reports through `examples/hf_llm/hf_utils.py`. Tests validate native delegation, zero-initialized decoder caches, three-method packaging, and eager numerics. Runtime execution support and runner integration follow in later diffs. Random-weight loading uses checkpoint generation settings when present, warns when the file is absent, and propagates malformed-config errors. Vector generation supports an unset EOS token by using an empty stop set. Authored with assistance from Codex. Reviewed By: digantdesai Differential Revision: D125295057
1 parent 25837ca commit eccf758

7 files changed

Lines changed: 787 additions & 46 deletions

File tree

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
load("@fbcode_macros//build_defs:build_file_migration.bzl", "fbcode_target")
2+
load("@fbsource//xplat/executorch/build:runtime_wrapper.bzl", "runtime")
3+
4+
oncall("executorch")
5+
6+
fbcode_target(
7+
_kind = runtime.python_library,
8+
name = "hf_whisper",
9+
srcs = ["export_hf_whisper.py"],
10+
visibility = ["//executorch/backends/native/test/..."],
11+
deps = [
12+
"//caffe2:torch",
13+
"//executorch/backends/native/examples/hf_llm:hf_utils",
14+
"//executorch/exir:lib",
15+
"//executorch/exir/native:native",
16+
"fbsource//third-party/pypi/transformers:transformers",
17+
],
18+
)
19+
20+
fbcode_target(
21+
_kind = runtime.python_binary,
22+
name = "export_hf_whisper",
23+
main_function = "executorch.backends.native.examples.audio.export_hf_whisper.main",
24+
deps = [
25+
":hf_whisper",
26+
"fbsource//third-party/pypi/safetensors:safetensors",
27+
],
28+
)

0 commit comments

Comments
 (0)