Qwen2.5MoE instability

[sempervictus]

Seeing some "drunken behavior" on the part of Qwen/Qwen2-57B-A14B-Instruct and DavidAU/Qwen2.5-2X11B-CODER-Dueling-Wolverines-28B i attempted to adjust their parameters to stabilize them a bit - --penalty 1.06 --temperature 0.5 --top-k 40 --top-p 0.95. Upstream model seems to be OK but the 2x11b model which works without those runtime params now blows up on-request with:

thread 'tokio-runtime-worker' (838) panicked at src/openai/logits_processor.rs:151:51:
called `Result::unwrap()` on an `Err` value: A weight is negative, too large or not a valid number
   0: candle_core::error::Error::bt
   1: candle_vllm::openai::logits_processor::LogitsProcessor::sample_multinomial
   2: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
   3: rayon::iter::plumbing::Folder::consume_iter
   4: rayon::iter::plumbing::bridge_producer_consumer::helper
   5: rayon::iter::collect::collect_with_consumer
   6: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
   7: rayon::iter::from_par_iter::collect_extended
   8: candle_vllm::openai::logits_processor::LogitsProcessor::sample
   9: candle_vllm::openai::pipelines::pipeline::DefaultPipeline::sample
  10: candle_vllm::openai::pipelines::llm_engine::LLMEngine::generate_once
  11: rayon::iter::plumbing::Folder::consume_iter
  12: rayon::iter::plumbing::bridge_producer_consumer::helper
  13: rayon::iter::collect::collect_with_consumer
  14: tokio::runtime::handle::Handle::block_on
  15: tokio::runtime::task::raw::poll
  16: std::sys::backtrace::__rust_begin_short_backtrace
  17: core::ops::function::FnOnce::call_once{{vtable.shim}}
  18: std::sys::pal::unix::thread::Thread::new::thread_start
  19: <unknown>
  20: clone

which looks like what we saw with the 2x9 model without any parameters.

Something's still a bit off here @guoqingbao. Any chance the data here provides any insight to localization or cause?

[sempervictus]

@guoqingbao - looks like qwen3 moe models might also be susceptible to "drunken loops" of long outputs now. Checking to see if runtime tuning params are sufficient or "if more must be done"

[guoqingbao]

I think your observation is under expectation. By default, we use model's generation_config.json file to setup the sampling parameters, for example the sampling config of the upstream model you mentioned: https://huggingface.co/Qwen/Qwen2-57B-A14B-Instruct/blob/main/generation_config.json, while, the custom model DavidAU/Qwen2.5-2X11B-CODER-Dueling-Wolverines-28B does not provide such configation file, instead, the author said users' need to use the following parameters:

Requirements [Qwen 2.5 7B Coder default settings]:

Temp .5 to .7 (or lower)
topk: 20, topp: .8, minp: .05
rep pen: 1.1 (can be lower)
Jinja Template (embedded) or CHATML template.
A System Prompt is not required. (ran tests with blank system prompt)

If no sampling config was given, it is likely for the custom model to generate garbage outputs.

[sempervictus]

Hmm, we dont have a minp setting :-).

[guoqingbao]

Hmm, we dont have a minp setting :-).

I will add that but this can be ignored in most cases.