DeepSeek V4 Roadmap

[fzyzcjy]

Information

Please temporarily refer to #23600 for details

Example performance

TODO

Roadmap

• Finish W4A16 support on Hopper
• Support non-standard extra chat template feature
• Support pipeline parallel
• Optimize MegaMoE (implemented, to be tested)
• Further integrate and optimize various kernels
• HiCache support
• TODO

[ishandhanani]

You're a beast Tom

[Azure-Tang]

sooooo beast

[DarkSharpness]

Some verified launch commands if you don't want to use docker images:

# Hopper TP4 FP8 low latency
git switch deepseek_v4
pip install -e python
pip install tilelang==0.1.8
pip install apache-tvm-ffi==0.1.9
pip install git+https://github.com/deepseek-ai/FlashMLA/ --no-build-isolation

SGLANG_DSV4_FP4_EXPERTS=0 \
SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 \
sglang serve \
  --trust-remote-code \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --tp 4 \
  --speculative-algo EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 \
  --port 30000

# Hopper Balance
git switch deepseek_v4
pip install -e python
pip install tilelang==0.1.8
pip install apache-tvm-ffi==0.1.9
pip install git+https://github.com/deepseek-ai/FlashMLA/ --no-build-isolation

# NOTE: need to install latests DeepGEMM and DeepEP (git clone and run `install.sh`)

SGLANG_DSV4_FP4_EXPERTS=0 \
SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256 \
SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 \
sglang serve \
  --trust-remote-code \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --tp 4 \
  --dp 4 \
  --enable-dp-attention \
  --moe-a2a-backend deepep \
  --speculative-algo EAGLE \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 2 \
  --cuda-graph-max-bs 128 \
  --max-running-requests 128 \
  --deepep-config '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}' \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 \
  --port 30000

[mmangkad]

Using the docker image,

4xB300

SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 \
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Pro \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algo EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --chunked-prefill-size 4096 \
  --disable-flashinfer-autotune \
  --mem-fraction-static 0.82 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 \
  --port 30000

/workspace/sglang/python/sglang/srt/layers/attention/fla/utils.py:215: UserWarning: Triton is not supported on current platform, roll back to CPU.
  warnings.warn(
Traceback (most recent call last):
  File "/usr/local/bin/sglang", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/workspace/sglang/python/sglang/cli/main.py", line 42, in main
    args.func(args, extra_argv)
  File "/workspace/sglang/python/sglang/cli/serve.py", line 71, in serve
    server_args = prepare_server_args(extra_argv)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/sglang/python/sglang/srt/server_args.py", line 5410, in prepare_server_args
    return ServerArgs.from_cli_args(raw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/sglang/python/sglang/srt/server_args.py", line 4892, in from_cli_args
    return cls(**{attr: getattr(args, attr) for attr in attrs})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 330, in __init__
  File "/workspace/sglang/python/sglang/srt/server_args.py", line 709, in __post_init__
    self._handle_missing_default_values()
  File "/workspace/sglang/python/sglang/srt/server_args.py", line 835, in _handle_missing_default_values
    self.device = get_device()
                  ^^^^^^^^^^^^
  File "/workspace/sglang/python/sglang/srt/utils/common.py", line 1959, in get_device
    raise RuntimeError("No accelerator (CUDA, XPU, HPU, NPU) is available.")
RuntimeError: No accelerator (CUDA, XPU, HPU, NPU) is available.

8xB200

SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 \
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Pro \
  --tp 8 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algo EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --chunked-prefill-size 4096 \
  --disable-flashinfer-autotune \
  --mem-fraction-static 0.82 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 \
  --port 30000

Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, 131.90it/s]
[2026-04-24 06:29:09] WARNING hf_transformers_utils.py:193: SGLANG_APPLY_CONFIG_BACKUP=auto: checkpoint has num_hidden_layers=61, dispatching to 'large'.
[2026-04-24 06:29:09] WARNING hf_transformers_utils.py:212: SGLANG_APPLY_CONFIG_BACKUP=large: using packaged /workspace/sglang/python/sglang/srt/configs/config_backup_large.json instead of the checkpoint's config.json at /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Pro/snapshots/0366e4e064385807ea86b088a5c6c878ff23343b.
[2026-04-24 06:29:10] INFO model_config.py:448: Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-04-24 06:29:10] INFO server_args.py:1183: Use compressed attention backend for DeepseekV4ForCausalLM, setting page_size to 256.
[2026-04-24 06:29:10] WARNING server_args.py:1189: Setting max_running_requests to 256 for DeepseekV4ForCausalLM.
[2026-04-24 06:29:10] WARNING server_args.py:1195: Setting KV cache dtype to fp8_e4m3 for DeepseekV4ForCausalLM.
[2026-04-24 06:29:10] WARNING server_args.py:1212: Spec v2 is enabled for EAGLE speculative decoding.
[2026-04-24 06:29:10] INFO server_args.py:1216: Setting swa_full_tokens_ratio to 0.1 for DeepseekV4ForCausalLM.
[2026-04-24 06:29:10] WARNING server_args.py:2275: Spec v2 is enabled for eagle/eagle3 speculative decoding and overlap schedule is turned on.
<frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
[2026-04-24 06:29:10] server_args=ServerArgs(model_path='deepseek-ai/DeepSeek-V4-Pro', tokenizer_path='deepseek-ai/DeepSeek-V4-Pro', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{"enable_multithread_load": true, "num_threads": 64}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=30000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='fp8_e4m3', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.82, max_running_requests=256, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=256, swa_full_tokens_ratio=0.1, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='cuda', tp_size=8, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=11713072, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='deepseek-ai/DeepSeek-V4-Pro', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='deepseek-v4', tool_call_parser='deepseekv4', tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='compressed', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', nsa_prefill_backend=None, nsa_decode_backend=None, disable_flashinfer_autotune=True, speculative_algorithm='EAGLE', speculative_draft_model_path='deepseek-ai/DeepSeek-V4-Pro', speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=3, speculative_eagle_topk=1, speculative_num_draft_tokens=4, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='flashinfer_mxfp4', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='flashinfer_mxfp4', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_hisparse=False, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=512, cuda_graph_bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48, 52, 56, 60, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='in-seq-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_enable_fake_auto=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-04-24 06:29:10] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-04-24 06:29:12] No HuggingFace chat template found
[2026-04-24 06:29:12] No chat template found, defaulting to 'string' content format
<frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
[2026-04-24 06:29:19 TP4] SGLANG_APPLY_CONFIG_BACKUP=auto: checkpoint has num_hidden_layers=61, dispatching to 'large'.
[2026-04-24 06:29:19 TP4] SGLANG_APPLY_CONFIG_BACKUP=large: using packaged /workspace/sglang/python/sglang/srt/configs/config_backup_large.json instead of the checkpoint's config.json at /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Pro/snapshots/0366e4e064385807ea86b088a5c6c878ff23343b.
[2026-04-24 06:29:19 TP4] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-04-24 06:29:20 TP4] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-04-24 06:29:20 TP4] Init torch distributed begin.
[2026-04-24 06:29:20 TP4] Context: self.device='cuda' self.gpu_id=4 os.environ.get('CUDA_VISIBLE_DEVICES')='4,5,6,7' self.tp_rank=4 self.tp_size=8
[2026-04-24 06:29:20 TP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2989, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 350, in __init__
    self.init_model_worker()
  File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 539, in init_model_worker
    self.init_tp_model_worker()
  File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 501, in init_tp_model_worker
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 242, in __init__
    self._init_model_runner()
  File "/workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 325, in _init_model_runner
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 377, in __init__
    self.total_gpu_memory = self.init_torch_distributed()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 752, in init_torch_distributed
    torch.get_device_module(self.device).set_device(self.gpu_id)
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 567, in set_device
    torch._C._cuda_setDevice(device)
torch.AcceleratorError: CUDA error: invalid device ordinal
GPU device may be out of range, do you have enough GPUs?
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[huangmengwei-cmss]

Any plans for sm_80 support?

[zhangxiaolei123456]

we use this command
my hardware: h20
image:
docker pull lmsysorg/sglang:deepseek-v4-hopper
command:

SGLANG_DSV4_FP4_EXPERTS=0 SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 sglang serve --trust-remote-code --model-path /data00/models/DeepSeek-V4-Flash-Base --tp 8 --cuda-graph-max-bs 256 --max-running-requests 256 --enable-metrics --host 0.0.0.0 --port 30000 --mem-fraction-static 0.8 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4

the result is error

curl -X POST http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{
    "model": "deepseek-v4-base",
    "messages": [
        {
            "role": "user",
            "content": "做一份北京出行攻略"
        }
    ],
    "max_tokens": 500,
    "temperature": 0.6
}'
{"id":"436efe87f67d421fba749c48527ef422","object":"chat.completion","created":1777012095,"model":"deepseek-v4-base","choices":[{"index":0,"message":{"role":"assistant","content":" dével dével dével dével dével dével dével dével dével económ buruj buruj buruj buruj dével buruj تضيفلها buruj dével buruj buruj buruj buruj Pagklasipikar buruj dével buruj buruj buruj Britonhon burujانزياحGiyaennettunaGiya Britonhon Britonhon Britonhon Britonhon Britonhonennettunaennettuna BritonhonGiya BritonhonGiyaPreferences Britonhonennettunarilsennettunaennettunaennettunaennettunaennettuna _“_ennettunaennettunaennettunaennettunaennettuna _“_ _“_ _“_ _“_ _“_ennettunaennettunaIDAD _“_ Britonhonobyl _“_ _“_ _“_ _“_ ninete _“_ _“_ _“_Kadaghanon _“_ _“_ _“_ _“_Kadaghanon _“_Kadaghanon _“_KadaghanonKadaghanon _“_KadaghanonKadaghanonKadaghanonKadaghanon _“_Kadaghanon _“_Kadaghanon Towards _“_KadaghanonKadaghanonKadaghanonKadaghanonPreferencesKadaghanonKadaghanonKadaghanon _“_Kadaghanon _“_ _“_KadaghanonKadaghanonKadaghanonKadaghanonekakKadaghanonKadaghanonKadaghanon _“_ _“_Kadaghanon _“_ _“_ _“_ _“_ _“_Kadaghanon _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ citiz _“_ _“_ _“_ _“_ _“_ _“_rón _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ مفصله _“_bull _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_木瓜 _“_ مفصله _“_ _“_ _“_ _“_ _“_ _“_ _“_ مفصله _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ مفصله مفصله مفصله مفصله مفصله _“_ _“_ _“_ _“_ _“_ مفصله _“_ Abigail _“_ مفصله _“_ _“_ _“_ _“_ مفصله _“_ مفصله مفصله مفصله مفصله مفصله مفصله مفصله مفصله مفصله مفصله مفصله _“_ مفصله مفصله _“_ _“_ مفصله مفصله _“_ مفصله _“_ _“_ _“_ مفصله مفصله مفصله _“_ _“_ _“_ _“_ _“_ _“_ مفصله _“_ _“_ _“_ _“_ مفصله _“_ _“_ _“_ _“_ مفصله _“_ _“_ _“_ _“_ مفصله مفصله مفصله _“_ مفصله _“_ مفصله _“_ _“_ _“_ _“_ _“_ _“_ مفصله _“_ مفصله _“_ _“_ مفصله _“_ _“_ مفصله مفصله _“_ _“_ _“_ _“_ _“_ _“_ _“_ المطلع _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_ _“_","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":9,"total_tokens":509,"completion_tokens":500,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}

[fzyzcjy]

@zhangxiaolei123456 maybe use the main model instead of the base model

[mmangkad]

#23602 (comment) can be fixed by manually setting CUDA_VISIBLE_DEVICES. The image seems to hardcode CUDA_VISIBLE_DEVICES=4,5,6,7.