DeepSeek-V3.2-Exp

[createthis]

Don't merge. WIP.

When switching to this branch from deepseek_v3_2_exp_simple, you need to run:

git submodule update --init --recursive

Then recompile the project (I am assuming you have a single blackwell 6000 pro and 768gb of system ram):

rm -Rf build && cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES="120"
cmake --build build --config Release

You can add LLAMA_SPARSE_PROF=1 to get performance profiling of the kernels.

There are a lot of different kernels in this branch gated by env vars. This is probably the most interesting config
as it uses the vendored VLLM top-k kernel:

LLAMA_FP8_INDEXER_CACHE=1 LLAMA_SPARSE_TOPK_VLLM=1 ./build/bin/llama-server \
    --model  /data2/DeepSeek-V3.2-Exp-GGUF/q4_k_m/DeepSeek-V3.2-Exp-Q4_K_M-00001-of-00009.gguf \
    --alias DeepSeek-V3.2-Exp:671b-q4_k_m \
    --no-webui \
    --numa numactl \
    --threads 32 \
    --ctx-size 163840 \
    --n-gpu-layers 62 \
    -ot "blk\.(3|4|5|6|7|8|9)\.ffn_.*=CUDA0" \
    -ot exps=CPU \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 0.6 \
    --top-p 0.95 \
    --min-p 0.1 \
    --log-colors on \
    --flash-attn on \
    --host 0.0.0.0 \
    --prio 2 \
    --jinja \
    --port 11434

3.96 tok/s

[PROFILE_FP8_GATHER] TILELANG_INDEXER D=128 H=64 Tc=1 kv=163840 avg_ms=0.019 over 50 calls
[PROFILE_WMMA_HGRP_ONLY] TILELANG_INDEXER D=128 H=64 Tc=1 kv=163840 avg_ms=0.173 over 50 calls
[PROFILE] IDX_TILE CUDA D=128 H=64 Tc=1 kv=163840 avg_ms=0.205 over 50 calls
[PROFILE] SPARSE_TOPK_RADIX2_VLLM N=163840 T=1 k=2048 avg_ms=0.192 over 50 calls
[PROFILE] SPARSE_MLA_DECODE D=576 Hq=128 Hkv=1 Dv=512 Nkv=163840 K=2048 avg_ms=1.805 over 50 calls