Deepseek v3 2 exp simple #31
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
but it's a starting point.
based on tilelang and PDF description.
src/llama-sparse-indexer.cpp where it belongs. - Added idx_compute_scores_tile() to encapsulate: - K·Q → ReLU → per-head weight → reduction → K-scale
values in addition to indices. Include a summary with the threshold.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a branch forked from the
deepseek_v3_2_expbranch at commit 86f7d8 on October 28th 2025. This was the last commit before I started exploring cuda kernels.Starts out at about 6.16 tok/s decode on my hardware. I see 17-20 tok/s prefill. Honestly, my best CUDA branch's performance is no better than that. I'm sure it's possible to get better performance on my hardware, but this branch is a win because it is very simple and non-invasive regarding GGML.
Falls into degenerate generation after about 2k tokens, just like my CUDA branches.
NOTE: Setting
LLAMA_SPARSE_TOPK=256causes the degenerate generation to happen at 500-600 context instead of 2k.LLAMA_SPARSE_TOPKdefaults to2048internally. I'm not sure if this is a useful test or if the model just can't work with lower than2048.