marfrit/rk-llama.cpp
1
0
Fork 0
Code Issues Pull Requests Actions 27 Packages Projects Releases Wiki Activity

rknpu2: add Q4_K dequant + W4A4_HADAMARD default pattern

Browse Source 
Q4_K weights previously hit the "Unsupported weight type for NPU pipeline"
assert and fell back to CPU. Route them through dequantize_row_q4_K and map
GGML_TYPE_Q4_K to W4A4_HADAMARD, matching the existing Q4_0 handling.

Note: W4A4_HADAMARD is the lower-accuracy of the two W4A4 pipelines
(bench: 3.17 vs 3.43 tg64 for standard). This is an enablement, not a
performance choice.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
This commit is contained in:
mfritsche
2026-07-09 23:24:08 +02:00
parent 05cf2f9dd5
commit 137b4ceebe
2 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-rknpu2/ggml-rknpu2.cpp
+3
View File
@@ -958,6 +958,9 @@ static void dequantize_row(
} else if (tensor->type == GGML_TYPE_Q4_0) {
const block_q4_0* src = (const block_q4_0*)raw_data;
dequantize_row_q4_0(src + (size_t)n * (K / QK4_0), row_out, K);
} else if (tensor->type == GGML_TYPE_Q4_K) {
const block_q4_K* src = (const block_q4_K*)raw_data;
dequantize_row_q4_K(src + (size_t)n * (K / QK_K), row_out, K);
} else {
GGML_ASSERT(false && "Unsupported weight type for NPU pipeline");
}
ggml/src/ggml-rknpu2/rknpu2-configuration.cpp
+1
View File
@@ -212,6 +212,7 @@ Rknpu2ConfigManager::Rknpu2ConfigManager() {
rk3588_config.default_patterns[(int)GGML_TYPE_Q8_0] = {"W8A8_STANDARD"};
rk3588_config.default_patterns[(int)GGML_TYPE_Q6_K] = {"W8A8_STANDARD", "W4A4_HADAMARD"};
rk3588_config.default_patterns[(int)GGML_TYPE_Q4_0] = {"W4A4_HADAMARD"};
rk3588_config.default_patterns[(int)GGML_TYPE_Q4_K] = {"W4A4_HADAMARD"};
device_configs["RK3588"] = rk3588_config;