marfrit/rk-llama.cpp
1
0
Fork 0
Code Issues Pull Requests Actions 32 Packages Projects Releases Wiki Activity

rknpu2: saturate int8 activation quantization

Browse Source 
quantize_fp32_to_int8 cast roundf(v) straight to int8_t. With scale = amax/127
the value should never exceed 127, but one ULP of error in iscale can round to
128, which wraps to -128 and injects a sign-flipped activation outlier. The
int4 path already clamped to +/-7; int8 did not.

Latent, not active: greedy output on gemma-4-12b-it-Q8_0 is byte-identical
before and after (sha 11b2174379e0c2040246, 708 B). This is insurance.

Found by a source review of the NPU math path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWpfhDgYNA21tETDP9ueBE
This commit is contained in:
mfritsche
2026-07-10 09:01:57 +02:00
parent 3cd6e67b55
commit 447672d647
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-rknpu2/rknpu2-quantization.cpp
+5 -1
View File
@@ -26,7 +26,11 @@ void convert_fp32_to_fp16(const float * src, uint16_t * dst, size_t n_elements)
void quantize_fp32_to_int8(const float * src, int8_t * dst, size_t n_elements, float scale) {
const float iscale = (scale == 0.0f) ? 0.0f : 1.0f / scale;
for (size_t i = 0; i < n_elements; ++i) {
dst[i] = (int8_t)roundf(src[i] * iscale);
// Saturate. scale = amax/127 should keep |v| <= 127, but one ULP of error in
// iscale can round to 128, which wraps to -128 and injects a sign-flipped
// outlier. The int4 path below already clamps.
const float v = roundf(src[i] * iscale);
dst[i] = (int8_t)std::max(-127.0f, std::min(127.0f, v));
}
}