Markus Fritsche
4a0bfcde15
Adds optional out_scales[N] (per-output-channel requant scale) to rkt_npu_matmul; per N-tile it uses the tile's channel scale (exact per-channel with tile_n=1). This is what gemma4's per-channel weight quantization needs. Verified on NPU: per-channel test (out_scales[oc]=1/(oc+1)) -> row 144 160 176 192 208 224 240 255 matches CPU exactly. All 5 gemmtest cases pass. Next U6: ggml glue (Q8_0 weights -> per-oc int8 + per-oc out_scale array; F32 activations -> int8 + activation scale; call this primitive). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
36 lines
1.5 KiB
C
36 lines
1.5 KiB
C
/* SPDX-License-Identifier: MIT
|
|
*
|
|
* rkt_npu_matmul.h — reusable arbitrary-size INT8 matmul on the rocket NPU.
|
|
*
|
|
* The U6 core primitive: Y[M][N] = X[M][K] * W[K][N] computed by 2D-tiling
|
|
* into single-op rocket matmuls (rkt_gemm_plan + rkt_build_matmul_regcmd +
|
|
* librocket). Intended to be called from a ggml MUL_MAT backend.
|
|
*/
|
|
#ifndef RKT_NPU_MATMUL_H
|
|
#define RKT_NPU_MATMUL_H
|
|
|
|
#include <stdint.h>
|
|
|
|
/*
|
|
* fd open /dev/accel/accel0 (rocket_open).
|
|
* X [M][K] uint8 row-major activations.
|
|
* Wc [N][K] uint8 conv-order weights (Wc[out_ch][in_ch]).
|
|
* bias [N] int32 per-output-channel bias, or NULL for none.
|
|
* izp/wzp/ozp int8 zero-points (0-255).
|
|
* in/wt/out_scale per-tensor requant scales.
|
|
* tile_m/tile_n desired tile sizes; each (m x n, full-K) tile must satisfy
|
|
* rkt_gemm_op_fits (m limited by CBUF input banks, K*n <= ~327k).
|
|
* Y [M][N] uint8 output (caller-allocated).
|
|
*
|
|
* Returns 0 on success, -1 on error (bad args / tile too big / submit fail).
|
|
* Allocates weights/bias/input/output/regcmd BOs per tile (correctness-first;
|
|
* weight-reuse across M-tiles is a later perf optimization).
|
|
*/
|
|
int rkt_npu_matmul(int fd, const uint8_t *X, const uint8_t *Wc,
|
|
const int32_t *bias, uint32_t M, uint32_t N, uint32_t K,
|
|
uint8_t izp, uint8_t wzp, uint8_t ozp,
|
|
float in_scale, float wt_scale, float out_scale, const float *out_scales,
|
|
uint32_t tile_m, uint32_t tile_n, uint8_t *Y);
|
|
|
|
#endif /* RKT_NPU_MATMUL_H */
|