quant method about kv cache

[sitabulaixizawaluduo]

Describe the bug
How can I quant kv cache but not weight?

I tried it like this

recipe = """
quant_stage:
quant_modifiers:
QuantizationModifier:
kv_cache_scheme:
num_bits: 8
type: float
observer: "minmax"
strategy: token
dynamic: true
symmetric: true
"""

but the config.json after quantization has no messages about compressed-tensors.
Does llm-compressor support only quant kv cache now? If can, what shout I do?

[Kha-Zix-1]

Maybe you can quantize kv cache when execute the vllm server? For example

python -m "vllm.entrypoints.openai.api_server" --model "{path}" --tensor-parallel-size {d} --gpu-memory-utilization 0.9 --served-model-name {model} --max-model-len 32768 --max-seq-len-to-capture 32768 --kv-cache-dtype fp8 --block-size 16 --port {port}

use --kv-cache-dtype fp8

[sitabulaixizawaluduo]

Maybe you can quantize kv cache when execute the vllm server? For example

python -m "vllm.entrypoints.openai.api_server" --model "{path}" --tensor-parallel-size {d} --gpu-memory-utilization 0.9 --served-model-name {model} --max-model-len 32768 --max-seq-len-to-capture 32768 --kv-cache-dtype fp8 --block-size 16 --port {port}

use --kv-cache-dtype fp8

I thought llmcompressor could use the kv cache scale per token to achieve higher accuracy and avoid online quantization, but now the quantization result is per tensor

[dsikka]

HI @sitabulaixizawaluduo
Can you share the exact code you ran to quantize your model?
Thanks