docs: add empirical energy efficiency data to quantization concept guide

[hongping-zh]

Summary

Adds an "Energy efficiency in practice" section to the quantization concept guide (docs/source/concept_guides/quantization.mdx), providing empirical data that addresses the existing "(in theory)" qualifier in the introduction.

Motivation

The current documentation states that quantization "consumes less energy (in theory)." Through systematic benchmarking across multiple GPU architectures and model sizes, we found that this assumption does not always hold — particularly for small models (<3B parameters) where dequantization overhead can increase energy consumption by 25–56%.

This addition provides practitioners with concrete, data-backed guidance on when quantization helps (and when it doesn't) from an energy perspective.

Changes

Added a new subsection covering:

• Large models (≥5B): NF4 achieves near-FP16 energy with memory savings ✓
• Small models (<3B): NF4 can increase energy by 25–56% despite 75% memory compression
• INT8 mixed-precision: 17–33% energy overhead as a justified accuracy trade-off
• Batch size: 84–96% per-token energy reduction from BS=1 to BS=8–64

Data Source

• Hardware: NVIDIA RTX 4090D (Ada Lovelace), RTX 5090 (Blackwell), A800
• Models: TinyLlama-1.1B through Qwen2.5-14B (1B–14B parameter range)
• Methodology: NVML power monitoring at 10 Hz, repeated trials, CV < 3%
• Full dataset: Zenodo
• Toolkit: EcoCompute-AI
• Interactive dashboard: https://hongping-zh.github.io/ecocompute-dynamic-eval/

Related

• bitsandbytes PR: bitsandbytes-foundation/bitsandbytes#1882
• Transformers PR: huggingface/transformers#44407 (approved by @SunMarc)

Notes

• Content follows existing .mdx formatting style and uses the <Tip> component
• The section is concise (~200 words) and focuses on practical implications
• Happy to adjust scope, framing, or placement based on maintainer feedback

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[regisss]

Interesting!

[hongping-zh]

Thank you again for the merge, Régis! I've been thinking about extending this into a cross-backend energy benchmark (GPTQ/AWQ/GGUF). Would email or HF community be the best way to share a brief proposal?

[hongping-zh]

Thanks for the pointer to the serving documentation.

One observation worth sharing: after filing this issue, the PyTorch torchao team confirmed that energy efficiency is not a priority for torchao, and that native Hugging Face inference is not their target path — their focus is on serving engines like vLLM/SGLang with torch.compile.

This creates an important gap: thousands of developers and researchers use HF Transformers directly for inference — many without realizing that their quantization choices may actually increase energy consumption rather than reduce it. Our benchmarks show that for models under ~4B parameters, NF4 quantization raises energy by 20–55%, and FP8 (torchao) can incur up to +701% energy penalty in the current eager-mode path.

This is exactly why we built EcoCompute AI — to provide systematic, hardware-aware energy benchmarks so developers can make informed deployment decisions. We believe energy efficiency metrics deserve the same attention as throughput and latency in the ML ecosystem.

We'd welcome any collaboration or feedback from the HF team on integrating energy-awareness into the Transformers ecosystem.

[hongping-zh]

Thanks for the pointer to the serving documentation.

One observation worth sharing: after filing this issue, the PyTorch torchao team confirmed that energy efficiency is not a priority for torchao, and that native Hugging Face inference is not their target path — their focus is on serving engines like vLLM/SGLang with torch.compile.

This creates an important gap: thousands of developers and researchers use HF Transformers directly for inference — many without realizing that their quantization choices may actually increase energy consumption rather than reduce it. Our benchmarks show that for models under ~4B parameters, NF4 quantization raises energy by 20–55%, and FP8 (torchao) can incur up to +701% energy penalty in the current eager-mode path.

This is exactly why we built EcoCompute AI — to provide systematic, hardware-aware energy benchmarks so developers can make informed deployment decisions. We believe energy efficiency metrics deserve the same attention as throughput and latency in the ML ecosystem.

We'd welcome any collaboration or feedback from the HF team on integrating energy-awareness into the Transformers ecosystem.

[hongping-zh]

Hi @regisss, hope you're doing well! Quick update on the energy efficiency work since you merged this PR — and a request.

Progress since PR merge

The dataset behind this documentation has grown significantly:

• 113+ configurations now covering five precision methods: FP16, NF4, INT8 (default), INT8 (pure bnb), and FP8
• FP8 (torchao) Paradox discovered: up to +701% energy overhead on RTX 5090 Blackwell — filed as torchao Issue #4094, and the torchao maintainers confirmed that energy efficiency is not their current priority and native HF inference is not their target path
• Dataset mirrored on HF Hub: hongpingzhang/ecocompute-energy-efficiency
• Zenodo archive: DOI 10.5281/zenodo.18900289
• Interactive AI advisor: EcoCompute on ClawHub — lets developers query these benchmarks conversationally

This gap matters: thousands of developers use HF Transformers directly for quantized inference, and many don't realize their quantization choices may increase energy consumption. For models under ~4B parameters, NF4 raises energy by 20–55%, and FP8 can be 7x worse in the current eager-mode path.

Request: brief feedback for future research

I'm preparing a research paper based on this work: "EcoCompute: Energy Efficiency Benchmark for Quantized Language Models". The paper formalizes the findings that are now part of the Optimum official documentation, including the quantization paradoxes and the five-precision benchmark methodology.

Since you reviewed and merged the original data into HuggingFace's official docs, would you be open to providing brief feedback on the research direction? Even a simple confirmation like:

"The energy efficiency data presented in this work has been reviewed and incorporated into the HuggingFace Optimum official documentation, providing practical value to the ML community."

This would help establish credibility for the research direction, especially since the data is already trusted enough to be part of HF's official guides.

Completely understand if this isn't possible — happy to discuss via email: contact@hongping-zh.com

Thanks again for your support on this work, Régis! Looking forward to continuing the energy efficiency conversation.