125 lines
5.3 KiB
Markdown
125 lines
5.3 KiB
Markdown
---
|
|
order: 2
|
|
title: "Lab 2 - Quantization Tradeoffs: Comparing 2-bit, 4-bit, and 8-bit"
|
|
description: Download Gemma 4 E2B in three GGUF quantizations and compare size, metadata, and output quality.
|
|
---
|
|
|
|
<!-- breakout-style: instruction-rails -->
|
|
<!-- step-style: underline -->
|
|
<!-- objective-style: divider -->
|
|
|
|
In this lab, we will:
|
|
|
|
- Download the same Gemma model in `UD-IQ2_M`, `Q4_K_M`, and `Q8_0`
|
|
- Compare file size and GGUF metadata across those quantizations
|
|
- Observe how lower precision changes the model's behavior
|
|
- Build intuition for when a smaller quant may or may not be worth it
|
|
|
|
<div class="lab-callout lab-callout--info">
|
|
<strong>Lab Flow Guide</strong><br />
|
|
<strong>Explore</strong> sections focus on comparison and trade-off analysis.<br />
|
|
<strong>Execute</strong> sections require collecting evidence from each quantized model.
|
|
</div>
|
|
|
|
## Objective 1: Understand the Model and the Quantizations
|
|
|
|
For this lab, we will use the Hugging Face repository for **Unsloth's GGUF release of Gemma 4 E2B Instruct**:
|
|
|
|
<https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF>
|
|
|
|
This repository currently exposes multiple GGUF variants of the same base model. We will focus on one file from each of these precision bands:
|
|
|
|
| Precision band | GGUF file | Why we are using it | File Size |
|
|
| -------------- | ------------------------------ | --------------------------------------- |-----------|
|
|
| 2-bit | `gemma-4-E2B-it-UD-IQ2_M.gguf` | Most aggressive compression in this lab | 2.4 GB |
|
|
| 4-bit | `gemma-4-E2B-it-Q4_K_M.gguf` | Common middle-ground quant | 3.17 GB |
|
|
| 8-bit | `gemma-4-E2B-it-Q8_0.gguf` | Highest-quality quant in this lab | 5.05 GB |
|
|
|
|
Even though the filenames differ, these are all the same underlying instruction-tuned Gemma 4 E2B model. The main variable we are changing is how the weights are stored.
|
|
|
|
When we say these files are the same model, we mean that the overall neural network is still the same:
|
|
|
|
- The same architecture
|
|
- The same layer count
|
|
- The same tokenizer
|
|
- The same training and instruction tuning
|
|
- The same general behavior the model learned during training
|
|
|
|
What changes is the numeric representation of the learned weights.
|
|
|
|
Imagine one learned weight in the original model is:
|
|
|
|
```text
|
|
0.156347
|
|
```
|
|
|
|
That number came from training. It is one of many values the model uses while computing each next token. Quantization does not invent a new model from scratch. Instead, it takes that trained value and asks:
|
|
|
|
```text
|
|
How can we store a close-enough version of this number using fewer bits?
|
|
```
|
|
|
|
If we use a simplified integer-style quantization scheme, the math looks like this:
|
|
|
|
```text
|
|
scale = max(|w|) / (2^(bits - 1) - 1)
|
|
q = round(w / scale)
|
|
w_hat = q * scale
|
|
```
|
|
|
|
Where:
|
|
|
|
- `w` is the original weight
|
|
- `q` is the stored integer bucket
|
|
- `scale` maps integers back into the original numeric range
|
|
- `w_hat` is the reconstructed approximation used at inference time
|
|
|
|
So if the original trained value was `0.156347`, a lower-bit quantized file may not store that exact number anymore. It may store an integer bucket like `1`, `5`, or `22`, plus a scale, and reconstruct an approximation such as:
|
|
|
|
- `0.000000`
|
|
- `0.130029`
|
|
- `0.146806`
|
|
- `0.157782`
|
|
|
|
Those are not identical to the original weight, but they may still be close enough for useful inference.
|
|
|
|
<div data-quantization-explorer></div>
|
|
|
|
### Explore: Interactive precision viewer
|
|
|
|
The viewer below zooms out from one weight and instead shows a toy layer with 16 stored values. Real GGUF schemes such as `Q4_K_M` and `UD-IQ2_M` are more sophisticated than this toy example, but the core idea is the same:
|
|
|
|
- Fewer bits means fewer representable values
|
|
- More weights get pushed into the same small set of stored buckets
|
|
- The layer becomes more compressed as precision drops
|
|
|
|
<div data-quantization-grid-explorer></div>
|
|
|
|
### Explore: Compare the same prompts through the hosted chat widget
|
|
|
|
If your instructor provides an OpenAI-compatible endpoint, you can compare the same prompts through the embedded chat tool below:
|
|
|
|
- Paste the lab endpoint and API key into the settings row
|
|
- Switch between `Q8_0`, `Q4_K_M`, and `UD-IQ2_M`
|
|
- Re-run the same prompt so you can compare coherence, stability, and SVG output
|
|
- Try a visual prompt such as `Draw a pelican riding a bicycle.`
|
|
|
|
The widget keeps the transcript in your browser so you can switch models without losing your place. Refresh the page to clear the chat history.
|
|
|
|
<div data-objective5-chat></div>
|
|
|
|
## Objective 6: Reflect on the Tradeoff
|
|
|
|
By this point, you should have:
|
|
|
|
- Compared three quantized versions of the same model
|
|
- Measured the storage savings directly
|
|
- Verified that the core model metadata remains largely the same
|
|
- Observed where output quality begins to degrade
|
|
|
|
The important takeaway is not that one quant is always "best." The important takeaway is that quantization is a deployment decision. The right choice depends on your hardware limits, acceptable quality loss, and the task you need the model to perform.
|
|
|
|
## Conclusion
|
|
|
|
This lab isolates quantization as the main variable. By downloading **Gemma 4 E2B Instruct** in `UD-IQ2_M`, `Q4_K_M`, and `Q8_0`, you can directly observe one of the most important tradeoffs in local inference: balancing model quality against disk usage and resource constraints.
|