Motivation
Quantization is essential to run LLM on the local workstation (12-16 GB of GPU memory). In this post, I summarize my attempt to maximize GPU resources using llama-cpp-python.
The content includes some of my mistakes, as I got into some areas due to my lack of understanding.
[Read More]