https://akenji3.github.io/en/tags/elyza/ Recent content in Elyza on akenji's lab Hugo en Fri, 03 May 2024 00:00:00 +0000 https://akenji3.github.io/en/post/20240503_llama-cpp-python/ Fri, 03 May 2024 00:00:00 +0000 https://akenji3.github.io/en/post/20240503_llama-cpp-python/ <h2 id="motivation">Motivation</h2> <p>Quantization is essential to run LLM on the local workstation (12-16 GB of GPU memory). In this post, I summarize my attempt to maximize GPU resources using llama-cpp-python.</p> <p>The content includes some of my mistakes, as I got into some areas due to my lack of understanding.</p>