Try PLaMo Beta Version

Introduction

I read this article on August 8th. According to the article, a subsidiary of Preferred Networks (PFE) will start offering a free trial of LLM, which has Japanese language performance that exceeds GPT-4, prior to offering a commercial version.

I immediately applied for the free trial, received an email of acceptance, and waited for the notification of account issuance. I had received the notification e-mail on August 9th, but I had overlooked it and completely forgot that I had applied for it. Recently, after reading this post, I remembered about the free trial, rechecked my email, and found the account notification.

In this post, I will summarize what I tried of the free trial version.

Sources

1. PLaMo beta free trial application News release from the originator (Preferred Networks).
2. Trial Application PLaMo beta trial API usage application page. Log in from this page to obtain an API key, and samples of how to use the API are also provided.
3. PLaMo beta free trial Qiita article by an actual user.

Using from jupyterlab

Since it is the same API as ChatGPT, it will be easy to try it out using the code so far.

See this post for the code.

API key settings

Set two environment variables as follows, referring to the API key and the sample code obtained this time.

import os

os.environ['OPENAI_API_KEY'] = 'xxxxxxxxxxxxxx'
os.environ['API_HOST'] = 'URL'

I also referred to the sample code for the llm instance creation part.

How to use PLaMo beta

This time, we decided to pose the question in the following three forms of use.

1. only PLaMo β (used by itself)
2. RAG with PLaMo β (vector store by FAISS)
3. RAG using PLaMo β (ensemble of vector store by FAISS + keyword search by BM25)

Code for ensemble section

The following is the code for the ensemble section, which was not included in the previous postings.

# Prepare Ensemble Retriever

from langchain.retrievers import EnsembleRetriever

ensemble_retriever = EnsembleRetriever(
    retrievers = [bm25_retriever, faiss_retriever],
    weights = [0.5, 0.5]
)

At the same time, the chain section was also changed as follows.

# Build a Chain with LangChain（LCEL）

from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableParallel, RunnablePassthrough

prompt = ChatPromptTemplate.from_template(template)
output_parser = StrOutputParser()
setup_and_retrieval = RunnableParallel(
    {"context": ensemble_retriever, "question": RunnablePassthrough()}
)
chain = setup_and_retrieval | prompt | llm | output_parser

I realized that with langchain, I could try many things with only a few changes to my existing code.

Summary

Execution Result

The question is the same as posted here. The output results are posted as is (line breaks inserted).

Evaluation

The responses are interesting results. The facts are as follows.

1. only the PLaMo beta

   • The Nancy Grace Roman Space Telescope has an aperture of 2.4 meters, not 6.5 meters.
   • The information about the B2FH paper is incorrect.

2. RAG (Vector Store by FAISS)

   • The content of the answer for the “r process” seems to be an explanation of the “rp process”. It seems like a vector search problem to me.

   • The Nancy Grace Roman Space Telescope response, “…detailed information about the specific performance and capabilities of the telescope is not available from the information provided.” This is a good response to the question.

   • In response to the B2FH paper, “In addition, the context provided does not specify the relevance of b(.02) and TOI-1670 to the B2FH paper, so we cannot answer these questions.” also appears to be a vector search issue. “TOI-1670” is a star with an extrasolar planet.

3. RAG (ensemble search of VS and keyword search)

   • The answer for r-process, “It is faster than r-process (rapid neutron capture process).” is incorrect. It seems like a vector search problem to me.

   • Nancy Grace Roman Space Telescope answer, “It will have a primary mirror about 2.4 meters in diameter and will be located at the L2 Lagrangian point, about 15,000 light years from Earth.” The description of the aperture is correct, but the L2 Lagrangian point is located 1.5 million Km from the earth. Looks like it got the units wrong.

   • I don’t know why it mention the “anthropic principle” in your answer to the B2FH paper.

Finally

I think it is good that the RAG by VS includes “don’t know” or “cannot answer” responses. This is because the template includes the following statement: “In addition, please do not include information that is not in the context information in your response. If you cannot derive an answer from the context information, please answer “I don’t know. The reason for this is that the template is set to “I don’t know”.

Actually, I had conducted ensemble search using the “Llama-3-ELYZA-JP-8B-q4_k_m.gguf” model, but the results were not so good, so I had refrained from evaluating ensemble search, but when I used PLaMo β this time, I obtained relatively good results. However, when I used PLaMo β this time, I were able to obtain relatively good results.