使用 Finetuning API 微调 MistralAI 模型¶
本笔记本将演示如何使用 MistralAI 的微调 API 对 open-mistral-7b 进行微调。
具体而言，我们尝试通过 mistral-large-latest 生成训练数据来实现知识蒸馏，进而微调 open-mistral-7b 模型。
所有训练数据均来自索引数据的两个不同部分，分别生成训练集和评估集。
为避免对 open-mistral-7b 和 mistral-large-latest 产生偏向性，我们将使用 mistral-small-largest 创建合成的训练和评估问题。
随后通过我们封装的 MistraAIFinetuneEngine 抽象层进行微调。
评估环节使用 ragas 库实现（后文将详细说明）。
可通过 Weights & Biases 平台监控各项指标。
In [ ]:
Copied!
%pip install llama-index-finetuning
%pip install llama-index-finetuning-callbacks
%pip install llama-index-llms-mistralai
%pip install llama-index-embeddings-mistralai
%pip install llama-index-finetuning
%pip install llama-index-finetuning-callbacks
%pip install llama-index-llms-mistralai
%pip install llama-index-embeddings-mistralai
In [ ]:
Copied!
# !pip install llama-index pypdf sentence-transformers ragas
# !pip install llama-index pypdf sentence-transformers ragas
In [ ]:
Copied!
# NESTED ASYNCIO LOOP NEEDED TO RUN ASYNC IN A NOTEBOOK
import nest_asyncio
nest_asyncio.apply()
# NESTED ASYNCIO LOOP NEEDED TO RUN ASYNC IN A NOTEBOOK
import nest_asyncio
nest_asyncio.apply()
设置 API 密钥¶
In [ ]:
Copied!
import os
os.environ["MISTRAL_API_KEY"] = "<YOUR MISTRALAI API KEY>"
import os
os.environ["MISTRAL_API_KEY"] = ""
下载数据¶
在此，我们首先下载将用于生成训练数据的 PDF 文件。
In [ ]:
Copied!
!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf
!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20.7M 100 20.7M 0 0 249k 0 0:01:25 0:01:25 --:--:-- 266k0 0:01:31 0:00:25 0:01:06 267k 251k
下一步是生成训练和评估数据集。
我们将基于下载的PDF文档的不同章节生成40个训练问题和40个评估问题。
可以使用open-mistral-7b模型处理评估问题来获取基线性能指标。
然后，我们将使用mistral-large-latest模型处理训练问题来生成训练数据。
加载数据¶
In [ ]:
Copied!
from llama_index.core import SimpleDirectoryReader
from llama_index.core.evaluation import DatasetGenerator
documents = SimpleDirectoryReader(
input_files=["IPCC_AR6_WGII_Chapter03.pdf"]
).load_data()
from llama_index.core import SimpleDirectoryReader
from llama_index.core.evaluation import DatasetGenerator
documents = SimpleDirectoryReader(
input_files=["IPCC_AR6_WGII_Chapter03.pdf"]
).load_data()
配置大语言模型与嵌入模型¶
In [ ]:
Copied!
from llama_index.llms.mistralai import MistralAI
from llama_index.embeddings.mistralai import MistralAIEmbedding
open_mistral = MistralAI(
model="open-mistral-7b", temperature=0.1
) # model to be finetuning
mistral_small = MistralAI(
model="mistral-small-latest", temperature=0.1
) # model for question generation
embed_model = MistralAIEmbedding()
from llama_index.llms.mistralai import MistralAI
from llama_index.embeddings.mistralai import MistralAIEmbedding
open_mistral = MistralAI(
model="open-mistral-7b", temperature=0.1
) # model to be finetuning
mistral_small = MistralAI(
model="mistral-small-latest", temperature=0.1
) # model for question generation
embed_model = MistralAIEmbedding()
训练与评估数据生成¶
In [ ]:
Copied!
question_gen_query = (
"You are a Teacher/ Professor. Your task is to setup "
"a quiz/examination. Using the provided context, formulate "
"a single question that captures an important fact from the "
"context. Restrict the question to the context information provided."
"You should generate only question and nothing else."
)
dataset_generator = DatasetGenerator.from_documents(
documents[:80],
question_gen_query=question_gen_query,
llm=mistral_small,
)
question_gen_query = (
"You are a Teacher/ Professor. Your task is to setup "
"a quiz/examination. Using the provided context, formulate "
"a single question that captures an important fact from the "
"context. Restrict the question to the context information provided."
"You should generate only question and nothing else."
)
dataset_generator = DatasetGenerator.from_documents(
documents[:80],
question_gen_query=question_gen_query,
llm=mistral_small,
)
/Users/ravithejad/Desktop/llamaindex/lib/python3.9/site-packages/llama_index/core/evaluation/dataset_generation.py:215: DeprecationWarning: Call to deprecated class DatasetGenerator. (Deprecated in favor of `RagDatasetGenerator` which should be used instead.) return cls(
我们将生成40个训练问题和40个评估问题
In [ ]:
Copied!
# Note: This might take sometime.
questions = dataset_generator.generate_questions_from_nodes(num=40)
print("Generated ", len(questions), " questions")
# Note: This might take sometime.
questions = dataset_generator.generate_questions_from_nodes(num=40)
print("Generated ", len(questions), " questions")
Generated 40 questions
/Users/ravithejad/Desktop/llamaindex/lib/python3.9/site-packages/llama_index/core/evaluation/dataset_generation.py:312: DeprecationWarning: Call to deprecated class QueryResponseDataset. (Deprecated in favor of `LabelledRagDataset` which should be used instead.) return QueryResponseDataset(queries=queries, responses=responses_dict)
In [ ]:
Copied!
questions[10:15]
questions[10:15]
Out[ ]:
['What is the estimated relative human dependence on marine ecosystems for coastal protection, nutrition, fisheries economic benefits, and overall, as depicted in Figure 3.1?', 'What are the limitations of the overall index mentioned in the context, and how were values for reference regions computed?', 'What are the primary non-climate drivers that alter marine ecosystems and their services, as mentioned in the context?', 'What are the main challenges in detecting and attributing climate impacts on marine-dependent human systems, according to the provided context?', 'What new insights have been gained from experimental evidence about evolutionary adaptation, particularly in relation to eukaryotic organisms and their limited adaptation options to climate change, as mentioned in Section 3.3.4 of the IPCC AR6 WGII Chapter 03?']
In [ ]:
Copied!
with open("train_questions.txt", "w") as f:
for question in questions:
f.write(question + "\n")
with open("train_questions.txt", "w") as f:
for question in questions:
f.write(question + "\n")
现在，让我们基于一组完全不同的文档生成问题，以创建我们的评估数据集。
In [ ]:
Copied!
dataset_generator = DatasetGenerator.from_documents(
documents[80:],
question_gen_query=question_gen_query,
llm=mistral_small,
)
dataset_generator = DatasetGenerator.from_documents(
documents[80:],
question_gen_query=question_gen_query,
llm=mistral_small,
)
/Users/ravithejad/Desktop/llamaindex/lib/python3.9/site-packages/llama_index/core/evaluation/dataset_generation.py:215: DeprecationWarning: Call to deprecated class DatasetGenerator. (Deprecated in favor of `RagDatasetGenerator` which should be used instead.) return cls(
In [ ]:
Copied!
# Note: This might take sometime.
questions = dataset_generator.generate_questions_from_nodes(num=40)
print("Generated ", len(questions), " questions")
# Note: This might take sometime.
questions = dataset_generator.generate_questions_from_nodes(num=40)
print("Generated ", len(questions), " questions")
Generated 40 questions
/Users/ravithejad/Desktop/llamaindex/lib/python3.9/site-packages/llama_index/core/evaluation/dataset_generation.py:312: DeprecationWarning: Call to deprecated class QueryResponseDataset. (Deprecated in favor of `LabelledRagDataset` which should be used instead.) return QueryResponseDataset(queries=queries, responses=responses_dict)
In [ ]:
Copied!
with open("eval_questions.txt", "w") as f:
for question in questions:
f.write(question + "\n")
with open("eval_questions.txt", "w") as f:
for question in questions:
f.write(question + "\n")
使用 open-mistral-7b 查询引擎的初步评估¶
本次评估将采用 ragas 评估库。
Ragas 提供了丰富的 RAG 管道评估指标，具体说明可查阅此处。
本笔记本将使用以下两个核心指标：
answer_relevancy（答案相关性）- 衡量生成答案与提示问题的相关程度。若答案不完整或包含冗余信息，则得分较低。该指标通过计算 LLM 基于给定答案生成对应问题的概率来量化，取值范围 (0,1)，数值越高越好。faithfulness（事实一致性）- 评估生成答案与给定上下文的事实一致性。采用多步骤验证范式：首先生成答案中的陈述语句，随后逐条验证这些陈述与上下文的匹配度。结果标准化至 (0,1) 范围，数值越高越好。
In [ ]:
Copied!
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
In [ ]:
Copied!
from llama_index.core import VectorStoreIndex
from llama_index.embeddings.mistralai import MistralAIEmbedding
# limit the context window to 2048 tokens so that refine is used
from llama_index.core import Settings
Settings.context_window = 2048
Settings.llm = open_mistral
Settings.embed_model = MistralAIEmbedding()
index = VectorStoreIndex.from_documents(
documents,
)
query_engine = index.as_query_engine(similarity_top_k=2)
from llama_index.core import VectorStoreIndex
from llama_index.embeddings.mistralai import MistralAIEmbedding
# limit the context window to 2048 tokens so that refine is used
from llama_index.core import Settings
Settings.context_window = 2048
Settings.llm = open_mistral
Settings.embed_model = MistralAIEmbedding()
index = VectorStoreIndex.from_documents(
documents,
)
query_engine = index.as_query_engine(similarity_top_k=2)
In [ ]:
Copied!
contexts = []
answers = []
for question in questions:
response = query_engine.query(question)
contexts.append([x.node.get_content() for x in response.source_nodes])
answers.append(str(response))
contexts = []
answers = []
for question in questions:
response = query_engine.query(question)
contexts.append([x.node.get_content() for x in response.source_nodes])
answers.append(str(response))
In [ ]:
Copied!
# We will use OpenAI LLM for evaluation using RAGAS
os.environ["OPENAI_API_KEY"] = "<YOUR OPENAI API KEY>"
# We will use OpenAI LLM for evaluation using RAGAS
os.environ["OPENAI_API_KEY"] = ""
In [ ]:
Copied!
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness
ds = Dataset.from_dict(
{
"question": questions,
"answer": answers,
"contexts": contexts,
}
)
result = evaluate(ds, [answer_relevancy, faithfulness])
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness
ds = Dataset.from_dict(
{
"question": questions,
"answer": answers,
"contexts": contexts,
}
)
result = evaluate(ds, [answer_relevancy, faithfulness])
Evaluating: 100%|██████████| 80/80 [01:12<00:00, 1.10it/s]
让我们在微调前检查一下结果。
In [ ]:
Copied!
print(result)
print(result)
{'answer_relevancy': 0.8151, 'faithfulness': 0.8360}
使用 mistral-large-latest 收集训练数据¶
在此，我们使用 mistral-large-latest 来收集希望 open-mistral-7b 进行微调所需的数据。
In [ ]:
Copied!
from llama_index.llms.mistralai import MistralAI
from llama_index.finetuning.callbacks import MistralAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager
finetuning_handler = MistralAIFineTuningHandler()
callback_manager = CallbackManager([finetuning_handler])
llm = MistralAI(model="mistral-large-latest", temperature=0.1)
llm.callback_manager = callback_manager
from llama_index.llms.mistralai import MistralAI
from llama_index.finetuning.callbacks import MistralAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager
finetuning_handler = MistralAIFineTuningHandler()
callback_manager = CallbackManager([finetuning_handler])
llm = MistralAI(model="mistral-large-latest", temperature=0.1)
llm.callback_manager = callback_manager
In [ ]:
Copied!
questions = []
with open("train_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
questions = []
with open("train_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
In [ ]:
Copied!
from llama_index.core import VectorStoreIndex
Settings.embed_model = MistralAIEmbedding()
Settings.llm = llm
index = VectorStoreIndex.from_documents(
documents,
)
query_engine = index.as_query_engine(similarity_top_k=2)
from llama_index.core import VectorStoreIndex
Settings.embed_model = MistralAIEmbedding()
Settings.llm = llm
index = VectorStoreIndex.from_documents(
documents,
)
query_engine = index.as_query_engine(similarity_top_k=2)
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK"
In [ ]:
Copied!
from tqdm import tqdm
contexts = []
answers = []
for question in tqdm(questions, desc="Processing questions"):
response = query_engine.query(question)
contexts.append(
"\n".join([x.node.get_content() for x in response.source_nodes])
)
answers.append(str(response))
from tqdm import tqdm
contexts = []
answers = []
for question in tqdm(questions, desc="Processing questions"):
response = query_engine.query(question)
contexts.append(
"\n".join([x.node.get_content() for x in response.source_nodes])
)
answers.append(str(response))
Processing questions: 0%| | 0/40 [00:00<?, ?it/s]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 2%|▎ | 1/40 [00:02<01:43, 2.66s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 5%|▌ | 2/40 [00:13<04:38, 7.34s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 8%|▊ | 3/40 [00:19<04:08, 6.71s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 10%|█ | 4/40 [00:24<03:35, 5.98s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 12%|█▎ | 5/40 [00:28<03:04, 5.26s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 15%|█▌ | 6/40 [00:33<02:55, 5.17s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 18%|█▊ | 7/40 [00:42<03:35, 6.52s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 20%|██ | 8/40 [00:45<02:50, 5.32s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 22%|██▎ | 9/40 [00:51<02:58, 5.75s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 25%|██▌ | 10/40 [00:55<02:29, 4.99s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 28%|██▊ | 11/40 [01:00<02:24, 5.00s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 30%|███ | 12/40 [01:03<02:08, 4.59s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 32%|███▎ | 13/40 [01:06<01:45, 3.90s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 35%|███▌ | 14/40 [01:15<02:21, 5.44s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 38%|███▊ | 15/40 [01:18<02:00, 4.82s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 40%|████ | 16/40 [01:29<02:42, 6.76s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 42%|████▎ | 17/40 [01:37<02:45, 7.20s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 45%|████▌ | 18/40 [01:42<02:18, 6.31s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 48%|████▊ | 19/40 [01:47<02:03, 5.89s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 50%|█████ | 20/40 [01:48<01:33, 4.66s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 52%|█████▎ | 21/40 [01:53<01:25, 4.50s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 55%|█████▌ | 22/40 [01:55<01:11, 3.98s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 57%|█████▊ | 23/40 [01:58<00:59, 3.47s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 60%|██████ | 24/40 [02:00<00:48, 3.03s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 62%|██████▎ | 25/40 [02:04<00:50, 3.38s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 65%|██████▌ | 26/40 [02:06<00:42, 3.03s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 68%|██████▊ | 27/40 [02:22<01:29, 6.89s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 70%|███████ | 28/40 [02:25<01:08, 5.73s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 72%|███████▎ | 29/40 [02:32<01:08, 6.24s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 75%|███████▌ | 30/40 [02:34<00:49, 4.99s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 78%|███████▊ | 31/40 [02:37<00:37, 4.19s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 80%|████████ | 32/40 [02:42<00:36, 4.54s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 82%|████████▎ | 33/40 [02:46<00:30, 4.39s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 85%|████████▌ | 34/40 [02:48<00:22, 3.67s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 88%|████████▊ | 35/40 [02:53<00:20, 4.01s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 90%|█████████ | 36/40 [02:58<00:17, 4.28s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 92%|█████████▎| 37/40 [03:13<00:22, 7.45s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 95%|█████████▌| 38/40 [03:16<00:12, 6.20s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 98%|█████████▊| 39/40 [03:18<00:05, 5.06s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing questions: 100%|██████████| 40/40 [03:30<00:00, 5.25s/it]
In [ ]:
Copied!
import json
from typing import List
def convert_data_jsonl_format(
questions: List[str],
contexts: List[str],
answers: List[str],
output_file: str,
) -> None:
with open(output_file, "w") as outfile:
for context, question, answer in zip(contexts, questions, answers):
message_dict = {
"messages": [
{
"role": "assistant",
"content": "You are a helpful assistant to answer user queries based on provided context.",
},
{
"role": "user",
"content": f"context: {context} \n\n question: {question}",
},
{"role": "assistant", "content": answer},
]
}
# Write the JSON object in a single line
json.dump(message_dict, outfile)
# Add a newline character after each JSON object
outfile.write("\n")
import json
from typing import List
def convert_data_jsonl_format(
questions: List[str],
contexts: List[str],
answers: List[str],
output_file: str,
) -> None:
with open(output_file, "w") as outfile:
for context, question, answer in zip(contexts, questions, answers):
message_dict = {
"messages": [
{
"role": "assistant",
"content": "You are a helpful assistant to answer user queries based on provided context.",
},
{
"role": "user",
"content": f"context: {context} \n\n question: {question}",
},
{"role": "assistant", "content": answer},
]
}
# Write the JSON object in a single line
json.dump(message_dict, outfile)
# Add a newline character after each JSON object
outfile.write("\n")
In [ ]:
Copied!
convert_data_jsonl_format(questions, contexts, answers, "training.jsonl")
convert_data_jsonl_format(questions, contexts, answers, "training.jsonl")
创建 MistralAIFinetuneEngine¶
我们创建一个 MistralAIFinetuneEngine：该微调引擎将负责启动微调任务，并返回一个可直接接入 LlamaIndex 工作流的 LLM 模型。
我们使用默认构造函数，但也可以通过 from_finetuning_handler 类方法直接将我们的微调处理器传入该引擎。
In [ ]:
Copied!
from llama_index.llms.mistralai import MistralAI
from llama_index.finetuning.callbacks import MistralAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager
from llama_index.finetuning.mistralai import MistralAIFinetuneEngine
# Wandb for monitorning the training logs
wandb_integration_dict = {
"project": "mistralai",
"run_name": "finetuning",
"api_key": "<api_key>",
}
finetuning_engine = MistralAIFinetuneEngine(
base_model="open-mistral-7b",
training_path="training.jsonl",
# validation_path="<validation file>", # validation file is optional
verbose=True,
training_steps=5,
learning_rate=0.0001,
wandb_integration_dict=wandb_integration_dict,
)
from llama_index.llms.mistralai import MistralAI
from llama_index.finetuning.callbacks import MistralAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager
from llama_index.finetuning.mistralai import MistralAIFinetuneEngine
# Wandb for monitorning the training logs
wandb_integration_dict = {
"project": "mistralai",
"run_name": "finetuning",
"api_key": "",
}
finetuning_engine = MistralAIFinetuneEngine(
base_model="open-mistral-7b",
training_path="training.jsonl",
# validation_path="", # validation file is optional
verbose=True,
training_steps=5,
learning_rate=0.0001,
wandb_integration_dict=wandb_integration_dict,
)
In [ ]:
Copied!
# starts the finetuning of open-mistral-7b
finetuning_engine.finetune()
# starts the finetuning of open-mistral-7b
finetuning_engine.finetune()
In [ ]:
Copied!
# This will show the current status of the job - 'RUNNING'
finetuning_engine.get_current_job()
# This will show the current status of the job - 'RUNNING'
finetuning_engine.get_current_job()
INFO:httpx:HTTP Request: GET https://api.mistral.ai/v1/fine_tuning/jobs/19f5943a-3000-4568-b227-e45c36ac15f1 "HTTP/1.1 200 OK" HTTP Request: GET https://api.mistral.ai/v1/fine_tuning/jobs/19f5943a-3000-4568-b227-e45c36ac15f1 "HTTP/1.1 200 OK"
Out[ ]:
DetailedJob(id='19f5943a-3000-4568-b227-e45c36ac15f1', hyperparameters=TrainingParameters(training_steps=5, learning_rate=0.0001), fine_tuned_model=None, model='open-mistral-7b', status='RUNNING', job_type='FT', created_at=1718613228, modified_at=1718613229, training_files=['07270085-65e6-441e-b99d-ef6f75dd5a30'], validation_files=[], object='job', integrations=[WandbIntegration(type='wandb', project='mistralai', name=None, run_name='finetuning')], events=[Event(name='status-updated', data={'status': 'RUNNING'}, created_at=1718613229), Event(name='status-updated', data={'status': 'QUEUED'}, created_at=1718613228)], checkpoints=[], estimated_start_time=None)
In [ ]:
Copied!
# This will show the current status of the job - 'SUCCESS'
finetuning_engine.get_current_job()
# This will show the current status of the job - 'SUCCESS'
finetuning_engine.get_current_job()
INFO:httpx:HTTP Request: GET https://api.mistral.ai/v1/fine_tuning/jobs/19f5943a-3000-4568-b227-e45c36ac15f1 "HTTP/1.1 200 OK" HTTP Request: GET https://api.mistral.ai/v1/fine_tuning/jobs/19f5943a-3000-4568-b227-e45c36ac15f1 "HTTP/1.1 200 OK"
Out[ ]:
DetailedJob(id='19f5943a-3000-4568-b227-e45c36ac15f1', hyperparameters=TrainingParameters(training_steps=5, learning_rate=0.0001), fine_tuned_model='ft:open-mistral-7b:35c6fa92:20240617:19f5943a', model='open-mistral-7b', status='SUCCESS', job_type='FT', created_at=1718613228, modified_at=1718613306, training_files=['07270085-65e6-441e-b99d-ef6f75dd5a30'], validation_files=[], object='job', integrations=[WandbIntegration(type='wandb', project='mistralai', name=None, run_name='finetuning')], events=[Event(name='status-updated', data={'status': 'SUCCESS'}, created_at=1718613306), Event(name='status-updated', data={'status': 'RUNNING'}, created_at=1718613229), Event(name='status-updated', data={'status': 'QUEUED'}, created_at=1718613228)], checkpoints=[], estimated_start_time=None)
In [ ]:
Copied!
ft_llm = finetuning_engine.get_finetuned_model(temperature=0.1)
ft_llm = finetuning_engine.get_finetuned_model(temperature=0.1)
INFO:httpx:HTTP Request: GET https://api.mistral.ai/v1/fine_tuning/jobs/19f5943a-3000-4568-b227-e45c36ac15f1 "HTTP/1.1 200 OK" HTTP Request: GET https://api.mistral.ai/v1/fine_tuning/jobs/19f5943a-3000-4568-b227-e45c36ac15f1 "HTTP/1.1 200 OK" INFO:llama_index.finetuning.mistralai.base:status of the job_id: 19f5943a-3000-4568-b227-e45c36ac15f1 is SUCCESS status of the job_id: 19f5943a-3000-4568-b227-e45c36ac15f1 is SUCCESS
评估¶
当微调后的模型创建完成后，下一步就是在我们的评估数据集上再次运行微调模型，以衡量性能提升情况。
In [ ]:
Copied!
from llama_index.core import Settings
from llama_index.llms.mistralai import MistralAI
from llama_index.embeddings.mistralai import MistralAIEmbedding
# Setting up finetuned llm
Settings.llm = ft_llm
Settings.context_window = (
2048 # limit the context window artifically to test refine process
)
Settings.embed_model = MistralAIEmbedding()
from llama_index.core import Settings
from llama_index.llms.mistralai import MistralAI
from llama_index.embeddings.mistralai import MistralAIEmbedding
# Setting up finetuned llm
Settings.llm = ft_llm
Settings.context_window = (
2048 # limit the context window artifically to test refine process
)
Settings.embed_model = MistralAIEmbedding()
In [ ]:
Copied!
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
In [ ]:
Copied!
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=2, llm=ft_llm)
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=2, llm=ft_llm)
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK"
In [ ]:
Copied!
contexts = []
answers = []
for question in tqdm(questions, desc="Processing Questions"):
response = query_engine.query(question)
contexts.append([x.node.get_content() for x in response.source_nodes])
answers.append(str(response))
contexts = []
answers = []
for question in tqdm(questions, desc="Processing Questions"):
response = query_engine.query(question)
contexts.append([x.node.get_content() for x in response.source_nodes])
answers.append(str(response))
Processing Questions: 0%| | 0/40 [00:00<?, ?it/s]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 2%|▎ | 1/40 [00:06<04:16, 6.58s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 5%|▌ | 2/40 [00:11<03:32, 5.58s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 8%|▊ | 3/40 [00:14<02:45, 4.48s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 10%|█ | 4/40 [00:19<02:41, 4.48s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 12%|█▎ | 5/40 [00:21<02:16, 3.90s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 15%|█▌ | 6/40 [00:27<02:25, 4.29s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 18%|█▊ | 7/40 [00:29<02:00, 3.66s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 20%|██ | 8/40 [00:31<01:41, 3.16s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 22%|██▎ | 9/40 [00:33<01:25, 2.74s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 25%|██▌ | 10/40 [00:36<01:26, 2.90s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 28%|██▊ | 11/40 [00:38<01:18, 2.71s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 30%|███ | 12/40 [00:40<01:08, 2.44s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 32%|███▎ | 13/40 [00:47<01:42, 3.81s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 35%|███▌ | 14/40 [00:50<01:30, 3.46s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 38%|███▊ | 15/40 [01:01<02:22, 5.71s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 40%|████ | 16/40 [01:06<02:14, 5.59s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 42%|████▎ | 17/40 [01:09<01:49, 4.76s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 45%|████▌ | 18/40 [01:13<01:39, 4.54s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 48%|████▊ | 19/40 [01:16<01:27, 4.15s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 50%|█████ | 20/40 [01:20<01:19, 3.99s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 52%|█████▎ | 21/40 [01:27<01:34, 4.97s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 55%|█████▌ | 22/40 [01:30<01:17, 4.33s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 57%|█████▊ | 23/40 [01:32<01:02, 3.70s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 60%|██████ | 24/40 [01:40<01:18, 4.90s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 62%|██████▎ | 25/40 [01:44<01:12, 4.80s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 65%|██████▌ | 26/40 [01:48<01:02, 4.45s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 68%|██████▊ | 27/40 [01:54<01:02, 4.80s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 70%|███████ | 28/40 [02:03<01:15, 6.28s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 72%|███████▎ | 29/40 [02:06<00:57, 5.21s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 75%|███████▌ | 30/40 [02:09<00:44, 4.47s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 78%|███████▊ | 31/40 [02:11<00:33, 3.77s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 80%|████████ | 32/40 [02:13<00:26, 3.30s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 82%|████████▎ | 33/40 [02:15<00:19, 2.77s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 85%|████████▌ | 34/40 [02:21<00:22, 3.73s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 88%|████████▊ | 35/40 [02:24<00:18, 3.70s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 90%|█████████ | 36/40 [02:29<00:16, 4.02s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 92%|█████████▎| 37/40 [02:36<00:14, 4.83s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 95%|█████████▌| 38/40 [02:38<00:08, 4.12s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 98%|█████████▊| 39/40 [02:40<00:03, 3.42s/it]
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Processing Questions: 100%|██████████| 40/40 [02:44<00:00, 4.11s/it]
In [ ]:
Copied!
ds = Dataset.from_dict(
{
"question": questions,
"answer": answers,
"contexts": contexts,
}
)
result = evaluate(ds, [answer_relevancy, faithfulness])
ds = Dataset.from_dict(
{
"question": questions,
"answer": answers,
"contexts": contexts,
}
)
result = evaluate(ds, [answer_relevancy, faithfulness])
Evaluating: 0%| | 0/80 [00:00<?, ?it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 1%|▏ | 1/80 [00:05<06:55, 5.26s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 2%|▎ | 2/80 [00:12<08:11, 6.30s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 11%|█▏ | 9/80 [00:13<01:15, 1.06s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 12%|█▎ | 10/80 [00:15<01:23, 1.20s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 14%|█▍ | 11/80 [00:17<01:33, 1.35s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 18%|█▊ | 14/80 [00:18<00:58, 1.12it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 19%|█▉ | 15/80 [00:19<01:03, 1.03it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 20%|██ | 16/80 [00:20<01:01, 1.03it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 26%|██▋ | 21/80 [00:22<00:32, 1.80it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 28%|██▊ | 22/80 [00:25<00:52, 1.10it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 41%|████▏ | 33/80 [00:27<00:19, 2.43it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 42%|████▎ | 34/80 [00:32<00:36, 1.25it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 49%|████▉ | 39/80 [00:33<00:23, 1.73it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 55%|█████▌ | 44/80 [00:34<00:15, 2.31it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 56%|█████▋ | 45/80 [00:35<00:17, 1.98it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 57%|█████▊ | 46/80 [00:39<00:30, 1.12it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 62%|██████▎ | 50/80 [00:40<00:19, 1.54it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 68%|██████▊ | 54/80 [00:44<00:20, 1.26it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 71%|███████▏ | 57/80 [00:45<00:15, 1.51it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 76%|███████▋ | 61/80 [00:47<00:11, 1.66it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 78%|███████▊ | 62/80 [00:49<00:13, 1.31it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 82%|████████▎ | 66/80 [00:50<00:07, 1.76it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
Evaluating: 88%|████████▊ | 70/80 [00:52<00:05, 1.86it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 89%|████████▉ | 71/80 [00:52<00:04, 1.87it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 90%|█████████ | 72/80 [00:55<00:06, 1.32it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 91%|█████████▏| 73/80 [00:57<00:07, 1.05s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 92%|█████████▎| 74/80 [00:58<00:05, 1.06it/s]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 94%|█████████▍| 75/80 [01:01<00:07, 1.41s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 95%|█████████▌| 76/80 [01:03<00:06, 1.61s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 96%|█████████▋| 77/80 [01:07<00:06, 2.27s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 98%|█████████▊| 78/80 [01:16<00:07, 3.94s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 99%|█████████▉| 79/80 [01:17<00:03, 3.07s/it]
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Evaluating: 100%|██████████| 80/80 [01:19<00:00, 1.01it/s]
让我们用微调后的模型检查结果
In [ ]:
Copied!
print(result)
print(result)
{'answer_relevancy': 0.8016, 'faithfulness': 0.8924}
观察结果：¶
open-mistral-7b模型：'answer_relevancy'（答案相关性）得分：0.8151，'faithfulness'（忠实度）得分：0.8360
open-mistral-7b-finetuned微调模型：'answer_relevancy'（答案相关性）得分：0.8016，'faithfulness'（忠实度）得分：0.8924
可见微调后模型的忠实度评分有所提升，而答案相关性评分则出现小幅下降。
探索差异¶
让我们快速比较响应中的差异，以证明微调确实产生了某些改变。
In [ ]:
Copied!
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK"
In [ ]:
Copied!
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
questions = []
with open("eval_questions.txt", "r") as f:
for line in f:
questions.append(line.strip())
In [ ]:
Copied!
print(questions[20])
print(questions[20])
What are some examples of economic losses due to disruptions in ocean and coastal ecosystem services partly attributable to climate change, as mentioned in the context?
原始 open-mistral-7b¶
In [ ]:
Copied!
from llama_index.core.response.notebook_utils import display_response
from llama_index.core.response.notebook_utils import display_response
In [ ]:
Copied!
query_engine = index.as_query_engine(llm=open_mistral)
response = query_engine.query(questions[20])
display_response(response)
query_engine = index.as_query_engine(llm=open_mistral)
response = query_engine.query(questions[20])
display_response(response)
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Final Response: Economic losses due to disruptions in ocean and coastal ecosystem services partly attributable to climate change, as mentioned in the context, include:
- The estimated loss of 800 million USD in the farmed-salmon industry in Chile due to a climate-driven harmful algal bloom.
- The closure of the Dungeness crab and razor clam fishery in the USA due to a climate-driven algal bloom, which harmed 84% of surveyed residents from 16 California coastal communities.
- Substantial economic losses for fisheries resulting from recent climate-driven harmful algal blooms and marine pathogen outbreaks in Asia, North America, and South America.
- Declines in tourism and real estate values, associated with climate-driven harmful algal blooms, in the USA, France, and England.
微调版 open-mistral-7b¶
In [ ]:
Copied!
query_engine = index.as_query_engine(llm=ft_llm)
response = query_engine.query(questions[20])
display_response(response)
query_engine = index.as_query_engine(llm=ft_llm)
response = query_engine.query(questions[20])
display_response(response)
INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/embeddings "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK" HTTP Request: POST https://api.mistral.ai/v1/chat/completions "HTTP/1.1 200 OK"
Final Response: Economic losses due to disruptions in ocean and coastal ecosystem services partly attributable to climate change have been recorded in various industries and regions. For instance, substantial economic losses for fisheries resulting from climate-driven harmful algal blooms and marine pathogen outbreaks have been reported in Asia, North America, and South America. A notable example is the 2016 event in Chile, which caused an estimated loss of 800 million USD in the farmed-salmon industry and led to regional government protests. Similarly, the recent closure of the Dungeness crab and razor clam fishery in the USA due to a climate-driven algal bloom harmed 84% of surveyed residents from 16 California coastal communities.
In addition, declines in tourism and real estate values, associated with climate-driven harmful algal blooms, have been recorded in the USA, France, and England. These disruptions can have significant economic impacts on coastal communities, particularly those that heavily rely on tourism for income or have a strong cultural identity linked to the ocean.
Moreover, changes in the abundance or quality of goods from the ocean, such as non-food products like dietary supplements, food preservatives, pharmaceuticals, biofuels, sponges, cosmetic products, jewellery coral, cultured pearls, and aquarium species, will also be affected by climate change. These changes can lead to economic losses for industries that rely on these resources.
Finally, the vulnerability of communities to losses in marine ecosystem services varies within and among communities. However, climate-change impacts exacerbate existing inequalities already experienced by some communities, including Indigenous Peoples, Pacific Island countries and territories, and marginalized peoples, such as migrants and women in fisheries and mariculture. These inequities increase the risk to their fundamental human rights by disrupting livelihoods and food security, while leading to loss of social, economic, and cultural rights.
如我们所见，经过微调的模型提供了更全面的回答！这与ragas评估中更高的忠实度评分相符，因为该答案更能代表检索到的上下文内容。
结论¶
综上所述，仅使用约40个问题进行微调确实有助于提升我们的评估分数！
答案相关性（answer_relevancy）: 0.8151 -> 0.8016
答案相关性略有下降，但幅度非常微小。
忠实度（faithfulness）: 0.8360 -> 0.8924
忠实度似乎得到了提升！这意味着生成的答案能更好地满足原始提问的需求。