使用 LabelledRagDatatset 对 RAG 流水线进行基准测试¶

LabelledRagDataset 旨在用于评估任意给定的 RAG 流水线，这些流水线可能存在多种配置（例如选择 LLM、设置 similarity_top_k 值、调整 chunk_size 等参数）。我们将这个抽象概念类比于传统机器学习数据集，其中 X 特征用于预测真实标签 y。在本场景中，我们将 query 和检索到的 contexts 作为"特征"，而将查询的答案（称为 reference_answer）作为真实标签。

当然，这类数据集由多个观测样本或示例组成。对于 LabelledRagDataset 而言，这些样本由一组 LabelledRagDataExample 构成。

本笔记本将演示如何从零开始构建 LabelledRagDataset。请注意，另一种方法是直接从 llama-hub 下载社区提供的 LabelledRagDataset，以便在其上评估/基准测试您自己的 RAG 流水线。

LabelledRagDataExample 类¶

In [ ]:

%pip install llama-index-llms-openai
%pip install llama-index-readers-wikipedia

%pip install llama-index-llms-openai %pip install llama-index-readers-wikipedia

In [ ]:

from llama_index.core.llama_dataset import (
    LabelledRagDataExample,
    CreatedByType,
    CreatedBy,
)

# constructing a LabelledRagDataExample
query = "This is a test query, is it not?"
query_by = CreatedBy(type=CreatedByType.AI, model_name="gpt-4")
reference_answer = "Yes it is."
reference_answer_by = CreatedBy(type=CreatedByType.HUMAN)
reference_contexts = ["This is a sample context"]

rag_example = LabelledRagDataExample(
    query=query,
    query_by=query_by,
    reference_contexts=reference_contexts,
    reference_answer=reference_answer,
    reference_answer_by=reference_answer_by,
)

from llama_index.core.llama_dataset import ( LabelledRagDataExample, CreatedByType, CreatedBy, ) # constructing a LabelledRagDataExample query = "This is a test query, is it not?" query_by = CreatedBy(type=CreatedByType.AI, model_name="gpt-4") reference_answer = "Yes it is." reference_answer_by = CreatedBy(type=CreatedByType.HUMAN) reference_contexts = ["This is a sample context"] rag_example = LabelledRagDataExample( query=query, query_by=query_by, reference_contexts=reference_contexts, reference_answer=reference_answer, reference_answer_by=reference_answer_by, )

LabelledRagDataExample 是一个 Pydantic Model，因此可以在 json 或 dict 格式之间相互转换。

In [ ]:

print(rag_example.json())

print(rag_example.json())

{"query": "This is a test query, is it not?", "query_by": {"model_name": "gpt-4", "type": "ai"}, "reference_contexts": ["This is a sample context"], "reference_answer": "Yes it is.", "reference_answer_by": {"model_name": "", "type": "human"}}

In [ ]:

LabelledRagDataExample.parse_raw(rag_example.json())

LabelledRagDataExample.parse_raw(rag_example.json())

Out[ ]:

LabelledRagDataExample(query='This is a test query, is it not?', query_by=CreatedBy(model_name='gpt-4', type=<CreatedByType.AI: 'ai'>), reference_contexts=['This is a sample context'], reference_answer='Yes it is.', reference_answer_by=CreatedBy(model_name='', type=<CreatedByType.HUMAN: 'human'>))

In [ ]:

rag_example.dict()

rag_example.dict()

Out[ ]:

{'query': 'This is a test query, is it not?',
 'query_by': {'model_name': 'gpt-4', 'type': <CreatedByType.AI: 'ai'>},
 'reference_contexts': ['This is a sample context'],
 'reference_answer': 'Yes it is.',
 'reference_answer_by': {'model_name': '',
  'type': <CreatedByType.HUMAN: 'human'>}}

In [ ]:

LabelledRagDataExample.parse_obj(rag_example.dict())

LabelledRagDataExample.parse_obj(rag_example.dict())

Out[ ]:

LabelledRagDataExample(query='This is a test query, is it not?', query_by=CreatedBy(model_name='gpt-4', type=<CreatedByType.AI: 'ai'>), reference_contexts=['This is a sample context'], reference_answer='Yes it is.', reference_answer_by=CreatedBy(model_name='', type=<CreatedByType.HUMAN: 'human'>))

让我们创建第二个示例，这样就能得到一个（稍微）更有趣的 LabelledRagDataset。

In [ ]:

query = "This is a test query, is it so?"
reference_answer = "I think yes, it is."
reference_contexts = ["This is a second sample context"]

rag_example_2 = LabelledRagDataExample(
    query=query,
    query_by=query_by,
    reference_contexts=reference_contexts,
    reference_answer=reference_answer,
    reference_answer_by=reference_answer_by,
)

query = "This is a test query, is it so?" reference_answer = "I think yes, it is." reference_contexts = ["This is a second sample context"] rag_example_2 = LabelledRagDataExample( query=query, query_by=query_by, reference_contexts=reference_contexts, reference_answer=reference_answer, reference_answer_by=reference_answer_by, )

LabelledRagDataset 类¶

In [ ]:

from llama_index.core.llama_dataset import LabelledRagDataset

rag_dataset = LabelledRagDataset(examples=[rag_example, rag_example_2])

from llama_index.core.llama_dataset import LabelledRagDataset rag_dataset = LabelledRagDataset(examples=[rag_example, rag_example_2])

可以使用一个便捷方法将数据集以 pandas.DataFrame 形式查看。

In [ ]:

rag_dataset.to_pandas()

rag_dataset.to_pandas()

Out[ ]:

	query	reference_contexts	reference_answer	reference_answer_by	query_by
0	This is a test query, is it not?	[This is a sample context]	Yes it is.	human	ai (gpt-4)
1	This is a test query, is it so?	[This is a second sample context]	I think yes, it is.	human	ai (gpt-4)

序列化¶

要将数据集持久化保存到磁盘或从磁盘加载，可使用 save_json 和 from_json 方法。

In [ ]:

rag_dataset.save_json("rag_dataset.json")

rag_dataset.save_json("rag_dataset.json")

In [ ]:

reload_rag_dataset = LabelledRagDataset.from_json("rag_dataset.json")

reload_rag_dataset = LabelledRagDataset.from_json("rag_dataset.json")

In [ ]:

reload_rag_dataset.to_pandas()

reload_rag_dataset.to_pandas()

Out[ ]:

	query	reference_contexts	reference_answer	reference_answer_by	query_by
0	This is a test query, is it not?	[This is a sample context]	Yes it is.	human	ai (gpt-4)
1	This is a test query, is it so?	[This is a second sample context]	I think yes, it is.	human	ai (gpt-4)

基于维基百科构建合成型 LabelledRagDataset¶

在本节中，我们将首先使用合成生成器创建一个 LabelledRagDataset。最终，我们将使用 GPT-4 为合成型 LabelledRagDataExample 生成 query（查询）和 reference_answer（参考答案）。

注意：如果已拥有针对某文本语料库的查询语句、参考答案及上下文内容，则无需通过数据合成即可进行预测及后续的预测效果评估。

In [ ]:

import nest_asyncio

nest_asyncio.apply()

import nest_asyncio nest_asyncio.apply()

In [ ]:

!pip install wikipedia -q

!pip install wikipedia -q

In [ ]:

# wikipedia pages
from llama_index.readers.wikipedia import WikipediaReader
from llama_index.core import VectorStoreIndex

cities = [
    "San Francisco",
]

documents = WikipediaReader().load_data(
    pages=[f"History of {x}" for x in cities]
)
index = VectorStoreIndex.from_documents(documents)

# wikipedia pages from llama_index.readers.wikipedia import WikipediaReader from llama_index.core import VectorStoreIndex cities = [ "San Francisco", ] documents = WikipediaReader().load_data( pages=[f"History of {x}" for x in cities] ) index = VectorStoreIndex.from_documents(documents)

RagDatasetGenerator 可在文档集合上构建，用于生成 LabelledRagDataExample 实例。

In [ ]:

# generate questions against chunks
from llama_index.core.llama_dataset.generator import RagDatasetGenerator
from llama_index.llms.openai import OpenAI

# set context for llm provider
llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

# instantiate a DatasetGenerator
dataset_generator = RagDatasetGenerator.from_documents(
    documents,
    llm=llm,
    num_questions_per_chunk=2,  # set the number of questions per nodes
    show_progress=True,
)

# generate questions against chunks from llama_index.core.llama_dataset.generator import RagDatasetGenerator from llama_index.llms.openai import OpenAI # set context for llm provider llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3) # instantiate a DatasetGenerator dataset_generator = RagDatasetGenerator.from_documents( documents, llm=llm, num_questions_per_chunk=2, # set the number of questions per nodes show_progress=True, )

Parsing nodes:   0%|          | 0/1 [00:00<?, ?it/s]

In [ ]:

len(dataset_generator.nodes)

len(dataset_generator.nodes)

Out[ ]:

13

In [ ]:

# since there are 13 nodes, there should be a total of 26 questions
rag_dataset = dataset_generator.generate_dataset_from_nodes()

# since there are 13 nodes, there should be a total of 26 questions rag_dataset = dataset_generator.generate_dataset_from_nodes()

100%|███████████████████████████████████████████████████████| 13/13 [00:02<00:00,  5.04it/s]
100%|█████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.14s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.95s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:13<00:00,  6.55s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:07<00:00,  3.89s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.66s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.85s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.03s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.07s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.48s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.34s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.50s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.35s/it]
100%|█████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.34s/it]

In [ ]:

rag_dataset.to_pandas()

rag_dataset.to_pandas()

Out[ ]:

	query	reference_contexts	reference_answer	reference_answer_by	query_by
0	How did the gold rush of 1849 impact the devel...	[The history of the city of San Francisco, Cal...	The gold rush of 1849 had a significant impact...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
1	What were the early European settlements estab...	[The history of the city of San Francisco, Cal...	The early European settlements established in ...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
2	How did the arrival of Europeans impact the se...	[== Arrival of Europeans and early settlement ...	The arrival of Europeans had a significant imp...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
3	What were some of the challenges faced by the ...	[== Arrival of Europeans and early settlement ...	The early settlers of San Francisco faced seve...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
4	How did the California gold rush impact the po...	[== 1848 gold rush ==\nThe California gold rus...	The California gold rush in the mid-19th centu...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
5	Discuss the role of Chinese immigrants in the ...	[== 1848 gold rush ==\nThe California gold rus...	Chinese immigrants played a significant role i...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
6	How did San Francisco transform into a major c...	[== Paris of the West ==\n\nIt was during the ...	San Francisco transformed into a major city du...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
7	What were some significant developments and ch...	[== Paris of the West ==\n\nIt was during the ...	During the late 19th and early 20th centuries,...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
8	How did Abe Ruef contribute to Eugene Schmitz'...	[== Corruption and graft trials ==\n\nMayor Eu...	Abe Ruef contributed $16,000 to Eugene Schmitz...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
9	Describe the impact of the 1906 earthquake and...	[== Corruption and graft trials ==\n\nMayor Eu...	The 1906 earthquake and fire had a devastating...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
10	How did the 1906 San Francisco earthquake impa...	[=== Reconstruction ===\nAlmost immediately af...	The 1906 San Francisco earthquake had a signif...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
11	What major events and developments took place ...	[=== Reconstruction ===\nAlmost immediately af...	During the 1930s and World War II, several maj...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
12	How did the post-World War II era contribute t...	[== Post-World War II ==\nAfter World War II, ...	After World War II, many American military per...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
13	Discuss the impact of urban renewal initiative...	[== Post-World War II ==\nAfter World War II, ...	M. Justin Herman led urban renewal initiatives...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
14	How did San Francisco become a center of count...	[== 1960 – 1970s ==\n\n\n=== "Summer of Love" ...	San Francisco became a center of countercultur...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
15	Explain the role of San Francisco as a "Gay Me...	[== 1960 – 1970s ==\n\n\n=== "Summer of Love" ...	During the 1960s and beyond, San Francisco bec...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
16	How did the construction of BART and Muni impa...	[=== New public infrastructure ===\nThe 1970s ...	The construction of BART and Muni in the 1970s...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
17	What were the major challenges faced by San Fr...	[=== New public infrastructure ===\nThe 1970s ...	In the 1980s, San Francisco faced several majo...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
18	How did the 1989 Loma Prieta earthquake impact...	[=== 1989 Loma Prieta earthquake ===\n\nOn Oct...	The 1989 Loma Prieta earthquake had significan...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
19	Discuss the effects of the dot-com boom in the...	[=== 1989 Loma Prieta earthquake ===\n\nOn Oct...	The dot-com boom in the late 1990s had signifi...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
20	How did the redevelopment of the Mission Bay n...	[== 2010s ==\nThe early 2000s and into the 201...	The redevelopment of the Mission Bay neighborh...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
21	What significant events occurred in San Franci...	[== 2010s ==\nThe early 2000s and into the 201...	In 2010, the San Francisco Giants won their fi...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
22	In the context of San Francisco's history, dis...	[=== Cultural themes ===\nBerglund, Barbara (2...	The 1906 earthquake had a significant impact o...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
23	How did different ethnic and religious communi...	[=== Cultural themes ===\nBerglund, Barbara (2...	Two specific communities mentioned in the sour...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
24	In the context of San Francisco's history, wha...	[=== Gold rush & early days ===\nHittell, John...	Some significant events and developments durin...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)
25	How did politics shape the growth and transfor...	[=== Gold rush & early days ===\nHittell, John...	The provided sources offer a comprehensive und...	ai (gpt-3.5-turbo)	ai (gpt-3.5-turbo)

In [ ]:

rag_dataset.save_json("rag_dataset.json")

rag_dataset.save_json("rag_dataset.json")