微调 GPT-3.5-Turbo¶

本笔记本将演示如何对 gpt-3.5-turbo 进行微调。

具体而言，我们尝试通过使用 GPT-4 生成训练数据来蒸馏 GPT-4 的知识，进而微调 GPT-3.5。

所有训练数据均来自索引数据的两个不同部分，分别生成训练集和评估集。

随后我们使用 OpenAIFinetuneEngine 封装抽象层进行微调。

评估环节采用 ragas 库完成，具体细节将在后文详述。

In [ ]:

%pip install llama-index-finetuning
%pip install llama-index-finetuning-callbacks
%pip install llama-index-llms-openai

%pip install llama-index-finetuning %pip install llama-index-finetuning-callbacks %pip install llama-index-llms-openai

In [ ]:

# !pip install llama-index pypdf sentence-transformers ragas

# !pip install llama-index pypdf sentence-transformers ragas

In [ ]:

import os
import openai

import os import openai

In [ ]:

os.environ["OPENAI_API_KEY"] = "sk-..."
openai.api_key = os.environ["OPENAI_API_KEY"]

os.environ["OPENAI_API_KEY"] = "sk-..." openai.api_key = os.environ["OPENAI_API_KEY"]

数据准备¶

在此步骤中，我们首先下载将用于生成训练数据的 PDF 文件。

In [ ]:

!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf

!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.7M  100 20.7M    0     0   397k      0  0:00:53  0:00:53 --:--:--  417k84k      0  0:00:55  0:00:24  0:00:31  406k    0   395k      0  0:00:53  0:00:48  0:00:05  403k0   396k      0  0:00:53  0:00:53 --:--:--  406k

下一步是生成训练和评估数据集。

我们将针对下载的 PDF 文档不同章节生成 40 个问题。

可以使用 GPT-3.5 处理评估问题来获取基线性能指标。

然后，我们将使用 GPT-4 处理训练问题来生成训练数据。训练数据将通过 OpenAIFineTuningHandler 进行收集。

若您不愿花费时间/Token，此步骤完全可选——评估和训练问题及训练数据均已提供在本文件夹中！

训练数据生成¶

In [ ]:

from llama_index.core import SimpleDirectoryReader
from llama_index.llms.openai import OpenAI
from llama_index.core.evaluation import DatasetGenerator

documents = SimpleDirectoryReader(
    input_files=["IPCC_AR6_WGII_Chapter03.pdf"]
).load_data()

# Shuffle the documents
import random

random.seed(42)
random.shuffle(documents)

gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

from llama_index.core import SimpleDirectoryReader from llama_index.llms.openai import OpenAI from llama_index.core.evaluation import DatasetGenerator documents = SimpleDirectoryReader( input_files=["IPCC_AR6_WGII_Chapter03.pdf"] ).load_data() # Shuffle the documents import random random.seed(42) random.shuffle(documents) gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

In [ ]:

question_gen_query = (
    "You are a Teacher/ Professor. Your task is to setup "
    "a quiz/examination. Using the provided context, formulate "
    "a single question that captures an important fact from the "
    "context. Restrict the question to the context information provided."
)

dataset_generator = DatasetGenerator.from_documents(
    documents[:50],
    question_gen_query=question_gen_query,
    llm=gpt_35_llm,
)

question_gen_query = ( "You are a Teacher/ Professor. Your task is to setup " "a quiz/examination. Using the provided context, formulate " "a single question that captures an important fact from the " "context. Restrict the question to the context information provided." ) dataset_generator = DatasetGenerator.from_documents( documents[:50], question_gen_query=question_gen_query, llm=gpt_35_llm, )

In [ ]:

# NOTE: this may take some time. Go grab a coffee!
questions = dataset_generator.generate_questions_from_nodes(num=40)
print("Generated ", len(questions), " questions")

# NOTE: this may take some time. Go grab a coffee! questions = dataset_generator.generate_questions_from_nodes(num=40) print("Generated ", len(questions), " questions")

Generated  40  questions

In [ ]:

with open("train_questions.txt", "w") as f:
    for question in questions:
        f.write(question + "\n")

with open("train_questions.txt", "w") as f: for question in questions: f.write(question + "\n")

评估集生成¶

现在，我们将基于一组完全不同的文档来生成问题，以创建评估数据集。

In [ ]:

dataset_generator = DatasetGenerator.from_documents(
    documents[
        50:
    ],  # since we generated ~1 question for 40 documents, we can skip the first 40
    question_gen_query=question_gen_query,
    llm=gpt_35_llm,
)

dataset_generator = DatasetGenerator.from_documents( documents[ 50: ], # since we generated ~1 question for 40 documents, we can skip the first 40 question_gen_query=question_gen_query, llm=gpt_35_llm, )

In [ ]:

# NOTE: this may take some time. Go grab a coffee!
questions = dataset_generator.generate_questions_from_nodes(num=40)
print("Generated ", len(questions), " questions")

# NOTE: this may take some time. Go grab a coffee! questions = dataset_generator.generate_questions_from_nodes(num=40) print("Generated ", len(questions), " questions")

Generated  40  questions

In [ ]:

with open("eval_questions.txt", "w") as f:
    for question in questions:
        f.write(question + "\n")

with open("eval_questions.txt", "w") as f: for question in questions: f.write(question + "\n")

基于 GPT-3.5-Turbo 查询引擎的初始评估¶

本次评估将使用 ragas 评估库。

Ragas 提供了丰富的 RAG 管道评估指标，具体可查阅指标说明文档。

本笔记本将采用以下两个核心指标：

• answer_relevancy（答案相关性）- 衡量生成答案与提示问题的相关程度。若答案不完整或包含冗余信息，则得分较低。该指标通过计算 LLM 基于给定答案生成对应问题的概率来量化，取值范围 (0,1)，数值越高越好。
• faithfulness（事实忠实度）- 评估生成答案与给定上下文的事实一致性。采用多步骤验证范式：首先生成答案中的陈述语句，随后逐条验证这些陈述与上下文的匹配度。结果归一化至 (0,1) 范围，数值越高越好。

In [ ]:

questions = []
with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("eval_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

from llama_index.core import VectorStoreIndex

# limit the context window to 2048 tokens so that refine is used
from llama_index.core import Settings

Settings.context_window = 2048

index = VectorStoreIndex.from_documents(
    documents,
)

query_engine = index.as_query_engine(similarity_top_k=2, llm=gpt_35_llm)

from llama_index.core import VectorStoreIndex # limit the context window to 2048 tokens so that refine is used from llama_index.core import Settings Settings.context_window = 2048 index = VectorStoreIndex.from_documents( documents, ) query_engine = index.as_query_engine(similarity_top_k=2, llm=gpt_35_llm)

In [ ]:

contexts = []
answers = []

for question in questions:
    response = query_engine.query(question)
    contexts.append([x.node.get_content() for x in response.source_nodes])
    answers.append(str(response))

contexts = [] answers = [] for question in questions: response = query_engine.query(question) contexts.append([x.node.get_content() for x in response.source_nodes]) answers.append(str(response))

In [ ]:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness

ds = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
    }
)

result = evaluate(ds, [answer_relevancy, faithfulness])
print(result)

from datasets import Dataset from ragas import evaluate from ragas.metrics import answer_relevancy, faithfulness ds = Dataset.from_dict( { "question": questions, "answer": answers, "contexts": contexts, } ) result = evaluate(ds, [answer_relevancy, faithfulness]) print(result)

evaluating with [answer_relevancy]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [01:02<00:00, 20.69s/it]

evaluating with [faithfulness]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:52<00:00, 77.37s/it]

{'ragas_score': 0.8356, 'answer_relevancy': 0.9725, 'faithfulness': 0.7325}

使用 GPT-4 收集训练数据¶

在此环节，我们利用 GPT-4 和 OpenAIFineTuningHandler 来收集所需的训练数据。

In [ ]:

from llama_index.llms.openai import OpenAI
from llama_index.finetuning.callbacks import OpenAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager

finetuning_handler = OpenAIFineTuningHandler()
callback_manager = CallbackManager([finetuning_handler])

llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)
llm.callback_manager = callback_manager

from llama_index.llms.openai import OpenAI from llama_index.finetuning.callbacks import OpenAIFineTuningHandler from llama_index.core.callbacks import CallbackManager finetuning_handler = OpenAIFineTuningHandler() callback_manager = CallbackManager([finetuning_handler]) llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3) llm.callback_manager = callback_manager

In [ ]:

questions = []
with open("train_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("train_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(
    documents,
)

query_engine = index.as_query_engine(similarity_top_k=2, llm=llm)

from llama_index.core import VectorStoreIndex index = VectorStoreIndex.from_documents( documents, ) query_engine = index.as_query_engine(similarity_top_k=2, llm=llm)

In [ ]:

for question in questions:
    response = query_engine.query(question)

for question in questions: response = query_engine.query(question)

创建 OpenAIFinetuneEngine¶

我们创建一个 OpenAIFinetuneEngine：该微调引擎将负责启动微调任务，并返回一个可直接接入 LlamaIndex 工作流的 LLM 模型。

此处使用默认构造函数，但我们也可以通过 from_finetuning_handler 类方法直接将自定义的 finetuning_handler 传入该引擎。

In [ ]:

finetuning_handler.save_finetuning_events("finetuning_events.jsonl")

finetuning_handler.save_finetuning_events("finetuning_events.jsonl")

In [ ]:

from llama_index.finetuning import OpenAIFinetuneEngine

finetune_engine = OpenAIFinetuneEngine(
    "gpt-3.5-turbo",
    "finetuning_events.jsonl",
    # start_job_id="<start-job-id>"  # if you have an existing job, can specify id here
)

# finetune_engine = OpenAIFinetuneEngine.from_finetuning_handler(
#     finetuning_handler,
#     "gpt-3.5-turbo",
#     "tmp.jsonl"
# )

from llama_index.finetuning import OpenAIFinetuneEngine finetune_engine = OpenAIFinetuneEngine( "gpt-3.5-turbo", "finetuning_events.jsonl", # start_job_id="" # if you have an existing job, can specify id here ) # finetune_engine = OpenAIFinetuneEngine.from_finetuning_handler( # finetuning_handler, # "gpt-3.5-turbo", # "tmp.jsonl" # )

In [ ]:

finetune_engine.finetune()

finetune_engine.finetune()

Num examples: 61
First example:
{'role': 'system', 'content': "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines."}
{'role': 'user', 'content': 'Context information is below.\n---------------------\npage_label: 410\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\nIt is challenging to apply this experimental approach to communities or ecosystems (see Figure \nBox\xa03.1.1).To date, most research on community or ecosystem response to climate-induced drivers has been in large-volume (>10,000 l) \nmesocosms (Riebesell and Gattuso, 2014), or at natural analogues such as CO 2 seeps, in which only one driver (ocean acidification) is \naltered (see (4) in Figure Box\xa03.1.1).Only very recently have two drivers been incorporated into climate-change manipulation studies \nexamining responses of primary producers to secondary consumers (see (5) in Figure Box\xa03.1.1a; Nagelkerken et\xa0al., 2020).Therefore, \n‘natural experiments’ from the geological past (Reddin et\xa0al., 2020) provide insights into how food webs and their constituents respond to \ncomplex change involving multiple drivers.Contemporary observations are occasionally long enough (>50\xa0years) to capture community \nresponses to complex climate change.For example, Brun et\xa0al.(2019) reported a shift in zooplankton community structure in the North \nAtlantic (1960–2014), with major biogeochemical ramifications.Conducting sufficiently long manipulation experiments to study the effect of adaptation on organisms is equally difficult (see Figure \nBox\xa03.1.1b), with much research restricted to multi-year studies of the microevolution of fast-growing (more than one division per day) \nphytoplankton species responding to single drivers (Lohbeck et\xa0al., 2012; Schaum et\xa0al., 2016).In a few experimental evolution studies \n(see (7) in Figure Box\xa03.1.1a; Brennan et\xa0al., 2017), multiple drivers have been used, but none have used communities or ecosystems (see \nFigure Box\xa03.1.1b).Nevertheless, the fossil record provides limited evidence of adaptations to less rapid (relative to present day) climate \nchange (Jackson et\xa0al., 2018).Despite the need to explore ecological or biogeochemical responses to projected future ocean conditions, \nlogistical challenges require that assessments of climate-change impacts at scales larger than mesocosms use large-scale, long-term in \nsitu observational studies (as documented in Section\xa03.4).\n\npage_label: 409\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\n3\n409Oceans and Coastal Ecosystems and Their Services  Chapter 3\nunderlies inhibited thermal adaptation under nitrogen-limited \nconditions (low confidence) (Aranguren-Gassis et\xa0 al., 2019).When \nselection is strong due to unfavourable environmental conditions, \nmicrobial populations can encounter functional and evolutionary \ntrade-offs evidenced by reducing growth rates while increasing \ntolerance and metabolism of reactive oxygen species (Lindberg and \nCollins, 2020).Other trade-offs can be observed in offspring quality \nand number (Lindberg and Collins, 2020).These findings contribute \ntowards a mechanistic framework describing the range of evolutionary \nstrategies in response to multiple drivers (Collins et\xa0al., 2020), but other \nhazards, such as extreme events (e.g., MHWs), still need to be included \nbecause their characteristics may alter the potential for adaptation of \nspecies and populations to climate change (Gruber et\xa0al., 2021).3.3.5 Ecological Response to Multiple Drivers\nAssessing ecological responses to multiple climate-induced drivers \nrequires a combination of approaches, including laboratory- and \nfield-based experiments, field observations (e.g., natural gradients, \nclimate analogues), study of paleo-analogues and the development \nof mechanistic and empirical models (Clapham, 2019; Gissi et\xa0 al., \n2021).Experimental studies of food-web responses are often limited \nto an individual driver, although recent manipulations have used a \nmatrix of >1000-l mesocosms to explore ecological responses to both \nwarming and acidification (see Box\xa0 3.1; Nagelkerken et\xa0 al., 2020).Hence, complementary approaches are needed to indirectly explore \nthe mechanisms underlying ecosystem responses to global climate \nchange (Parmesan et\xa0al., 2013).Observations from time series longer \nthan modes of natural variability (i.e., decades) are essential for \nrevealing and attributing ecological responses to climate change (e.g., \nSection\xa03.4; Barton et\xa0al., 2015b; Brun et\xa0al., 2019).Also, paleorecords \nprovide insights into the influence of multiple drivers on marine \nbiota (Cross-Chapter Box\xa0 PALEO in Chapter\xa0 1; Reddin et\xa0 al., 2020).Specifically, associations between vulnerabilities and traits of marine \nectotherms in laboratory experiments correspond with organismal \nresponses to ancient hyperthermal events (medium confidence) \n(Reddin et\xa0 al., 2020).This corroboration suggests that responses to \nmultiple drivers inferred from the fossil record can help provide insights \ninto the future status of functional groups, and hence food webs, under \nrapid climate change.Multi-species and integrated end-to-end ecosystem models are \npowerful tools to explore and project outcomes to the often-interacting \ncumulative effects of climate change and other anthropogenic drivers \n(Section\xa03.1; Kaplan and Marshall, 2016; Koenigstein et\xa0al., 2016; Peck \nand Pinnegar, 2018; Tittensor et\xa0 al., 2018; Gissi et\xa0 al., 2021).These \nmodels can integrate some aspects of the knowledge accrued from \nmanipulation experiments, paleo- and contemporary observations, help \ntest the relative importance of specific drivers and driver combinations, \nand identify synergistic or antagonistic responses (Koenigstein et\xa0al., \n2016; Payne et\xa0al., 2016; Skogen et\xa0al., 2018; Tittensor et\xa0al., 2018).As these models are associated with wide-ranging uncertainties \n(SM3.2.2; Payne et\xa0 al., 2016; Trolle et\xa0 al., 2019; Heneghan et\xa0 al., \n2021), they cannot be expected to accurately project the trajectories \nof complex marine ecosystems under climate change; hence, they are \nmost useful for assessing overall trends and in particular for providing a plausible envelope of trajectories across a range of assumptions \n(Fulton et\xa0al., 2018; Peck et\xa0al., 2018; Tittensor et\xa0al., 2018).\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: What are some approaches used to assess ecological responses to multiple climate-induced drivers in the context of climate change and the oceans?\nAnswer: '}
{'role': 'assistant', 'content': 'Several approaches are used to assess ecological responses to multiple climate-induced drivers. These include laboratory- and field-based experiments, field observations such as natural gradients and climate analogues, the study of paleo-analogues, and the development of mechanistic and empirical models. Experimental studies often focus on individual drivers, but recent manipulations have used large-volume mesocosms to explore ecological responses to both warming and acidification. Observations from time series longer than modes of natural variability are essential for revealing and attributing ecological responses to climate change. Paleorecords also provide insights into the influence of multiple drivers on marine biota. Multi-species and integrated end-to-end ecosystem models are powerful tools to explore and project outcomes to the often-interacting cumulative effects of climate change and other anthropogenic drivers. These models can integrate some aspects of the knowledge accrued from manipulation experiments, paleo- and contemporary observations, help test the relative importance of specific drivers and driver combinations, and identify synergistic or antagonistic responses.'}
No errors found
Num examples missing system message: 21
Num examples missing user message: 0

#### Distribution of num_messages_per_example:
min / max: 2, 3
mean / median: 2.6557377049180326, 3.0
p5 / p95: 2.0, 3.0

#### Distribution of num_total_tokens_per_example:
min / max: 229, 2011
mean / median: 1274.27868852459, 1385.0
p5 / p95: 533.0, 1848.0

#### Distribution of num_assistant_tokens_per_example:
min / max: 11, 334
mean / median: 72.36065573770492, 37.0
p5 / p95: 23.0, 193.0

0 examples may be over the 4096 token limit, they will be truncated during fine-tuning
Dataset has ~77731 tokens that will be charged for during training
By default, you'll train for 3 epochs on this dataset
By default, you'll be charged for ~233193 tokens
As of Augest 22, 2023, fine-tuning gpt-3.5-turbo is $0.008 / 1K Tokens.
This means your total cost for training will be $0.621848 per epoch.
Waiting for file to be ready...

In [ ]:

finetune_engine.get_current_job()

finetune_engine.get_current_job()

Out[ ]:

<FineTuningJob fine_tuning.job id=ftjob-u9T7BF5zRxVX4n5b9Jtbb5cR at 0x2c641fe20> JSON: {
  "object": "fine_tuning.job",
  "id": "ftjob-u9T7BF5zRxVX4n5b9Jtbb5cR",
  "model": "gpt-3.5-turbo-0613",
  "created_at": 1693254044,
  "finished_at": null,
  "fine_tuned_model": null,
  "organization_id": "org-1ZDAvajC6v2ZtAP9hLEIsXRz",
  "result_files": [],
  "status": "running",
  "validation_file": null,
  "training_file": "file-j1fwmqIAoqZXWZQ8EqwHucXs",
  "hyperparameters": {
    "n_epochs": 3
  },
  "trained_tokens": null
}

In [ ]:

ft_llm = finetune_engine.get_finetuned_model(temperature=0.3)

ft_llm = finetune_engine.get_finetuned_model(temperature=0.3)

评估¶

经过一段时间后，您的模型将完成训练！

下一步是在我们的评估数据集上再次运行微调后的模型，以衡量性能提升情况。

In [ ]:

from llama_index.llms.openai import OpenAI
from llama_index.finetuning.callbacks import OpenAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager


# Option 1: pass in ft_llm directly into Settings
from llama_index.core import Settings

Settings.llm = ft_llm
Settings.context_window = (
    2048  # limit the context window artifically to test refine process
)

# # Option 2: you can also specify the model name manually
# ft_model_name = "ft:gpt-3.5-turbo-0613:..."
# Settings.llm = OpenAI(model=ft_model_name, temperature=0.3)

from llama_index.llms.openai import OpenAI from llama_index.finetuning.callbacks import OpenAIFineTuningHandler from llama_index.core.callbacks import CallbackManager # Option 1: pass in ft_llm directly into Settings from llama_index.core import Settings Settings.llm = ft_llm Settings.context_window = ( 2048 # limit the context window artifically to test refine process ) # # Option 2: you can also specify the model name manually # ft_model_name = "ft:gpt-3.5-turbo-0613:..." # Settings.llm = OpenAI(model=ft_model_name, temperature=0.3)

In [ ]:

questions = []
with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("eval_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine(similarity_top_k=2, llm=ft_llm)

from llama_index.core import VectorStoreIndex index = VectorStoreIndex.from_documents(documents) query_engine = index.as_query_engine(similarity_top_k=2, llm=ft_llm)

In [ ]:

contexts = []
answers = []

for question in questions:
    response = query_engine.query(question)
    contexts.append([x.node.get_content() for x in response.source_nodes])
    answers.append(str(response))

contexts = [] answers = [] for question in questions: response = query_engine.query(question) contexts.append([x.node.get_content() for x in response.source_nodes]) answers.append(str(response))

In [ ]:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness

ds = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
    }
)

result = evaluate(ds, [answer_relevancy, faithfulness])
print(result)

from datasets import Dataset from ragas import evaluate from ragas.metrics import answer_relevancy, faithfulness ds = Dataset.from_dict( { "question": questions, "answer": answers, "contexts": contexts, } ) result = evaluate(ds, [answer_relevancy, faithfulness]) print(result)

evaluating with [answer_relevancy]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:49<00:00, 16.34s/it]

evaluating with [faithfulness]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [04:04<00:00, 81.44s/it]

{'ragas_score': 0.8680, 'answer_relevancy': 0.9607, 'faithfulness': 0.7917}

探索差异¶

让我们快速比较响应中的差异，以证明微调确实改变了某些内容。

In [ ]:

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

from llama_index.core import VectorStoreIndex index = VectorStoreIndex.from_documents(documents)

In [ ]:

questions = []
with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("eval_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

print(questions[12])

print(questions[12])

What is a key barrier globally for ocean health, governance, and adaptation to climate change, according to the report?

原文¶

In [ ]:

from llama_index.core.response.notebook_utils import display_response
from llama_index.llms.openai import OpenAI


gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

from llama_index.core.response.notebook_utils import display_response from llama_index.llms.openai import OpenAI gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

In [ ]:

query_engine = index.as_query_engine(llm=gpt_35_llm)

response = query_engine.query(questions[12])

display_response(response)

query_engine = index.as_query_engine(llm=gpt_35_llm) response = query_engine.query(questions[12]) display_response(response)

Final Response: A key barrier globally for ocean health, governance, and adaptation to climate change, according to the report, is the availability of technology, knowledge, and financial support, as well as existing governance structures.

微调¶

In [ ]:

query_engine = index.as_query_engine(llm=ft_llm)

response = query_engine.query(questions[12])

display_response(response)

query_engine = index.as_query_engine(llm=ft_llm) response = query_engine.query(questions[12]) display_response(response)

Final Response: The report identifies a broad range of barriers and limits for adaptation to climate change in ecosystems and human systems. These include the availability of technology, knowledge, and financial support, as well as existing governance structures. Existing ocean-governance structures are already facing multi-dimensional, scale-related challenges because of climate change.

如我们所见，经过微调的模型提供了更全面的回答！这与ragas评估中更高的忠实度分数相符，因为该回答更能代表检索到的上下文内容。

结论¶

综上所述，仅使用约61个问题进行微调确实有助于提升我们的评估分数！

答案相关性：0.9725 -> 0.9607

答案相关性略有下降，但幅度非常小。

忠实度：0.7325 -> 0.7917

忠实度似乎得到了提升！这意味着生成的答案能更好地满足原始提问的需求。