提取元数据以优化文档索引与理解¶

在许多情况下，尤其是处理长文档时，文本片段可能缺乏必要的上下文来与其他相似片段进行区分。解决方法之一是在数据集或知识库中手动标记每个文本片段。然而，对于大量或持续更新的文档集而言，这种方法需要耗费大量人力且效率低下。

为解决这个问题，我们利用大语言模型（LLMs）提取与文档相关的特定上下文信息，从而更好地帮助检索模型和语言模型区分看似相似的文本段落。

这一功能通过我们全新的 Metadata Extractor 模块实现。

如果您在 Colab 上打开此 Notebook，可能需要安装 LlamaIndex 🦙。

In [ ]:

%pip install llama-index-llms-openai
%pip install llama-index-extractors-entity

%pip install llama-index-llms-openai %pip install llama-index-extractors-entity

In [ ]:

!pip install llama-index

!pip install llama-index

In [ ]:

import nest_asyncio

nest_asyncio.apply()

import os
import openai

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY_HERE"

import nest_asyncio nest_asyncio.apply() import os import openai os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY_HERE"

In [ ]:

from llama_index.llms.openai import OpenAI
from llama_index.core.schema import MetadataMode

from llama_index.llms.openai import OpenAI from llama_index.core.schema import MetadataMode

In [ ]:

llm = OpenAI(temperature=0.1, model="gpt-3.5-turbo", max_tokens=512)

llm = OpenAI(temperature=0.1, model="gpt-3.5-turbo", max_tokens=512)

我们创建一个节点解析器，用于提取文档标题以及与文档片段相关的假设性问题嵌入。

同时演示如何实例化 SummaryExtractor 和 KeywordExtractor，以及如何基于 BaseExtractor 基类创建自定义提取器

In [ ]:

from llama_index.core.extractors import (
    SummaryExtractor,
    QuestionsAnsweredExtractor,
    TitleExtractor,
    KeywordExtractor,
    BaseExtractor,
)
from llama_index.extractors.entity import EntityExtractor
from llama_index.core.node_parser import TokenTextSplitter

text_splitter = TokenTextSplitter(
    separator=" ", chunk_size=512, chunk_overlap=128
)


class CustomExtractor(BaseExtractor):
    def extract(self, nodes):
        metadata_list = [
            {
                "custom": (
                    node.metadata["document_title"]
                    + "\n"
                    + node.metadata["excerpt_keywords"]
                )
            }
            for node in nodes
        ]
        return metadata_list


extractors = [
    TitleExtractor(nodes=5, llm=llm),
    QuestionsAnsweredExtractor(questions=3, llm=llm),
    # EntityExtractor(prediction_threshold=0.5),
    # SummaryExtractor(summaries=["prev", "self"], llm=llm),
    # KeywordExtractor(keywords=10, llm=llm),
    # CustomExtractor()
]

transformations = [text_splitter] + extractors

from llama_index.core.extractors import ( SummaryExtractor, QuestionsAnsweredExtractor, TitleExtractor, KeywordExtractor, BaseExtractor, ) from llama_index.extractors.entity import EntityExtractor from llama_index.core.node_parser import TokenTextSplitter text_splitter = TokenTextSplitter( separator=" ", chunk_size=512, chunk_overlap=128 ) class CustomExtractor(BaseExtractor): def extract(self, nodes): metadata_list = [ { "custom": ( node.metadata["document_title"] + "\n" + node.metadata["excerpt_keywords"] ) } for node in nodes ] return metadata_list extractors = [ TitleExtractor(nodes=5, llm=llm), QuestionsAnsweredExtractor(questions=3, llm=llm), # EntityExtractor(prediction_threshold=0.5), # SummaryExtractor(summaries=["prev", "self"], llm=llm), # KeywordExtractor(keywords=10, llm=llm), # CustomExtractor() ] transformations = [text_splitter] + extractors

In [ ]:

from llama_index.core import SimpleDirectoryReader

from llama_index.core import SimpleDirectoryReader

我们首先分别加载 Uber 和 Lyft 2019 年与 2020 年的 10-K 年度 SEC 报告。

In [ ]:

!mkdir -p data
!wget -O "data/10k-132.pdf" "https://www.dropbox.com/scl/fi/6dlqdk6e2k1mjhi8dee5j/uber.pdf?rlkey=2jyoe49bg2vwdlz30l76czq6g&dl=1"
!wget -O "data/10k-vFinal.pdf" "https://www.dropbox.com/scl/fi/qn7g3vrk5mqb18ko4e5in/lyft.pdf?rlkey=j6jxtjwo8zbstdo4wz3ns8zoj&dl=1"

!mkdir -p data !wget -O "data/10k-132.pdf" "https://www.dropbox.com/scl/fi/6dlqdk6e2k1mjhi8dee5j/uber.pdf?rlkey=2jyoe49bg2vwdlz30l76czq6g&dl=1" !wget -O "data/10k-vFinal.pdf" "https://www.dropbox.com/scl/fi/qn7g3vrk5mqb18ko4e5in/lyft.pdf?rlkey=j6jxtjwo8zbstdo4wz3ns8zoj&dl=1"

In [ ]:

# Note the uninformative document file name, which may be a common scenario in a production setting
uber_docs = SimpleDirectoryReader(input_files=["data/10k-132.pdf"]).load_data()
uber_front_pages = uber_docs[0:3]
uber_content = uber_docs[63:69]
uber_docs = uber_front_pages + uber_content

# Note the uninformative document file name, which may be a common scenario in a production setting uber_docs = SimpleDirectoryReader(input_files=["data/10k-132.pdf"]).load_data() uber_front_pages = uber_docs[0:3] uber_content = uber_docs[63:69] uber_docs = uber_front_pages + uber_content

In [ ]:

from llama_index.core.ingestion import IngestionPipeline

pipeline = IngestionPipeline(transformations=transformations)

uber_nodes = pipeline.run(documents=uber_docs)

from llama_index.core.ingestion import IngestionPipeline pipeline = IngestionPipeline(transformations=transformations) uber_nodes = pipeline.run(documents=uber_docs)

In [ ]:

uber_nodes[1].metadata

uber_nodes[1].metadata

Out[ ]:

{'page_label': '2',
 'file_name': '10k-132.pdf',
 'document_title': 'Exploring the Diverse Landscape of 2019: A Comprehensive Annual Report on Uber Technologies, Inc.',
 'questions_this_excerpt_can_answer': '1. How many countries does Uber operate in?\n2. What is the total gross bookings of Uber in 2019?\n3. How many trips did Uber facilitate in 2019?'}

In [ ]:

# Note the uninformative document file name, which may be a common scenario in a production setting
lyft_docs = SimpleDirectoryReader(
    input_files=["data/10k-vFinal.pdf"]
).load_data()
lyft_front_pages = lyft_docs[0:3]
lyft_content = lyft_docs[68:73]
lyft_docs = lyft_front_pages + lyft_content

# Note the uninformative document file name, which may be a common scenario in a production setting lyft_docs = SimpleDirectoryReader( input_files=["data/10k-vFinal.pdf"] ).load_data() lyft_front_pages = lyft_docs[0:3] lyft_content = lyft_docs[68:73] lyft_docs = lyft_front_pages + lyft_content

In [ ]:

from llama_index.core.ingestion import IngestionPipeline

pipeline = IngestionPipeline(transformations=transformations)

lyft_nodes = pipeline.run(documents=lyft_docs)

from llama_index.core.ingestion import IngestionPipeline pipeline = IngestionPipeline(transformations=transformations) lyft_nodes = pipeline.run(documents=lyft_docs)

In [ ]:

lyft_nodes[2].metadata

lyft_nodes[2].metadata

Out[ ]:

{'page_label': '2',
 'file_name': '10k-vFinal.pdf',
 'document_title': 'Lyft, Inc. Annual Report on Form 10-K for the Fiscal Year Ended December 31, 2020',
 'questions_this_excerpt_can_answer': "1. Has Lyft, Inc. filed a report on and attestation to its management's assessment of the effectiveness of its internal control over financial reporting under Section 404(b) of the Sarbanes-Oxley Act?\n2. Is Lyft, Inc. considered a shell company according to Rule 12b-2 of the Exchange Act?\n3. What was the aggregate market value of Lyft, Inc.'s common stock held by non-affiliates on June 30, 2020?"}

由于我们提出的问题较为复杂，在以下所有问答流程中均采用了子问题查询引擎，并通过提示机制使其更加关注检索来源的相关性。

In [ ]:

from llama_index.core.question_gen import LLMQuestionGenerator
from llama_index.core.question_gen.prompts import (
    DEFAULT_SUB_QUESTION_PROMPT_TMPL,
)


question_gen = LLMQuestionGenerator.from_defaults(
    llm=llm,
    prompt_template_str="""
        Follow the example, but instead of giving a question, always prefix the question 
        with: 'By first identifying and quoting the most relevant sources, '. 
        """
    + DEFAULT_SUB_QUESTION_PROMPT_TMPL,
)

from llama_index.core.question_gen import LLMQuestionGenerator from llama_index.core.question_gen.prompts import ( DEFAULT_SUB_QUESTION_PROMPT_TMPL, ) question_gen = LLMQuestionGenerator.from_defaults( llm=llm, prompt_template_str=""" Follow the example, but instead of giving a question, always prefix the question with: 'By first identifying and quoting the most relevant sources, '. """ + DEFAULT_SUB_QUESTION_PROMPT_TMPL, )

无额外元数据的索引查询¶

In [ ]:

from copy import deepcopy

nodes_no_metadata = deepcopy(uber_nodes) + deepcopy(lyft_nodes)
for node in nodes_no_metadata:
    node.metadata = {
        k: node.metadata[k]
        for k in node.metadata
        if k in ["page_label", "file_name"]
    }
print(
    "LLM sees:\n",
    (nodes_no_metadata)[9].get_content(metadata_mode=MetadataMode.LLM),
)

from copy import deepcopy nodes_no_metadata = deepcopy(uber_nodes) + deepcopy(lyft_nodes) for node in nodes_no_metadata: node.metadata = { k: node.metadata[k] for k in node.metadata if k in ["page_label", "file_name"] } print( "LLM sees:\n", (nodes_no_metadata)[9].get_content(metadata_mode=MetadataMode.LLM), )

LLM sees:
 [Excerpt from document]
page_label: 65
file_name: 10k-132.pdf
Excerpt:
-----
See the section titled “Reconciliations of Non-GAAP Financial Measures” for our definition and a 
reconciliation of net income (loss) attributable to  Uber Technologies, Inc. to Adjusted EBITDA. 
            
  Year Ended December 31,   2017 to 2018   2018 to 2019   
(In millions, exce pt percenta ges)  2017   2018   2019   % Chan ge  % Chan ge  
Adjusted EBITDA ................................  $ (2,642) $ (1,847) $ (2,725)  30%  (48)%
-----

In [ ]:

from llama_index.core import VectorStoreIndex
from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata

from llama_index.core import VectorStoreIndex from llama_index.core.query_engine import SubQuestionQueryEngine from llama_index.core.tools import QueryEngineTool, ToolMetadata

In [ ]:

index_no_metadata = VectorStoreIndex(
    nodes=nodes_no_metadata,
)
engine_no_metadata = index_no_metadata.as_query_engine(
    similarity_top_k=10, llm=OpenAI(model="gpt-4")
)

index_no_metadata = VectorStoreIndex( nodes=nodes_no_metadata, ) engine_no_metadata = index_no_metadata.as_query_engine( similarity_top_k=10, llm=OpenAI(model="gpt-4") )

In [ ]:

final_engine_no_metadata = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=[
        QueryEngineTool(
            query_engine=engine_no_metadata,
            metadata=ToolMetadata(
                name="sec_filing_documents",
                description="financial information on companies",
            ),
        )
    ],
    question_gen=question_gen,
    use_async=True,
)

final_engine_no_metadata = SubQuestionQueryEngine.from_defaults( query_engine_tools=[ QueryEngineTool( query_engine=engine_no_metadata, metadata=ToolMetadata( name="sec_filing_documents", description="financial information on companies", ), ) ], question_gen=question_gen, use_async=True, )

In [ ]:

response_no_metadata = final_engine_no_metadata.query(
    """
    What was the cost due to research and development v.s. sales and marketing for uber and lyft in 2019 in millions of USD?
    Give your answer as a JSON.
    """
)
print(response_no_metadata.response)
# Correct answer:
# {"Uber": {"Research and Development": 4836, "Sales and Marketing": 4626},
#  "Lyft": {"Research and Development": 1505.6, "Sales and Marketing": 814 }}

response_no_metadata = final_engine_no_metadata.query( """ What was the cost due to research and development v.s. sales and marketing for uber and lyft in 2019 in millions of USD? Give your answer as a JSON. """ ) print(response_no_metadata.response) # Correct answer: # {"Uber": {"Research and Development": 4836, "Sales and Marketing": 4626}, # "Lyft": {"Research and Development": 1505.6, "Sales and Marketing": 814 }}

Generated 4 sub questions.
[sec_filing_documents] Q: What was the cost due to research and development for Uber in 2019
[sec_filing_documents] Q: What was the cost due to sales and marketing for Uber in 2019
[sec_filing_documents] Q: What was the cost due to research and development for Lyft in 2019
[sec_filing_documents] Q: What was the cost due to sales and marketing for Lyft in 2019
[sec_filing_documents] A: The cost due to sales and marketing for Uber in 2019 was $814,122 in thousands.
[sec_filing_documents] A: The cost due to research and development for Uber in 2019 was $1,505,640 in thousands.
[sec_filing_documents] A: The cost of research and development for Lyft in 2019 was $1,505,640 in thousands.
[sec_filing_documents] A: The cost due to sales and marketing for Lyft in 2019 was $814,122 in thousands.
{
  "Uber": {
    "Research and Development": 1505.64,
    "Sales and Marketing": 814.122
  },
  "Lyft": {
    "Research and Development": 1505.64,
    "Sales and Marketing": 814.122
  }
}

结果：如我们所见，问答代理似乎不知道去哪里寻找正确的文档，导致其完全混淆了Lyft和Uber的数据。

使用提取的元数据进行索引查询¶

In [ ]:

print(
    "LLM sees:\n",
    (uber_nodes + lyft_nodes)[9].get_content(metadata_mode=MetadataMode.LLM),
)

print( "LLM sees:\n", (uber_nodes + lyft_nodes)[9].get_content(metadata_mode=MetadataMode.LLM), )

LLM sees:
 [Excerpt from document]
page_label: 65
file_name: 10k-132.pdf
document_title: Exploring the Diverse Landscape of 2019: A Comprehensive Annual Report on Uber Technologies, Inc.
Excerpt:
-----
See the section titled “Reconciliations of Non-GAAP Financial Measures” for our definition and a 
reconciliation of net income (loss) attributable to  Uber Technologies, Inc. to Adjusted EBITDA. 
            
  Year Ended December 31,   2017 to 2018   2018 to 2019   
(In millions, exce pt percenta ges)  2017   2018   2019   % Chan ge  % Chan ge  
Adjusted EBITDA ................................  $ (2,642) $ (1,847) $ (2,725)  30%  (48)%
-----

In [ ]:

index = VectorStoreIndex(
    nodes=uber_nodes + lyft_nodes,
)
engine = index.as_query_engine(similarity_top_k=10, llm=OpenAI(model="gpt-4"))

index = VectorStoreIndex( nodes=uber_nodes + lyft_nodes, ) engine = index.as_query_engine(similarity_top_k=10, llm=OpenAI(model="gpt-4"))

In [ ]:

final_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=[
        QueryEngineTool(
            query_engine=engine,
            metadata=ToolMetadata(
                name="sec_filing_documents",
                description="financial information on companies.",
            ),
        )
    ],
    question_gen=question_gen,
    use_async=True,
)

final_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=[ QueryEngineTool( query_engine=engine, metadata=ToolMetadata( name="sec_filing_documents", description="financial information on companies.", ), ) ], question_gen=question_gen, use_async=True, )

In [ ]:

response = final_engine.query(
    """
    What was the cost due to research and development v.s. sales and marketing for uber and lyft in 2019 in millions of USD?
    Give your answer as a JSON.
    """
)
print(response.response)
# Correct answer:
# {"Uber": {"Research and Development": 4836, "Sales and Marketing": 4626},
#  "Lyft": {"Research and Development": 1505.6, "Sales and Marketing": 814 }}

response = final_engine.query( """ What was the cost due to research and development v.s. sales and marketing for uber and lyft in 2019 in millions of USD? Give your answer as a JSON. """ ) print(response.response) # Correct answer: # {"Uber": {"Research and Development": 4836, "Sales and Marketing": 4626}, # "Lyft": {"Research and Development": 1505.6, "Sales and Marketing": 814 }}

Generated 4 sub questions.
[sec_filing_documents] Q: What was the cost due to research and development for Uber in 2019
[sec_filing_documents] Q: What was the cost due to sales and marketing for Uber in 2019
[sec_filing_documents] Q: What was the cost due to research and development for Lyft in 2019
[sec_filing_documents] Q: What was the cost due to sales and marketing for Lyft in 2019
[sec_filing_documents] A: The cost due to sales and marketing for Uber in 2019 was $4,626 million.
[sec_filing_documents] A: The cost due to research and development for Uber in 2019 was $4,836 million.
[sec_filing_documents] A: The cost due to sales and marketing for Lyft in 2019 was $814,122 in thousands.
[sec_filing_documents] A: The cost of research and development for Lyft in 2019 was $1,505,640 in thousands.
{
  "Uber": {
    "Research and Development": 4836,
    "Sales and Marketing": 4626
  },
  "Lyft": {
    "Research and Development": 1505.64,
    "Sales and Marketing": 814.122
  }
}

结果：如我们所见，LLM 正确地回答了这些问题。

问题领域中发现的主要挑战¶

在本案例中，我们观察到向量嵌入提供的搜索质量较差。这可能是由于高度密集的金融文档与模型训练集的代表性不匹配所致。

为提高搜索质量，可采用其他更基于关键词的神经搜索方法（如ColBERTv2/PLAID）。这种方法尤其有助于通过特定关键词匹配来识别高相关性文本块。

其他有效改进措施包括：采用经金融数据集（如Bloomberg GPT）微调的模型。

此外，我们还可以通过提供文本块所在位置的更多上下文信息，来进一步丰富元数据。

本案例的改进方向¶

总体而言，可通过以下方式进一步提升本案例：更严格地评估元数据提取准确性，以及问答管道的准确率和召回率；引入更大规模的文档集及完整长度文档（这些文档可能包含更难消歧的复杂段落），从而对我们构建的系统进行压力测试并发现更多改进空间。