从向量数据库自动检索¶

本指南展示如何在 LlamaIndex 中执行自动检索功能。

除了支持语义搜索的查询字符串外，许多主流向量数据库还提供一组元数据过滤器。给定自然语言查询时，我们首先使用大语言模型(LLM)推断出一组元数据过滤器以及要传递给向量数据库的正确查询字符串（两者均可为空）。然后将这个完整的查询组合体在向量数据库上执行。

这种方法支持比单纯top-k语义搜索更动态、更具表现力的检索形式。特定查询的相关上下文可能只需要对元数据标签进行过滤，或者需要在过滤后的集合中结合使用过滤+语义搜索，亦或仅需原始语义搜索。

我们以Elasticsearch为例进行演示，但自动检索功能同样已在许多其他向量数据库中实现（如Pinecone、Weaviate等）。

安装配置¶

我们首先进行必要的导入定义。

如果您在 Colab 上打开此 Notebook，可能需要安装 LlamaIndex 🦙。

In [ ]:

%pip install llama-index-vector-stores-elasticsearch

%pip install llama-index-vector-stores-elasticsearch

In [ ]:

!pip install llama-index

!pip install llama-index

In [ ]:

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

import logging import sys logging.basicConfig(stream=sys.stdout, level=logging.INFO) logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

In [ ]:

# set up OpenAI
import os
import getpass

os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:")
import openai

openai.api_key = os.environ["OPENAI_API_KEY"]

# set up OpenAI import os import getpass os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:") import openai openai.api_key = os.environ["OPENAI_API_KEY"]

定义示例数据¶

我们向向量数据库中插入一些包含文本块的示例节点。请注意，每个 TextNode 不仅包含文本内容，还包含元数据（例如 category 和 country）。这些元数据字段将在底层向量数据库中以原样形式被转换/存储。

In [ ]:

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.elasticsearch import ElasticsearchStore

from llama_index.core import VectorStoreIndex, StorageContext from llama_index.vector_stores.elasticsearch import ElasticsearchStore

In [ ]:

from llama_index.core.schema import TextNode

nodes = [
    TextNode(
        text=(
            "A bunch of scientists bring back dinosaurs and mayhem breaks"
            " loose"
        ),
        metadata={"year": 1993, "rating": 7.7, "genre": "science fiction"},
    ),
    TextNode(
        text=(
            "Leo DiCaprio gets lost in a dream within a dream within a dream"
            " within a ..."
        ),
        metadata={
            "year": 2010,
            "director": "Christopher Nolan",
            "rating": 8.2,
        },
    ),
    TextNode(
        text=(
            "A psychologist / detective gets lost in a series of dreams within"
            " dreams within dreams and Inception reused the idea"
        ),
        metadata={"year": 2006, "director": "Satoshi Kon", "rating": 8.6},
    ),
    TextNode(
        text=(
            "A bunch of normal-sized women are supremely wholesome and some"
            " men pine after them"
        ),
        metadata={"year": 2019, "director": "Greta Gerwig", "rating": 8.3},
    ),
    TextNode(
        text="Toys come alive and have a blast doing so",
        metadata={"year": 1995, "genre": "animated"},
    ),
]

from llama_index.core.schema import TextNode nodes = [ TextNode( text=( "A bunch of scientists bring back dinosaurs and mayhem breaks" " loose" ), metadata={"year": 1993, "rating": 7.7, "genre": "science fiction"}, ), TextNode( text=( "Leo DiCaprio gets lost in a dream within a dream within a dream" " within a ..." ), metadata={ "year": 2010, "director": "Christopher Nolan", "rating": 8.2, }, ), TextNode( text=( "A psychologist / detective gets lost in a series of dreams within" " dreams within dreams and Inception reused the idea" ), metadata={"year": 2006, "director": "Satoshi Kon", "rating": 8.6}, ), TextNode( text=( "A bunch of normal-sized women are supremely wholesome and some" " men pine after them" ), metadata={"year": 2019, "director": "Greta Gerwig", "rating": 8.3}, ), TextNode( text="Toys come alive and have a blast doing so", metadata={"year": 1995, "genre": "animated"}, ), ]

使用 Elasticsearch 向量存储构建向量索引¶

此处我们将数据加载至向量存储中。如上所述，每个节点的文本和元数据都将转换为 Elasticsearch 中的对应表示形式。现在我们可以对该数据执行语义查询，并通过 Elasticsearch 进行元数据过滤。

In [ ]:

vector_store = ElasticsearchStore(
    index_name="auto_retriever_movies", es_url="http://localhost:9200"
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

vector_store = ElasticsearchStore( index_name="auto_retriever_movies", es_url="http://localhost:9200" ) storage_context = StorageContext.from_defaults(vector_store=vector_store)

In [ ]:

index = VectorStoreIndex(nodes, storage_context=storage_context)

index = VectorStoreIndex(nodes, storage_context=storage_context)

定义 VectorIndexAutoRetriever¶

我们定义核心模块 VectorIndexAutoRetriever。该模块接收 VectorStoreInfo 作为输入参数，其中包含向量存储集合的结构化描述及其支持的元数据过滤器信息。这些信息将被用于自动检索提示中，供大语言模型（LLM）推断元数据过滤器。

In [ ]:

from llama_index.core.retrievers import VectorIndexAutoRetriever
from llama_index.core.vector_stores import MetadataInfo, VectorStoreInfo


vector_store_info = VectorStoreInfo(
    content_info="Brief summary of a movie",
    metadata_info=[
        MetadataInfo(
            name="genre",
            description="The genre of the movie",
            type="string or list[string]",
        ),
        MetadataInfo(
            name="year",
            description="The year the movie was released",
            type="integer",
        ),
        MetadataInfo(
            name="director",
            description="The name of the movie director",
            type="string",
        ),
        MetadataInfo(
            name="rating",
            description="A 1-10 rating for the movie",
            type="float",
        ),
    ],
)
retriever = VectorIndexAutoRetriever(
    index, vector_store_info=vector_store_info
)

from llama_index.core.retrievers import VectorIndexAutoRetriever from llama_index.core.vector_stores import MetadataInfo, VectorStoreInfo vector_store_info = VectorStoreInfo( content_info="Brief summary of a movie", metadata_info=[ MetadataInfo( name="genre", description="The genre of the movie", type="string or list[string]", ), MetadataInfo( name="year", description="The year the movie was released", type="integer", ), MetadataInfo( name="director", description="The name of the movie director", type="string", ), MetadataInfo( name="rating", description="A 1-10 rating for the movie", type="float", ), ], ) retriever = VectorIndexAutoRetriever( index, vector_store_info=vector_store_info )

处理示例数据¶

我们尝试对一些示例数据进行处理。请注意元数据过滤器是如何被自动推断的——这有助于实现更精确的检索！

In [ ]:

retriever.retrieve(
    "What are 2 movies by Christopher Nolan were made before 2020?"
)

retriever.retrieve( "What are 2 movies by Christopher Nolan were made before 2020?" )

In [ ]:

retriever.retrieve("Has Andrei Tarkovsky directed any science fiction movies")

retriever.retrieve("Has Andrei Tarkovsky directed any science fiction movies")

INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using query str: science fiction
Using query str: science fiction
INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using filters: {'director': 'Andrei Tarkovsky'}
Using filters: {'director': 'Andrei Tarkovsky'}
INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using top_k: 2
Using top_k: 2
INFO:elastic_transport.transport:POST http://localhost:9200/auto_retriever_movies/_search [status:200 duration:0.042s]
POST http://localhost:9200/auto_retriever_movies/_search [status:200 duration:0.042s]

Out[ ]:

[]