Milvus 向量存储 - 元数据过滤¶

本笔记本演示了如何在 LlamaIndex 中使用 Milvus 向量存储，重点介绍元数据过滤功能。您将学习如何为文档添加元数据索引、使用 LlamaIndex 内置的元数据过滤器执行向量搜索，以及如何向向量存储应用 Milvus 的原生过滤表达式。

完成本笔记本后，您将掌握如何利用 Milvus 的过滤功能，根据文档元数据来缩小搜索结果范围。

前提条件¶

安装依赖项

在开始之前，请确保已安装以下依赖项：

In [ ]:

Copied!

! pip install llama-index-vector-stores-milvus llama-index
! pip install llama-index-vector-stores-milvus llama-index

如果您使用的是 Google Colab，可能需要重启运行时环境（点击界面顶部的"Runtime"菜单，然后从下拉菜单中选择"Restart session"选项。）

设置账户

本教程使用 OpenAI 进行文本嵌入和答案生成。您需要准备 OpenAI API 密钥。

In [ ]:

Copied!

import openai

openai.api_key = "sk-"
import openai

openai.api_key = "sk-"

要使用 Milvus 向量数据库，请指定您的 Milvus 服务器 URI（可选添加 TOKEN）。要启动 Milvus 服务器，您可以按照 Milvus 安装指南进行设置，或直接免费试用 Zilliz Cloud。

In [ ]:

Copied!

URI = "./milvus_filter_demo.db"  # Use Milvus-Lite for demo purpose
# TOKEN = ""
URI = "./milvus_filter_demo.db"  # Use Milvus-Lite for demo purpose
# TOKEN = ""

准备数据

在本示例中，我们将使用几本标题相似或相同但元数据（作者、体裁和出版年份）不同的书籍作为样本数据。这有助于演示 Milvus 如何基于向量相似性和元数据属性来筛选和检索文档。

In [ ]:

Copied!





from llama_index.core.schema import TextNode

nodes = [
    TextNode(
        text="Life: A User's Manual",
        metadata={
            "author": "Georges Perec",
            "genre": "Postmodern Fiction",
            "year": 1978,
        },
    ),
    TextNode(
        text="Life and Fate",
        metadata={
            "author": "Vasily Grossman",
            "genre": "Historical Fiction",
            "year": 1980,
        },
    ),
    TextNode(
        text="Life",
        metadata={
            "author": "Keith Richards",
            "genre": "Memoir",
            "year": 2010,
        },
    ),
    TextNode(
        text="The Life",
        metadata={
            "author": "Malcolm Knox",
            "genre": "Literary Fiction",
            "year": 2011,
        },
    ),
]
from llama_index.core.schema import TextNode

nodes = [
    TextNode(
        text="Life: A User's Manual",
        metadata={
            "author": "Georges Perec",
            "genre": "Postmodern Fiction",
            "year": 1978,
        },
    ),
    TextNode(
        text="Life and Fate",
        metadata={
            "author": "Vasily Grossman",
            "genre": "Historical Fiction",
            "year": 1980,
        },
    ),
    TextNode(
        text="Life",
        metadata={
            "author": "Keith Richards",
            "genre": "Memoir",
            "year": 2010,
        },
    ),
    TextNode(
        text="The Life",
        metadata={
            "author": "Malcolm Knox",
            "genre": "Literary Fiction",
            "year": 2011,
        },
    ),
]

构建索引¶

在本节中，我们将使用默认的嵌入模型（OpenAI的text-embedding-ada-002）将示例数据存储到Milvus中。标题将被转换为文本嵌入并存储在稠密嵌入字段中，而所有元数据将存储在标量字段中。

In [ ]:

Copied!





from llama_index.vector_stores.milvus import MilvusVectorStore
from llama_index.core import StorageContext, VectorStoreIndex


vector_store = MilvusVectorStore(
    uri=URI,
    # token=TOKEN,
    collection_name="test_filter_collection",  # Change collection name here
    dim=1536,  # Vector dimension depends on the embedding model
    overwrite=True,  # Drop collection if exists
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex(nodes, storage_context=storage_context)
from llama_index.vector_stores.milvus import MilvusVectorStore
from llama_index.core import StorageContext, VectorStoreIndex


vector_store = MilvusVectorStore(
    uri=URI,
    # token=TOKEN,
    collection_name="test_filter_collection",  # Change collection name here
    dim=1536,  # Vector dimension depends on the embedding model
    overwrite=True,  # Drop collection if exists
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex(nodes, storage_context=storage_context)

2025-04-22 08:31:09,871 [DEBUG][_create_connection]: Created new connection using: 19675caa8f894772b3db175b65d0063a (async_milvus_client.py:547)

元数据过滤器¶

本节将介绍如何将LlamaIndex内置的元数据过滤器和条件应用于Milvus搜索。

定义元数据过滤器

In [ ]:

Copied!





from llama_index.core.vector_stores import (
    MetadataFilter,
    MetadataFilters,
    FilterOperator,
)

filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="year", value=2000, operator=FilterOperator.GT
        )  # year > 2000
    ]
)
from llama_index.core.vector_stores import (
    MetadataFilter,
    MetadataFilters,
    FilterOperator,
)

filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="year", value=2000, operator=FilterOperator.GT
        )  # year > 2000
    ]
)

通过筛选条件从向量存储中检索

In [ ]:

Copied!





retriever = index.as_retriever(filters=filters, similarity_top_k=5)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")
retriever = index.as_retriever(filters=filters, similarity_top_k=5)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")

The Life
{'author': 'Malcolm Knox', 'genre': 'Literary Fiction', 'year': 2011}


Life
{'author': 'Keith Richards', 'genre': 'Memoir', 'year': 2010}

多重元数据过滤器¶

您还可以组合多个元数据过滤器来构建更复杂的查询。LlamaIndex 同时支持通过 AND 和 OR 条件连接过滤器，从而能根据文档的元数据属性实现更精准灵活的检索。

条件 AND（与）

尝试一个筛选示例：查找出版年份在 1979 年至 2010 年之间的书籍（具体条件为 1979 < 年份 ≤ 2010）：

In [ ]:

Copied!





from llama_index.core.vector_stores import FilterCondition

filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="year", value=1979, operator=FilterOperator.GT
        ),  # year > 1979
        MetadataFilter(
            key="year", value=2010, operator=FilterOperator.LTE
        ),  # year <= 2010
    ],
    condition=FilterCondition.AND,
)

retriever = index.as_retriever(filters=filters, similarity_top_k=5)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")
from llama_index.core.vector_stores import FilterCondition

filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="year", value=1979, operator=FilterOperator.GT
        ),  # year > 1979
        MetadataFilter(
            key="year", value=2010, operator=FilterOperator.LTE
        ),  # year <= 2010
    ],
    condition=FilterCondition.AND,
)

retriever = index.as_retriever(filters=filters, similarity_top_k=5)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")

Life and Fate
{'author': 'Vasily Grossman', 'genre': 'Historical Fiction', 'year': 1980}


Life
{'author': 'Keith Richards', 'genre': 'Memoir', 'year': 2010}

条件 OR（或）

尝试另一个示例，筛选出由乔治·佩雷克（Georges Perec）或基思·理查兹（Keith Richards）撰写的书籍：

In [ ]:

Copied!





filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="author", value="Georges Perec", operator=FilterOperator.EQ
        ),  # author is Georges Perec
        MetadataFilter(
            key="author", value="Keith Richards", operator=FilterOperator.EQ
        ),  # author is Keith Richards
    ],
    condition=FilterCondition.OR,
)

retriever = index.as_retriever(filters=filters, similarity_top_k=5)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")
filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="author", value="Georges Perec", operator=FilterOperator.EQ
        ),  # author is Georges Perec
        MetadataFilter(
            key="author", value="Keith Richards", operator=FilterOperator.EQ
        ),  # author is Keith Richards
    ],
    condition=FilterCondition.OR,
)

retriever = index.as_retriever(filters=filters, similarity_top_k=5)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")

Life
{'author': 'Keith Richards', 'genre': 'Memoir', 'year': 2010}


Life: A User's Manual
{'author': 'Georges Perec', 'genre': 'Postmodern Fiction', 'year': 1978}

使用 Milvus 的关键字参数¶

除了内置的过滤功能外，您还可以通过 string_expr 关键字参数使用 Milvus 的原生过滤表达式。这使您能够在搜索操作期间直接将特定过滤表达式传递给 Milvus，从而突破标准元数据过滤的限制，访问 Milvus 的高级过滤功能。

Milvus 提供强大而灵活的过滤选项，可实现向量数据的精准查询：

基础运算符：比较运算符、范围过滤器、算术运算符和逻辑运算符
过滤表达式模板：针对常见过滤场景的预定义模式
专用运算符：针对 JSON 或数组字段的数据类型特定运算符

有关 Milvus 过滤表达式的完整文档和示例，请参阅 Milvus 过滤功能的官方文档。

In [ ]:

Copied!





retriever = index.as_retriever(
    vector_store_kwargs={
        "string_expr": "genre like '%Fiction'",
    },
    similarity_top_k=5,
)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")
retriever = index.as_retriever(
    vector_store_kwargs={
        "string_expr": "genre like '%Fiction'",
    },
    similarity_top_k=5,
)
result_nodes = retriever.retrieve("Books about life")
for node in result_nodes:
    print(node.text)
    print(node.metadata)
    print("\n")

The Life
{'author': 'Malcolm Knox', 'genre': 'Literary Fiction', 'year': 2011}


Life and Fate
{'author': 'Vasily Grossman', 'genre': 'Historical Fiction', 'year': 1980}


Life: A User's Manual
{'author': 'Georges Perec', 'genre': 'Postmodern Fiction', 'year': 1978}