Google Vertex AI 向量搜索¶

本笔记本演示如何使用与 Google Cloud Vertex AI Vector Search 向量数据库相关的功能。

Google Vertex AI 向量搜索（原名为Vertex AI Matching Engine）提供业界领先的大规模低延迟向量数据库服务。这类向量数据库通常被称为向量相似度匹配或近似最近邻（ANN）服务。

注意：LlamaIndex要求Vertex AI向量搜索终端节点和已部署的索引必须预先创建。创建空索引可能需要长达1分钟时间，而将索引部署到终端节点可能耗时长达30分钟。

如需了解如何创建索引，请参阅章节创建索引并部署至终端节点
若您已有部署好的索引，请直接跳转至从文本创建向量存储

安装¶

如果您在 Colab 上打开此 Notebook，可能需要先安装 LlamaIndex 🦙。

In [ ]:

! pip install llama-index llama-index-vector-stores-vertexaivectorsearch llama-index-llms-vertex

! pip install llama-index llama-index-vector-stores-vertexaivectorsearch llama-index-llms-vertex

创建索引并部署至终端节点¶

• 本节演示如何创建新索引并将其部署至终端节点。

In [ ]:

# TODO : Set values as per your requirements

# Project and Storage Constants
PROJECT_ID = "[your_project_id]"
REGION = "[your_region]"
GCS_BUCKET_NAME = "[your_gcs_bucket]"
GCS_BUCKET_URI = f"gs://{GCS_BUCKET_NAME}"

# The number of dimensions for the textembedding-gecko@003 is 768
# If other embedder is used, the dimensions would probably need to change.
VS_DIMENSIONS = 768

# Vertex AI Vector Search Index configuration
# parameter description here
# https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.MatchingEngineIndex#google_cloud_aiplatform_MatchingEngineIndex_create_tree_ah_index
VS_INDEX_NAME = "llamaindex-doc-index"  # @param {type:"string"}
VS_INDEX_ENDPOINT_NAME = "llamaindex-doc-endpoint"  # @param {type:"string"}

# TODO : Set values as per your requirements # Project and Storage Constants PROJECT_ID = "[your_project_id]" REGION = "[your_region]" GCS_BUCKET_NAME = "[your_gcs_bucket]" GCS_BUCKET_URI = f"gs://{GCS_BUCKET_NAME}" # The number of dimensions for the textembedding-gecko@003 is 768 # If other embedder is used, the dimensions would probably need to change. VS_DIMENSIONS = 768 # Vertex AI Vector Search Index configuration # parameter description here # https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.MatchingEngineIndex#google_cloud_aiplatform_MatchingEngineIndex_create_tree_ah_index VS_INDEX_NAME = "llamaindex-doc-index" # @param {type:"string"} VS_INDEX_ENDPOINT_NAME = "llamaindex-doc-endpoint" # @param {type:"string"}

In [ ]:

from google.cloud import aiplatform

aiplatform.init(project=PROJECT_ID, location=REGION)

from google.cloud import aiplatform aiplatform.init(project=PROJECT_ID, location=REGION)

创建云存储桶¶

In [ ]:

# Create a bucket.
! gsutil mb -l $REGION -p $PROJECT_ID $GCS_BUCKET_URI

# Create a bucket. ! gsutil mb -l $REGION -p $PROJECT_ID $GCS_BUCKET_URI

创建空索引¶

注意： 创建索引时必须指定 "index_update_method" - BATCH_UPDATE 或 STREAM_UPDATE

批处理索引适用于需要定期批量更新索引的场景，比如处理按周或按月存储的数据系统。

流式索引则适用于需要实时更新索引数据的场景，例如当书店有新库存时希望尽快在线展示。

选择哪种类型非常重要，因为两者的设置和要求各不相同。

更多关于索引配置的详细信息，请参阅官方文档和API参考

In [ ]:

# NOTE : This operation can take upto 30 seconds

# check if index exists
index_names = [
    index.resource_name
    for index in aiplatform.MatchingEngineIndex.list(
        filter=f"display_name={VS_INDEX_NAME}"
    )
]

if len(index_names) == 0:
    print(f"Creating Vector Search index {VS_INDEX_NAME} ...")
    vs_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
        display_name=VS_INDEX_NAME,
        dimensions=VS_DIMENSIONS,
        distance_measure_type="DOT_PRODUCT_DISTANCE",
        shard_size="SHARD_SIZE_SMALL",
        index_update_method="STREAM_UPDATE",  # allowed values BATCH_UPDATE , STREAM_UPDATE
    )
    print(
        f"Vector Search index {vs_index.display_name} created with resource name {vs_index.resource_name}"
    )
else:
    vs_index = aiplatform.MatchingEngineIndex(index_name=index_names[0])
    print(
        f"Vector Search index {vs_index.display_name} exists with resource name {vs_index.resource_name}"
    )

# NOTE : This operation can take upto 30 seconds # check if index exists index_names = [ index.resource_name for index in aiplatform.MatchingEngineIndex.list( filter=f"display_name={VS_INDEX_NAME}" ) ] if len(index_names) == 0: print(f"Creating Vector Search index {VS_INDEX_NAME} ...") vs_index = aiplatform.MatchingEngineIndex.create_tree_ah_index( display_name=VS_INDEX_NAME, dimensions=VS_DIMENSIONS, distance_measure_type="DOT_PRODUCT_DISTANCE", shard_size="SHARD_SIZE_SMALL", index_update_method="STREAM_UPDATE", # allowed values BATCH_UPDATE , STREAM_UPDATE ) print( f"Vector Search index {vs_index.display_name} created with resource name {vs_index.resource_name}" ) else: vs_index = aiplatform.MatchingEngineIndex(index_name=index_names[0]) print( f"Vector Search index {vs_index.display_name} exists with resource name {vs_index.resource_name}" )

创建端点¶

要使用索引，您需要创建一个索引端点。它作为服务器实例运行，接收针对您索引的查询请求。端点可以是公共端点或私有端点。

下面我们来创建一个公共端点。

In [ ]:

endpoint_names = [
    endpoint.resource_name
    for endpoint in aiplatform.MatchingEngineIndexEndpoint.list(
        filter=f"display_name={VS_INDEX_ENDPOINT_NAME}"
    )
]

if len(endpoint_names) == 0:
    print(
        f"Creating Vector Search index endpoint {VS_INDEX_ENDPOINT_NAME} ..."
    )
    vs_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
        display_name=VS_INDEX_ENDPOINT_NAME, public_endpoint_enabled=True
    )
    print(
        f"Vector Search index endpoint {vs_endpoint.display_name} created with resource name {vs_endpoint.resource_name}"
    )
else:
    vs_endpoint = aiplatform.MatchingEngineIndexEndpoint(
        index_endpoint_name=endpoint_names[0]
    )
    print(
        f"Vector Search index endpoint {vs_endpoint.display_name} exists with resource name {vs_endpoint.resource_name}"
    )

endpoint_names = [ endpoint.resource_name for endpoint in aiplatform.MatchingEngineIndexEndpoint.list( filter=f"display_name={VS_INDEX_ENDPOINT_NAME}" ) ] if len(endpoint_names) == 0: print( f"Creating Vector Search index endpoint {VS_INDEX_ENDPOINT_NAME} ..." ) vs_endpoint = aiplatform.MatchingEngineIndexEndpoint.create( display_name=VS_INDEX_ENDPOINT_NAME, public_endpoint_enabled=True ) print( f"Vector Search index endpoint {vs_endpoint.display_name} created with resource name {vs_endpoint.resource_name}" ) else: vs_endpoint = aiplatform.MatchingEngineIndexEndpoint( index_endpoint_name=endpoint_names[0] ) print( f"Vector Search index endpoint {vs_endpoint.display_name} exists with resource name {vs_endpoint.resource_name}" )

将索引部署至终端节点¶

通过索引终端节点，您可以通过指定唯一的已部署索引ID来部署索引。

注意：此操作最多可能需要30分钟完成。

In [ ]:

# check if endpoint exists
index_endpoints = [
    (deployed_index.index_endpoint, deployed_index.deployed_index_id)
    for deployed_index in vs_index.deployed_indexes
]

if len(index_endpoints) == 0:
    print(
        f"Deploying Vector Search index {vs_index.display_name} at endpoint {vs_endpoint.display_name} ..."
    )
    vs_deployed_index = vs_endpoint.deploy_index(
        index=vs_index,
        deployed_index_id=VS_INDEX_NAME,
        display_name=VS_INDEX_NAME,
        machine_type="e2-standard-16",
        min_replica_count=1,
        max_replica_count=1,
    )
    print(
        f"Vector Search index {vs_index.display_name} is deployed at endpoint {vs_deployed_index.display_name}"
    )
else:
    vs_deployed_index = aiplatform.MatchingEngineIndexEndpoint(
        index_endpoint_name=index_endpoints[0][0]
    )
    print(
        f"Vector Search index {vs_index.display_name} is already deployed at endpoint {vs_deployed_index.display_name}"
    )

# check if endpoint exists index_endpoints = [ (deployed_index.index_endpoint, deployed_index.deployed_index_id) for deployed_index in vs_index.deployed_indexes ] if len(index_endpoints) == 0: print( f"Deploying Vector Search index {vs_index.display_name} at endpoint {vs_endpoint.display_name} ..." ) vs_deployed_index = vs_endpoint.deploy_index( index=vs_index, deployed_index_id=VS_INDEX_NAME, display_name=VS_INDEX_NAME, machine_type="e2-standard-16", min_replica_count=1, max_replica_count=1, ) print( f"Vector Search index {vs_index.display_name} is deployed at endpoint {vs_deployed_index.display_name}" ) else: vs_deployed_index = aiplatform.MatchingEngineIndexEndpoint( index_endpoint_name=index_endpoints[0][0] ) print( f"Vector Search index {vs_index.display_name} is already deployed at endpoint {vs_deployed_index.display_name}" )

从文本创建向量存储¶

注意：如果您已拥有现成的 Vertex AI 向量搜索索引和端点，可通过以下代码进行分配：

In [ ]:

# TODO : replace 1234567890123456789 with your actual index ID
vs_index = aiplatform.MatchingEngineIndex(index_name="1234567890123456789")

# TODO : replace 1234567890123456789 with your actual endpoint ID
vs_endpoint = aiplatform.MatchingEngineIndexEndpoint(
    index_endpoint_name="1234567890123456789"
)

# TODO : replace 1234567890123456789 with your actual index ID vs_index = aiplatform.MatchingEngineIndex(index_name="1234567890123456789") # TODO : replace 1234567890123456789 with your actual endpoint ID vs_endpoint = aiplatform.MatchingEngineIndexEndpoint( index_endpoint_name="1234567890123456789" )

In [ ]:

# import modules needed
from llama_index.core import (
    StorageContext,
    Settings,
    VectorStoreIndex,
    SimpleDirectoryReader,
)
from llama_index.core.schema import TextNode
from llama_index.core.vector_stores.types import (
    MetadataFilters,
    MetadataFilter,
    FilterOperator,
)
from llama_index.llms.vertex import Vertex
from llama_index.embeddings.vertex import VertexTextEmbedding
from llama_index.vector_stores.vertexaivectorsearch import VertexAIVectorStore

# import modules needed from llama_index.core import ( StorageContext, Settings, VectorStoreIndex, SimpleDirectoryReader, ) from llama_index.core.schema import TextNode from llama_index.core.vector_stores.types import ( MetadataFilters, MetadataFilter, FilterOperator, ) from llama_index.llms.vertex import Vertex from llama_index.embeddings.vertex import VertexTextEmbedding from llama_index.vector_stores.vertexaivectorsearch import VertexAIVectorStore

从纯文本创建简易向量存储（无需元数据过滤器）¶

In [ ]:

# setup storage
vector_store = VertexAIVectorStore(
    project_id=PROJECT_ID,
    region=REGION,
    index_id=vs_index.resource_name,
    endpoint_id=vs_endpoint.resource_name,
    gcs_bucket_name=GCS_BUCKET_NAME,
)

# set storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# setup storage vector_store = VertexAIVectorStore( project_id=PROJECT_ID, region=REGION, index_id=vs_index.resource_name, endpoint_id=vs_endpoint.resource_name, gcs_bucket_name=GCS_BUCKET_NAME, ) # set storage context storage_context = StorageContext.from_defaults(vector_store=vector_store)

使用 Vertex AI Embeddings 作为嵌入模型¶

In [ ]:

# configure embedding model
embed_model = VertexTextEmbedding(
    model_name="textembedding-gecko@003",
    project=PROJECT_ID,
    location=REGION,
)

# setup the index/query process, ie the embedding model (and completion if used)
Settings.embed_model = embed_model

# configure embedding model embed_model = VertexTextEmbedding( model_name="textembedding-gecko@003", project=PROJECT_ID, location=REGION, ) # setup the index/query process, ie the embedding model (and completion if used) Settings.embed_model = embed_model

将向量及映射文本块添加至向量存储¶

In [ ]:

# Input texts
texts = [
    "The cat sat on",
    "the mat.",
    "I like to",
    "eat pizza for",
    "dinner.",
    "The sun sets",
    "in the west.",
]
nodes = [
    TextNode(text=text, embedding=embed_model.get_text_embedding(text))
    for text in texts
]

vector_store.add(nodes)

# Input texts texts = [ "The cat sat on", "the mat.", "I like to", "eat pizza for", "dinner.", "The sun sets", "in the west.", ] nodes = [ TextNode(text=text, embedding=embed_model.get_text_embedding(text)) for text in texts ] vector_store.add(nodes)

执行相似性搜索¶

In [ ]:

# define index from vector store
index = VectorStoreIndex.from_vector_store(
    vector_store=vector_store, embed_model=embed_model
)
retriever = index.as_retriever()

# define index from vector store index = VectorStoreIndex.from_vector_store( vector_store=vector_store, embed_model=embed_model ) retriever = index.as_retriever()

In [ ]:

response = retriever.retrieve("pizza")
for row in response:
    print(f"Score: {row.get_score():.3f} Text: {row.get_text()}")

response = retriever.retrieve("pizza") for row in response: print(f"Score: {row.get_score():.3f} Text: {row.get_text()}")

Score: 0.703 Text: eat pizza for
Score: 0.626 Text: dinner.

添加带有元数据属性的文档并使用过滤器¶

In [ ]:

# Input text with metadata
records = [
    {
        "description": "A versatile pair of dark-wash denim jeans."
        "Made from durable cotton with a classic straight-leg cut, these jeans"
        " transition easily from casual days to dressier occasions.",
        "price": 65.00,
        "color": "blue",
        "season": ["fall", "winter", "spring"],
    },
    {
        "description": "A lightweight linen button-down shirt in a crisp white."
        " Perfect for keeping cool with breathable fabric and a relaxed fit.",
        "price": 34.99,
        "color": "white",
        "season": ["summer", "spring"],
    },
    {
        "description": "A soft, chunky knit sweater in a vibrant forest green. "
        "The oversized fit and cozy wool blend make this ideal for staying warm "
        "when the temperature drops.",
        "price": 89.99,
        "color": "green",
        "season": ["fall", "winter"],
    },
    {
        "description": "A classic crewneck t-shirt in a soft, heathered blue. "
        "Made from comfortable cotton jersey, this t-shirt is a wardrobe essential "
        "that works for every season.",
        "price": 19.99,
        "color": "blue",
        "season": ["fall", "winter", "summer", "spring"],
    },
    {
        "description": "A flowing midi-skirt in a delicate floral print. "
        "Lightweight and airy, this skirt adds a touch of feminine style "
        "to warmer days.",
        "price": 45.00,
        "color": "white",
        "season": ["spring", "summer"],
    },
]

nodes = []
for record in records:
    text = record.pop("description")
    embedding = embed_model.get_text_embedding(text)
    metadata = {**record}
    nodes.append(TextNode(text=text, embedding=embedding, metadata=metadata))

vector_store.add(nodes)

# Input text with metadata records = [ { "description": "A versatile pair of dark-wash denim jeans." "Made from durable cotton with a classic straight-leg cut, these jeans" " transition easily from casual days to dressier occasions.", "price": 65.00, "color": "blue", "season": ["fall", "winter", "spring"], }, { "description": "A lightweight linen button-down shirt in a crisp white." " Perfect for keeping cool with breathable fabric and a relaxed fit.", "price": 34.99, "color": "white", "season": ["summer", "spring"], }, { "description": "A soft, chunky knit sweater in a vibrant forest green. " "The oversized fit and cozy wool blend make this ideal for staying warm " "when the temperature drops.", "price": 89.99, "color": "green", "season": ["fall", "winter"], }, { "description": "A classic crewneck t-shirt in a soft, heathered blue. " "Made from comfortable cotton jersey, this t-shirt is a wardrobe essential " "that works for every season.", "price": 19.99, "color": "blue", "season": ["fall", "winter", "summer", "spring"], }, { "description": "A flowing midi-skirt in a delicate floral print. " "Lightweight and airy, this skirt adds a touch of feminine style " "to warmer days.", "price": 45.00, "color": "white", "season": ["spring", "summer"], }, ] nodes = [] for record in records: text = record.pop("description") embedding = embed_model.get_text_embedding(text) metadata = {**record} nodes.append(TextNode(text=text, embedding=embedding, metadata=metadata)) vector_store.add(nodes)

执行带过滤条件的相似性搜索¶

In [ ]:

# define index from vector store
index = VectorStoreIndex.from_vector_store(
    vector_store=vector_store, embed_model=embed_model
)

# define index from vector store index = VectorStoreIndex.from_vector_store( vector_store=vector_store, embed_model=embed_model )

In [ ]:

# simple similarity search without filter
retriever = index.as_retriever()
response = retriever.retrieve("pants")

for row in response:
    print(f"Text: {row.get_text()}")
    print(f"   Score: {row.get_score():.3f}")
    print(f"   Metadata: {row.metadata}")

# simple similarity search without filter retriever = index.as_retriever() response = retriever.retrieve("pants") for row in response: print(f"Text: {row.get_text()}") print(f" Score: {row.get_score():.3f}") print(f" Metadata: {row.metadata}")

Text: A pair of well-tailored dress pants in a neutral grey. Made from a wrinkle-resistant blend, these pants look sharp and professional for workwear or formal occasions.
   Score: 0.669
   Metadata: {'price': 69.99, 'color': 'grey', 'season': ['fall', 'winter', 'summer', 'spring']}
Text: A pair of tailored black trousers in a comfortable stretch fabric. Perfect for work or dressier events, these trousers provide a sleek, polished look.
   Score: 0.642
   Metadata: {'price': 59.99, 'color': 'black', 'season': ['fall', 'winter', 'spring']}

In [ ]:

# similarity search with text filter
filters = MetadataFilters(filters=[MetadataFilter(key="color", value="blue")])
retriever = index.as_retriever(filters=filters, similarity_top_k=3)
response = retriever.retrieve("denims")

for row in response:
    print(f"Text: {row.get_text()}")
    print(f"   Score: {row.get_score():.3f}")
    print(f"   Metadata: {row.metadata}")

# similarity search with text filter filters = MetadataFilters(filters=[MetadataFilter(key="color", value="blue")]) retriever = index.as_retriever(filters=filters, similarity_top_k=3) response = retriever.retrieve("denims") for row in response: print(f"Text: {row.get_text()}") print(f" Score: {row.get_score():.3f}") print(f" Metadata: {row.metadata}")

Text: A versatile pair of dark-wash denim jeans.Made from durable cotton with a classic straight-leg cut, these jeans transition easily from casual days to dressier occasions.
   Score: 0.704
   Metadata: {'price': 65.0, 'color': 'blue', 'season': ['fall', 'winter', 'spring']}
Text: A denim jacket with a faded wash and distressed details. This wardrobe staple adds a touch of effortless cool to any outfit.
   Score: 0.667
   Metadata: {'price': 79.99, 'color': 'blue', 'season': ['fall', 'spring', 'summer']}

In [ ]:

# similarity search with text and numeric filter
filters = MetadataFilters(
    filters=[
        MetadataFilter(key="color", value="blue"),
        MetadataFilter(key="price", operator=FilterOperator.GT, value=70.0),
    ]
)
retriever = index.as_retriever(filters=filters, similarity_top_k=3)
response = retriever.retrieve("denims")

for row in response:
    print(f"Text: {row.get_text()}")
    print(f"   Score: {row.get_score():.3f}")
    print(f"   Metadata: {row.metadata}")

# similarity search with text and numeric filter filters = MetadataFilters( filters=[ MetadataFilter(key="color", value="blue"), MetadataFilter(key="price", operator=FilterOperator.GT, value=70.0), ] ) retriever = index.as_retriever(filters=filters, similarity_top_k=3) response = retriever.retrieve("denims") for row in response: print(f"Text: {row.get_text()}") print(f" Score: {row.get_score():.3f}") print(f" Metadata: {row.metadata}")

Text: A denim jacket with a faded wash and distressed details. This wardrobe staple adds a touch of effortless cool to any outfit.
   Score: 0.667
   Metadata: {'price': 79.99, 'color': 'blue', 'season': ['fall', 'spring', 'summer']}

使用 Vertex AI 向量搜索和 Gemini Pro 解析、索引及查询 PDF 文档¶

In [ ]:

! mkdir -p ./data/arxiv/
! wget 'https://arxiv.org/pdf/1706.03762.pdf' -O ./data/arxiv/test.pdf

! mkdir -p ./data/arxiv/ ! wget 'https://arxiv.org/pdf/1706.03762.pdf' -O ./data/arxiv/test.pdf

E0501 00:56:50.842446801  266241 backup_poller.cc:127]                 Run client channel backup poller: UNKNOWN:pollset_work {created_time:"2024-05-01T00:56:50.841935606+00:00", children:[UNKNOWN:Bad file descriptor {created_time:"2024-05-01T00:56:50.841810434+00:00", errno:9, os_error:"Bad file descriptor", syscall:"epoll_wait"}]}
--2024-05-01 00:56:52--  https://arxiv.org/pdf/1706.03762.pdf
Resolving arxiv.org (arxiv.org)... 151.101.67.42, 151.101.195.42, 151.101.131.42, ...
Connecting to arxiv.org (arxiv.org)|151.101.67.42|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://arxiv.org/pdf/1706.03762 [following]
--2024-05-01 00:56:52--  http://arxiv.org/pdf/1706.03762
Connecting to arxiv.org (arxiv.org)|151.101.67.42|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2215244 (2.1M) [application/pdf]
Saving to: ‘./data/arxiv/test.pdf’

./data/arxiv/test.p 100%[===================>]   2.11M  --.-KB/s    in 0.07s   

2024-05-01 00:56:52 (31.5 MB/s) - ‘./data/arxiv/test.pdf’ saved [2215244/2215244]

In [ ]:

# load documents
documents = SimpleDirectoryReader("./data/arxiv/").load_data()
print(f"# of documents = {len(documents)}")

# load documents documents = SimpleDirectoryReader("./data/arxiv/").load_data() print(f"# of documents = {len(documents)}")

# of documents = 15

In [ ]:

# setup storage
vector_store = VertexAIVectorStore(
    project_id=PROJECT_ID,
    region=REGION,
    index_id=vs_index.resource_name,
    endpoint_id=vs_endpoint.resource_name,
    gcs_bucket_name=GCS_BUCKET_NAME,
)

# set storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# configure embedding model
embed_model = VertexTextEmbedding(
    model_name="textembedding-gecko@003",
    project=PROJECT_ID,
    location=REGION,
)

vertex_gemini = Vertex(
    model="gemini-pro",
    context_window=100000,
    temperature=0,
    additional_kwargs={},
)

# setup the index/query process, ie the embedding model (and completion if used)
Settings.llm = vertex_gemini
Settings.embed_model = embed_model

# setup storage vector_store = VertexAIVectorStore( project_id=PROJECT_ID, region=REGION, index_id=vs_index.resource_name, endpoint_id=vs_endpoint.resource_name, gcs_bucket_name=GCS_BUCKET_NAME, ) # set storage context storage_context = StorageContext.from_defaults(vector_store=vector_store) # configure embedding model embed_model = VertexTextEmbedding( model_name="textembedding-gecko@003", project=PROJECT_ID, location=REGION, ) vertex_gemini = Vertex( model="gemini-pro", context_window=100000, temperature=0, additional_kwargs={}, ) # setup the index/query process, ie the embedding model (and completion if used) Settings.llm = vertex_gemini Settings.embed_model = embed_model

In [ ]:

# define index from vector store
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# define index from vector store index = VectorStoreIndex.from_documents( documents, storage_context=storage_context )

In [ ]:

query_engine = index.as_query_engine()

query_engine = index.as_query_engine()

In [ ]:

response = query_engine.query(
    "who are the authors of paper Attention is All you need?"
)

print(f"Response:")
print("-" * 80)
print(response.response)
print("-" * 80)
print(f"Source Documents:")
print("-" * 80)
for source in response.source_nodes:
    print(f"Sample Text: {source.text[:50]}")
    print(f"Relevance score: {source.get_score():.3f}")
    print(f"File Name: {source.metadata.get('file_name')}")
    print(f"Page #: {source.metadata.get('page_label')}")
    print(f"File Path: {source.metadata.get('file_path')}")
    print("-" * 80)

response = query_engine.query( "who are the authors of paper Attention is All you need?" ) print(f"Response:") print("-" * 80) print(response.response) print("-" * 80) print(f"Source Documents:") print("-" * 80) for source in response.source_nodes: print(f"Sample Text: {source.text[:50]}") print(f"Relevance score: {source.get_score():.3f}") print(f"File Name: {source.metadata.get('file_name')}") print(f"Page #: {source.metadata.get('page_label')}") print(f"File Path: {source.metadata.get('file_path')}") print("-" * 80)

Response:
--------------------------------------------------------------------------------
The authors of the paper "Attention Is All You Need" are:

* Ashish Vaswani
* Noam Shazeer
* Niki Parmar
* Jakob Uszkoreit
* Llion Jones
* Aidan N. Gomez
* Łukasz Kaiser
* Illia Polosukhin
--------------------------------------------------------------------------------
Source Documents:
--------------------------------------------------------------------------------
Sample Text: Provided proper attribution is provided, Google he
Relevance score: 0.720
File Name: test.pdf
Page #: 1
File Path: /home/jupyter/llama_index/docs/docs/examples/vector_stores/data/arxiv/test.pdf
--------------------------------------------------------------------------------
Sample Text: length nis smaller than the representation dimensi
Relevance score: 0.678
File Name: test.pdf
Page #: 7
File Path: /home/jupyter/llama_index/docs/docs/examples/vector_stores/data/arxiv/test.pdf
--------------------------------------------------------------------------------

---

（由于您没有提供需要翻译的具体Markdown文本内容，我无法执行翻译任务。请提供需要翻译的英文Markdown文档片段，我将严格按照您要求的规则进行专业翻译，并保持所有格式元素不变。）

清理资源¶

请在完成实验后删除 Vertex AI 向量搜索索引和索引端点，以避免产生额外费用。请注意，只要端点处于运行状态，就会持续计费。

⚠️ 注意：启用 `CLEANUP_RESOURCES` 标志将删除向量搜索索引、索引端点及云存储桶。请谨慎执行此操作。

In [ ]:

CLEANUP_RESOURCES = False

CLEANUP_RESOURCES = False

• 卸载索引与删除索引端点

In [ ]:

if CLEANUP_RESOURCES:
    print(
        f"Undeploying all indexes and deleting the index endpoint {vs_endpoint.display_name}"
    )
    vs_endpoint.undeploy_all()
    vs_endpoint.delete()

if CLEANUP_RESOURCES: print( f"Undeploying all indexes and deleting the index endpoint {vs_endpoint.display_name}" ) vs_endpoint.undeploy_all() vs_endpoint.delete()

• 删除索引

In [ ]:

if CLEANUP_RESOURCES:
    print(f"Deleting the index {vs_index.display_name}")
    vs_index.delete()

if CLEANUP_RESOURCES: print(f"Deleting the index {vs_index.display_name}") vs_index.delete()

• 删除云存储桶中的内容

In [ ]:

if CLEANUP_RESOURCES and "GCS_BUCKET_NAME" in globals():
    print(f"Deleting contents from the Cloud Storage bucket {GCS_BUCKET_NAME}")

    shell_output = ! gsutil du -ash gs://$GCS_BUCKET_NAME
    print(shell_output)
    print(
        f"Size of the bucket {GCS_BUCKET_NAME} before deleting = {' '.join(shell_output[0].split()[:2])}"
    )

    # uncomment below line to delete contents of the bucket
    # ! gsutil -m rm -r gs://$GCS_BUCKET_NAME

if CLEANUP_RESOURCES and "GCS_BUCKET_NAME" in globals(): print(f"Deleting contents from the Cloud Storage bucket {GCS_BUCKET_NAME}") shell_output = ! gsutil du -ash gs://$GCS_BUCKET_NAME print(shell_output) print( f"Size of the bucket {GCS_BUCKET_NAME} before deleting = {' '.join(shell_output[0].split()[:2])}" ) # uncomment below line to delete contents of the bucket # ! gsutil -m rm -r gs://$GCS_BUCKET_NAME