Timescale 向量存储（PostgreSQL）¶

本笔记本展示如何使用 Postgres 向量存储 TimescaleVector 来存储和查询向量嵌入。

什么是 Timescale Vector？¶

Timescale Vector 是专为 AI 应用打造的 PostgreSQL++ 数据库。

Timescale Vector 让您能够在 PostgreSQL 中高效存储和查询数百万个向量嵌入：

• 通过受 DiskANN 启发的索引算法增强 pgvector，实现对百万级向量更快更精确的相似性搜索
• 通过自动化的时间分区和索引实现基于时间的高效向量搜索
• 提供熟悉的 SQL 接口用于查询向量嵌入和关系型数据

Timescale Vector 可伴随您从概念验证到生产环境：

• 通过单一数据库存储关系型元数据、向量嵌入和时间序列数据，简化运维
• 基于坚如磐石的 PostgreSQL 基础架构，具备流式备份与复制、高可用性和行级安全等企业级特性
• 企业级安全与合规保障，提供无忧体验

如何使用 Timescale Vector¶

Timescale Vector 可在云端 PostgreSQL 平台 Timescale 上使用（目前暂无自托管版本）。

LlamaIndex 用户可享受 Timescale Vector 90 天免费试用。

• 开始使用：注册 Timescale，创建新数据库后即可按照本笔记本操作！
• 查看 Timescale Vector 技术解析博客了解详细信息和性能基准
• 参阅 安装说明 获取在 Python 中使用 Timescale Vector 的更多细节

0. 环境配置¶

让我们导入本笔记本所需的所有依赖项。

如果您在 Colab 上打开此 Notebook，可能需要安装 LlamaIndex 🦙。

In [ ]:

%pip install llama-index-embeddings-openai
%pip install llama-index-vector-stores-timescalevector

%pip install llama-index-embeddings-openai %pip install llama-index-vector-stores-timescalevector

In [ ]:

!pip install llama-index

!pip install llama-index

In [ ]:

# import logging
# import sys

# Uncomment to see debug logs
# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

import timescale_vector
from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core import VectorStoreIndex
from llama_index.vector_stores.timescalevector import TimescaleVectorStore
from llama_index.core.vector_stores import VectorStoreQuery, MetadataFilters
import textwrap
import openai

# import logging # import sys # Uncomment to see debug logs # logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) # logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout)) import timescale_vector from llama_index.core import SimpleDirectoryReader, StorageContext from llama_index.core import VectorStoreIndex from llama_index.vector_stores.timescalevector import TimescaleVectorStore from llama_index.core.vector_stores import VectorStoreQuery, MetadataFilters import textwrap import openai

配置 OpenAI API 密钥¶

要为载入索引的文档创建嵌入向量，请按以下步骤配置您的 OpenAI API 密钥：

In [ ]:

# Get openAI api key by reading local .env file
# The .env file should contain a line starting with `OPENAI_API_KEY=sk-`
import os
from dotenv import load_dotenv, find_dotenv

_ = load_dotenv(find_dotenv())

# OR set it explicitly
# import os
# os.environ["OPENAI_API_KEY"] = "<your key>"
openai.api_key = os.environ["OPENAI_API_KEY"]

# Get openAI api key by reading local .env file # The .env file should contain a line starting with `OPENAI_API_KEY=sk-` import os from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) # OR set it explicitly # import os # os.environ["OPENAI_API_KEY"] = "" openai.api_key = os.environ["OPENAI_API_KEY"]

创建 PostgreSQL 数据库并获取 Timescale 服务 URL¶

您需要一个服务 URL 来连接到 Timescale 数据库实例。

首先，在 Timescale 上启动一个新的云数据库（使用上方链接可免费注册）。

要连接到您的云端 PostgreSQL 数据库，您需要服务 URI。该 URI 可在创建新数据库后下载的速查表或 .env 文件中找到。

URI 的格式类似这样：postgres://tsdbadmin:<password>@<id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require

In [ ]:

# Get the service url by reading local .env file
# The .env file should contain a line starting with `TIMESCALE_SERVICE_URL=postgresql://`
import os
from dotenv import load_dotenv, find_dotenv

_ = load_dotenv(find_dotenv())

TIMESCALE_SERVICE_URL = os.environ["TIMESCALE_SERVICE_URL"]

# OR set it explicitly
# TIMESCALE_SERVICE_URL = "postgres://tsdbadmin:<password>@<id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require"

# Get the service url by reading local .env file # The .env file should contain a line starting with `TIMESCALE_SERVICE_URL=postgresql://` import os from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) TIMESCALE_SERVICE_URL = os.environ["TIMESCALE_SERVICE_URL"] # OR set it explicitly # TIMESCALE_SERVICE_URL = "postgres://tsdbadmin:@.tsdb.cloud.timescale.com:/tsdb?sslmode=require"

1. 使用 Timescale Vector 实现简单相似性搜索¶

下载数据¶

In [ ]:

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

加载文档¶

在本示例中，我们将使用 SimpleDirectoryReader 来加载存储在 paul_graham_essay 目录中的文档。

SimpleDirectoryReader 是 LlamaIndex 最常用的数据连接器之一，用于从目录中读取单个或多个文件。

In [ ]:

# load sample data from the data directory using a SimpleDirectoryReader
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
print("Document ID:", documents[0].doc_id)

# load sample data from the data directory using a SimpleDirectoryReader documents = SimpleDirectoryReader("./data/paul_graham").load_data() print("Document ID:", documents[0].doc_id)

Document ID: 740ce1a1-4d95-40cc-b7f7-6d2874620a53

使用 TimescaleVectorStore 创建向量存储索引¶

接下来，为了执行相似性搜索，我们首先创建一个 TimescaleVector 向量存储来存储从文章内容生成的向量嵌入。TimescaleVectorStore 需要接收几个参数，包括我们之前加载的 service_url，以及用于存储向量的数据表名称 table_name。

然后我们基于先前加载的文档，在 Timescale 支撑的存储上创建一个向量存储索引。

In [ ]:

# Create a TimescaleVectorStore to store the documents
vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="paul_graham_essay",
)

# Create a new VectorStoreIndex using the TimescaleVectorStore
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# Create a TimescaleVectorStore to store the documents vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="paul_graham_essay", ) # Create a new VectorStoreIndex using the TimescaleVectorStore storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents( documents, storage_context=storage_context )

查询索引¶

现在我们已经将文档索引到 VectorStore 中，可以通过默认的 query_engine 来查询索引文档中的内容。

注意：您还可以配置查询引擎来调整返回的 top_k 最相似结果数量，以及通过元数据过滤器对结果进行筛选。更多细节请参阅配置标准查询设置章节。

In [ ]:

query_engine = index.as_query_engine()
response = query_engine.query("Did the author work at YC?")

query_engine = index.as_query_engine() response = query_engine.query("Did the author work at YC?")

In [ ]:

print(textwrap.fill(str(response), 100))

print(textwrap.fill(str(response), 100))

Yes, the author did work at YC.

In [ ]:

response = query_engine.query("What did the author work on before college?")

response = query_engine.query("What did the author work on before college?")

In [ ]:

print(textwrap.fill(str(response), 100))

print(textwrap.fill(str(response), 100))

Before college, the author worked on writing and programming. They wrote short stories and also
tried programming on the IBM 1401 computer using an early version of Fortran.

查询现有索引¶

在上面的示例中，我们从加载的文档创建了一个新的 Timescale Vector 向量存储和索引。接下来我们将探讨如何查询现有索引。我们只需要服务 URI 和要访问的表名即可。

In [ ]:

vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="paul_graham_essay",
)

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do before YC?")

vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="paul_graham_essay", ) index = VectorStoreIndex.from_vector_store(vector_store=vector_store) query_engine = index.as_query_engine() response = query_engine.query("What did the author do before YC?")

In [ ]:

print(textwrap.fill(str(response), 100))

print(textwrap.fill(str(response), 100))

Before YC, the author wrote all of YC's internal software in Arc. They also worked on HN and had
three projects: writing essays, working on YC, and working in Arc. However, they gradually stopped
working on Arc due to time constraints and the increasing dependence on it for infrastructure.

2. 使用 ANN 搜索索引加速查询¶

（注：这些索引属于 ANN 索引，与 LlamaIndex 中的索引概念不同）

通过在嵌入列上创建索引，可以加速相似性查询。建议在完成大部分数据导入后再执行此操作。

Timescale Vector 支持以下索引类型：

• timescale_vector_index：受 disk-ann 启发的图索引，用于快速相似性搜索（默认选项）
• pgvector 的 HNSW 索引：基于分层可导航小世界图的快速相似性搜索索引
• pgvector 的 IVFFLAT 索引：基于倒排文件的快速相似性搜索索引

重要说明：在 PostgreSQL 中，每张表对特定列只能建立一个索引。若需测试不同索引类型的性能，可通过以下方式实现：(1) 创建多张带不同索引的表；(2) 在同一表中创建多个向量列，并在每列上建立不同索引；(3) 删除并重建同一列上的索引后对比结果。

In [ ]:

# Instantiate the TimescaleVectorStore from part 1
vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="paul_graham_essay",
)

# Instantiate the TimescaleVectorStore from part 1 vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="paul_graham_essay", )

使用不带额外参数的 create_index() 函数将默认创建一个采用默认配置的 timescale_vector (DiskANN) 索引。

In [ ]:

# Create a timescale vector index (DiskANN)
vector_store.create_index()

# Create a timescale vector index (DiskANN) vector_store.create_index()

你也可以为索引指定参数。有关不同参数及其对性能影响的完整讨论，请参阅 Timescale Vector 文档。

In [ ]:

# drop old index
vector_store.drop_index()

# create new timescale vector index (DiskANN) with specified parameters
vector_store.create_index("tsv", max_alpha=1.0, num_neighbors=50)

# drop old index vector_store.drop_index() # create new timescale vector index (DiskANN) with specified parameters vector_store.create_index("tsv", max_alpha=1.0, num_neighbors=50)

Timescale Vector 同样支持 HNSW 和 ivfflat 索引类型：

In [ ]:

vector_store.drop_index()

# Create an HNSW index
# Note: You don't need to specify m and ef_construction parameters as we set smart defaults.
vector_store.create_index("hnsw", m=16, ef_construction=64)

vector_store.drop_index() # Create an HNSW index # Note: You don't need to specify m and ef_construction parameters as we set smart defaults. vector_store.create_index("hnsw", m=16, ef_construction=64)

In [ ]:

# Create an IVFFLAT index
# Note: You don't need to specify num_lists and num_records parameters as we set smart defaults.
vector_store.drop_index()
vector_store.create_index("ivfflat", num_lists=20, num_records=1000)

# Create an IVFFLAT index # Note: You don't need to specify num_lists and num_records parameters as we set smart defaults. vector_store.drop_index() vector_store.create_index("ivfflat", num_lists=20, num_records=1000)

我们通常推荐使用 timescale-vector 或 HNSW 索引。

In [ ]:

# drop the ivfflat index
vector_store.drop_index()
# Create a timescale vector index (DiskANN)
vector_store.create_index()

# drop the ivfflat index vector_store.drop_index() # Create a timescale vector index (DiskANN) vector_store.create_index()

3. 支持时间过滤的相似性搜索¶

Timescale Vector 的核心应用场景是实现高效的时间向量搜索。该功能通过自动按时间分区向量（及相关元数据）来实现，使您能够同时基于查询向量的相似性和时间范围进行高效检索。

基于时间的向量搜索功能适用于以下场景：

• 存储和检索大语言模型响应历史（例如聊天机器人）
• 查找与查询向量相似的最新嵌入（例如近期新闻）
• 将相似性搜索限定在相关时间范围内（例如针对知识库提出时间相关的问题）

为演示 TimescaleVector 的时间向量搜索功能，我们将以 TimescaleDB 的 git 日志历史作为样本数据集并进行提问。每个 git 提交条目都包含时间戳、自然语言消息和其他元数据（如作者、提交哈希值等）。

我们将展示如何创建带有时序 UUID 的节点，以及如何使用 TimescaleVector 向量存储执行带时间范围过滤器的相似性搜索。

从 git log CSV 文件中提取内容与元数据¶

首先将 git log csv 文件加载到 PostgreSQL 数据库中名为 timescale_commits 的新集合。

注意：由于这是演示，我们仅处理前 1000 条记录。实际应用中，您可以根据需要加载任意数量的记录。

In [ ]:

import pandas as pd
from pathlib import Path

file_path = Path("../data/csv/commit_history.csv")
# Read the CSV file into a DataFrame
df = pd.read_csv(file_path)

# Light data cleaning on CSV
df.dropna(inplace=True)
df = df.astype(str)
df = df[:1000]

import pandas as pd from pathlib import Path file_path = Path("../data/csv/commit_history.csv") # Read the CSV file into a DataFrame df = pd.read_csv(file_path) # Light data cleaning on CSV df.dropna(inplace=True) df = df.astype(str) df = df[:1000]

In [ ]:

# Take a look at the data in the csv (optional)
df.head()

# Take a look at the data in the csv (optional) df.head()

我们将定义一个辅助函数，用于根据节点的时间戳为其创建UUID及关联的向量嵌入。该函数将用于为每个git日志条目生成UUID。

重要说明：如果您正在处理文档/节点并希望为向量关联当前日期时间以实现基于时间的搜索，可以跳过此步骤。默认情况下，当节点被添加到Timescale Vector表中时，系统会自动生成UUID。但在我们的场景中，由于需要基于历史时间戳创建UUID，因此需要手动生成这些UUID。

In [ ]:

from timescale_vector import client


# Function to take in a date string in the past and return a uuid v1
def create_uuid(date_string: str):
    if date_string is None:
        return None
    time_format = "%a %b %d %H:%M:%S %Y %z"
    datetime_obj = datetime.strptime(date_string, time_format)
    uuid = client.uuid_from_time(datetime_obj)
    return str(uuid)

from timescale_vector import client # Function to take in a date string in the past and return a uuid v1 def create_uuid(date_string: str): if date_string is None: return None time_format = "%a %b %d %H:%M:%S %Y %z" datetime_obj = datetime.strptime(date_string, time_format) uuid = client.uuid_from_time(datetime_obj) return str(uuid)

In [ ]:

# Helper functions
from typing import List, Tuple


# Helper function to split name and email given an author string consisting of Name Lastname <email>
def split_name(input_string: str) -> Tuple[str, str]:
    if input_string is None:
        return None, None
    start = input_string.find("<")
    end = input_string.find(">")
    name = input_string[:start].strip()
    return name


from datetime import datetime, timedelta


def create_date(input_string: str) -> datetime:
    if input_string is None:
        return None
    # Define a dictionary to map month abbreviations to their numerical equivalents
    month_dict = {
        "Jan": "01",
        "Feb": "02",
        "Mar": "03",
        "Apr": "04",
        "May": "05",
        "Jun": "06",
        "Jul": "07",
        "Aug": "08",
        "Sep": "09",
        "Oct": "10",
        "Nov": "11",
        "Dec": "12",
    }

    # Split the input string into its components
    components = input_string.split()
    # Extract relevant information
    day = components[2]
    month = month_dict[components[1]]
    year = components[4]
    time = components[3]
    timezone_offset_minutes = int(
        components[5]
    )  # Convert the offset to minutes
    timezone_hours = timezone_offset_minutes // 60  # Calculate the hours
    timezone_minutes = (
        timezone_offset_minutes % 60
    )  # Calculate the remaining minutes
    # Create a formatted string for the timestamptz in PostgreSQL format
    timestamp_tz_str = (
        f"{year}-{month}-{day} {time}+{timezone_hours:02}{timezone_minutes:02}"
    )
    return timestamp_tz_str

# Helper functions from typing import List, Tuple # Helper function to split name and email given an author string consisting of Name Lastname def split_name(input_string: str) -> Tuple[str, str]: if input_string is None: return None, None start = input_string.find("<") end = input_string.find(">") name = input_string[:start].strip() return name from datetime import datetime, timedelta def create_date(input_string: str) -> datetime: if input_string is None: return None # Define a dictionary to map month abbreviations to their numerical equivalents month_dict = { "Jan": "01", "Feb": "02", "Mar": "03", "Apr": "04", "May": "05", "Jun": "06", "Jul": "07", "Aug": "08", "Sep": "09", "Oct": "10", "Nov": "11", "Dec": "12", } # Split the input string into its components components = input_string.split() # Extract relevant information day = components[2] month = month_dict[components[1]] year = components[4] time = components[3] timezone_offset_minutes = int( components[5] ) # Convert the offset to minutes timezone_hours = timezone_offset_minutes // 60 # Calculate the hours timezone_minutes = ( timezone_offset_minutes % 60 ) # Calculate the remaining minutes # Create a formatted string for the timestamptz in PostgreSQL format timestamp_tz_str = ( f"{year}-{month}-{day} {time}+{timezone_hours:02}{timezone_minutes:02}" ) return timestamp_tz_str

接下来，我们将定义一个函数来为每个 git 日志条目创建 TextNode。我们将使用之前定义的辅助函数 create_uuid()，基于时间戳为每个节点生成唯一标识符。同时利用 create_date() 和 split_name() 这两个辅助函数，从 git 日志条目中提取相关元数据并添加到节点中。

In [ ]:

from llama_index.core.schema import TextNode, NodeRelationship, RelatedNodeInfo


# Create a Node object from a single row of data
def create_node(row):
    record = row.to_dict()
    record_name = split_name(record["author"])
    record_content = (
        str(record["date"])
        + " "
        + record_name
        + " "
        + str(record["change summary"])
        + " "
        + str(record["change details"])
    )
    # Can change to TextNode as needed
    node = TextNode(
        id_=create_uuid(record["date"]),
        text=record_content,
        metadata={
            "commit": record["commit"],
            "author": record_name,
            "date": create_date(record["date"]),
        },
    )
    return node

from llama_index.core.schema import TextNode, NodeRelationship, RelatedNodeInfo # Create a Node object from a single row of data def create_node(row): record = row.to_dict() record_name = split_name(record["author"]) record_content = ( str(record["date"]) + " " + record_name + " " + str(record["change summary"]) + " " + str(record["change details"]) ) # Can change to TextNode as needed node = TextNode( id_=create_uuid(record["date"]), text=record_content, metadata={ "commit": record["commit"], "author": record_name, "date": create_date(record["date"]), }, ) return node

In [ ]:

nodes = [create_node(row) for _, row in df.iterrows()]

nodes = [create_node(row) for _, row in df.iterrows()]

接下来我们将为每个节点的内容创建向量嵌入，以便对与各节点关联的文本执行相似性搜索。我们将使用 OpenAIEmbedding 模型来生成这些嵌入向量。

In [ ]:

# Create embeddings for nodes
from llama_index.embeddings.openai import OpenAIEmbedding

embedding_model = OpenAIEmbedding()

for node in nodes:
    node_embedding = embedding_model.get_text_embedding(
        node.get_content(metadata_mode="all")
    )
    node.embedding = node_embedding

# Create embeddings for nodes from llama_index.embeddings.openai import OpenAIEmbedding embedding_model = OpenAIEmbedding() for node in nodes: node_embedding = embedding_model.get_text_embedding( node.get_content(metadata_mode="all") ) node.embedding = node_embedding

让我们检查集合中的第一个节点，看看它是什么样子的。

In [ ]:

print(nodes[0].get_content(metadata_mode="all"))

print(nodes[0].get_content(metadata_mode="all"))

commit: 44e41c12ab25e36c202f58e068ced262eadc8d16
author: Lakshmi Narayanan Sreethar
date: 2023-09-5 21:03:21+0850

Tue Sep 5 21:03:21 2023 +0530 Lakshmi Narayanan Sreethar Fix segfault in set_integer_now_func When an invalid function oid is passed to set_integer_now_func, it finds out that the function oid is invalid but before throwing the error, it calls ReleaseSysCache on an invalid tuple causing a segfault. Fixed that by removing the invalid call to ReleaseSysCache.  Fixes #6037

In [ ]:

print(nodes[0].get_embedding())

print(nodes[0].get_embedding())

[-0.005366453900933266, 0.0016374519327655435, 0.005981510039418936, -0.026256779208779335, -0.03944991156458855, 0.026299940422177315, -0.0200558640062809, -0.01252412423491478, -0.04241368919610977, -0.004758591763675213, 0.05639812350273132, 0.006578581873327494, 0.014833281747996807, 0.009509989991784096, 0.0009675443288870156, -0.013157163746654987, -0.002265996066853404, -0.017048921436071396, 0.006553404498845339, -0.00217068032361567, 0.009085564874112606, 0.011775985360145569, -0.02514895796775818, -0.002679630182683468, 0.0030608929228037596, -3.439458305365406e-05, -0.00363818253390491, -0.03939236328005791, 0.0016806137282401323, -0.01207092497497797, 0.01739421673119068, -0.02241537719964981, -0.01753808930516243, -0.023782167583703995, -0.01598426327109337, -0.02575322426855564, -0.016876274719834328, -0.006380756851285696, -0.0009149408433586359, 0.00704616867005825, -0.0013290246715769172, -0.009776154533028603, -0.013200325891375542, -0.024832438677549362, -0.0019404839258641005, 0.027220726013183594, -0.004765785299241543, -0.008553235791623592, -0.023120352998375893, 0.006920279935002327, 0.017739512026309967, 0.0166892409324646, -0.019408436492085457, 0.010207772254943848, 0.01595548912882805, 0.004783769138157368, 0.008855368942022324, 0.018084805458784103, -0.012603254057466984, -0.002003428293392062, -0.0008407564600929618, 0.00394211383536458, -0.018948042765259743, 0.005722539033740759, -0.004244246520102024, -0.011502627283334732, -0.000936971337068826, 0.006873521022498608, -0.0038593867793679237, 0.0003349537728354335, 0.02490437589585781, 0.022861381992697716, -0.013833366334438324, 0.005657796282321215, 0.027896929532289505, -0.020415544509887695, -0.007143282797187567, 0.014862056821584702, -0.00667569600045681, -0.020199736580252647, 0.01827184110879898, -0.0030698850750923157, -0.032975636422634125, 0.02595464698970318, -0.0014818893978372216, -0.004906061105430126, 0.01008548028767109, 0.009337342344224453, -0.009833703748881817, -0.0011680669849738479, 0.010653777979314327, -0.0006110096583142877, 0.016228847205638885, -0.010589035227894783, 0.0010997274657711387, 0.020300446078181267, 0.005715345498174429, 0.009862477891147137, -0.0015664147213101387, -0.009207856841385365, -0.013480877503752708, -0.01759563945233822, 0.007992131635546684, -0.012639221735298634, -0.016833113506436348, -0.01654536835849285, 0.009366116486489773, 0.004229859448969364, -0.0044168937020003796, -0.00028122629737481475, -0.028918424621224403, 0.030616123229265213, -0.017020147293806076, -0.02500508539378643, 0.01844448782503605, 0.00011554780940059572, 0.021278781816363335, -0.01503470353782177, -0.024760503321886063, -0.02408429980278015, 0.03734936937689781, 0.000861438165884465, 0.021365106105804443, -0.006740438751876354, 0.005557085387408733, -0.017005760222673416, -0.01831500232219696, -0.01458150427788496, -0.0207896139472723, -0.004100373946130276, 0.011214882135391235, 0.03228504955768585, 0.00543119665235281, 0.02251608669757843, 0.011373141780495644, 0.0207896139472723, 0.004032033961266279, 0.019768116995692253, -0.016329558566212654, -0.02755163423717022, -0.0001643296709517017, 0.04163677617907524, -0.02163846418261528, 0.019394047558307648, -0.028975974768400192, 0.040543343871831894, 0.006010284647345543, 0.009812122210860252, 0.024746114388108253, -0.027781831100583076, -0.0009360721451230347, 0.002836091909557581, -0.008733076974749565, 0.010754489339888096, -0.005380841437727213, 0.01586916483938694, 0.0003014584071934223, 0.006862730719149113, -0.033666227012872696, -0.01664607785642147, 0.001758844475261867, 0.0125528983771801, 0.0065066455863416195, 0.016228847205638885, 0.010466743260622025, 0.0251057967543602, -0.009215050376951694, -0.016027426347136497, 0.0116033386439085, 0.019667407497763634, 0.008905723690986633, 0.011517014354467392, -0.0036561666056513786, 0.02263118512928486, 0.027868153527379036, 0.02509140968322754, 0.011984600685536861, -0.016473431140184402, -0.013847753405570984, 0.01377581711858511, -0.010315677151083946, -0.0076612248085439205, 0.031076516956090927, 0.06526068598031998, -0.01297012995928526, -0.008610785007476807, 0.02340809814631939, 0.0038989519234746695, 0.009251018986105919, 0.003494309727102518, 0.009301373735070229, 0.003737095044925809, -0.03757956624031067, -0.0013029477559030056, -0.5865404605865479, -0.013444909825921059, 0.0021041391883045435, -0.004449265077710152, 0.0020052266772836447, 0.010776069946587086, 0.025695675984025, 0.01172563061118126, -0.02909107320010662, 0.015250512398779392, -0.014487987384200096, 0.04822615161538124, 0.014487987384200096, -0.030357152223587036, -0.01828622817993164, -0.00993441417813301, 0.010294096544384956, 0.0014764942461624742, 0.005524714011698961, -0.001750751631334424, -0.020386770367622375, 0.010977491736412048, -0.01082642562687397, -0.010092674754559994, 0.021091746166348457, -0.009229437448084354, 0.012013375759124756, -0.01284064445644617, -0.00015736083150841296, 0.009063984267413616, -0.013660718686878681, 0.0058807991445064545, 0.013955658301711082, 0.0028414870612323284, 0.046960070729255676, -0.01130120549350977, -0.04943468049168587, 0.006970635149627924, 0.0046219127252697945, 0.031335487961769104, -0.031306713819503784, -0.026357490569353104, 0.042902857065200806, 0.01129401195794344, -0.010941523127257824, -0.0027857364621013403, 0.00627644918859005, -0.023005254566669464, -0.0133226178586483, -0.05228336155414581, 0.025494253262877464, 0.016084974631667137, 0.014329726807773113, 0.03809750825166702, -0.025393541902303696, -0.02322106435894966, 0.006686486769467592, 0.01847326196730137, 0.02989676035940647, 0.0017372636357322335, -0.018530812114477158, 0.0418669730424881, -0.010092674754559994, 0.003927726298570633, -0.009876864962279797, -0.012322702445089817, 0.007977744564414024, 0.027177564799785614, 0.036255937069654465, 0.007841065526008606, 0.02255924977362156, 0.02076083980500698, 0.00024390929320361465, -0.01254570484161377, -0.005535504315048456, 0.014185854233801365, 0.01211408618837595, -0.020199736580252647, -0.02414184994995594, 0.006977829150855541, -0.010157416574656963, -0.025997808203101158, 0.017235957086086273, -0.008862562477588654, 0.017077697440981865, 0.02084716409444809, 0.0029583836439996958, -0.003523084335029125, 0.007387866266071796, -0.0035122937988489866, 0.01923578791320324, 0.010092674754559994, -0.0016896057641133666, 0.0016887065721675754, 0.025666899979114532, -0.0024979908484965563, -0.01455992367118597, 0.009351729415357113, -0.013631944544613361, -0.03240014612674713, -0.020573804154992104, -0.029004748910665512, -0.02096226066350937, -0.013495265506207943, 0.0216672383248806, 0.038011182099580765, -0.01664607785642147, -0.03078877180814743, 0.037234269082546234, 0.0005889791063964367, -0.0040392279624938965, -0.012365863658487797, -0.009862477891147137, -0.011783178895711899, -0.008862562477588654, -0.02922055870294571, 0.025407928973436356, 0.001157276565209031, 0.008589203469455242, -0.007941776886582375, 0.005686570890247822, 0.007190041244029999, 0.01579722948372364, -0.02337932400405407, 0.007003006525337696, -0.00768999895080924, 0.028645066544413567, -0.019710568711161613, -0.021307555958628654, -0.0257676113396883, 0.012660803273320198, -0.006607356481254101, 0.010459549725055695, 0.0007481383509002626, 0.02342248521745205, -0.007632450200617313, 0.02916300855576992, -0.02093348652124405, 0.035565346479415894, -0.011891083791851997, -0.02571006305515766, -0.0050067720003426075, -0.006441902834922075, -0.01040919404476881, -0.001439626794308424, -0.011502627283334732, -0.03838525339961052, -0.004352150950580835, 0.01746615394949913, -0.00197825045324862, -0.008661140687763691, 0.004237052984535694, -0.041377805173397064, 0.01595548912882805, -0.003506898647174239, -0.004805350210517645, -0.010229353792965412, -0.016372719779610634, 0.005852024536579847, -0.007006603758782148, 0.007790710311383009, 0.02512018382549286, -0.01458150427788496, 0.020429931581020355, -0.006862730719149113, -0.006783600896596909, -0.009898446500301361, 0.00603905925527215, -0.015279287472367287, -0.03827015310525894, -0.009409278631210327, 0.0021796722430735826, -0.011941439472138882, -0.009330148808658123, -0.010286902077496052, 0.01004231907427311, -0.023667069151997566, -0.007948970422148705, -0.013502459041774273, 0.00689150532707572, 0.028832102194428444, 0.02832854725420475, -0.0332346074283123, -0.012416219338774681, 0.009891252033412457, 0.017192795872688293, 0.01844448782503605, 0.0008421052480116487, 0.013560008257627487, 0.025292832404375076, -0.023954814299941063, 0.009912833571434021, -0.003154410282149911, -0.01086239330470562, 0.011509820818901062, 0.03752201795578003, -0.004481636453419924, -0.009013628587126732, 0.004283811431378126, 0.030299603939056396, 0.014164273627102375, 0.006959844846278429, 0.02920617163181305, -0.011977407149970531, -0.0028288981411606073, -0.023796554654836655, 0.001507966429926455, -0.008546042256057262, -0.019796893000602722, -0.021998144686222076, -0.000644280225969851, -0.014718183316290379, -0.013085227459669113, -0.005549891851842403, 0.008733076974749565, 0.042068395763635635, 0.00501756276935339, 0.01585477776825428, -0.02152336575090885, -0.01127243135124445, -0.005700958427041769, -0.003776659956201911, 0.028947200626134872, 0.004992384929209948, 0.0016985977999866009, -0.008143198676407337, 0.004729817155748606, 0.016444656997919083, 0.022372214123606682, 0.0038773708511143923, 0.0027857364621013403, 0.012365863658487797, 0.02819906175136566, 0.01549509633332491, 0.04822615161538124, -0.010222159326076508, 0.01845887489616871, -0.012236378155648708, 0.03939236328005791, -0.003064489923417568, -0.015552645549178123, 0.014200241304934025, 0.02571006305515766, -0.009466827847063541, -0.024501532316207886, -0.02660207450389862, 0.03274543955922127, -0.0028936408925801516, 0.0067907944321632385, 0.026213617995381355, -0.007431028410792351, 0.0007823081687092781, -0.004006856586784124, 0.02745092287659645, 0.02737898752093315, 0.014315339736640453, 0.004862899426370859, 0.010315677151083946, 0.01989760249853134, 0.041982073336839676, 0.01996953971683979, 0.011092590168118477, 0.001743558095768094, -0.01131559256464243, 0.015236125327646732, 0.005107482895255089, 0.018027257174253464, -0.014962767250835896, 0.0006712563335895538, 0.01573967933654785, -0.02673156000673771, -0.022156406193971634, -0.04419771209359169, 0.00019040661572944373, 0.020458707585930824, 0.011258043348789215, 0.01904875412583351, 0.015164189040660858, 0.010517098940908909, -0.021998144686222076, -0.039593786001205444, -0.01992637850344181, 0.04448545724153519, 0.011452271603047848, -0.0019728553015738726, -0.021120522171258926, 0.0043809255585074425, 0.0013056453317403793, -0.017753899097442627, -0.0200558640062809, -0.019005591049790382, 0.02409868687391281, 0.014804507605731487, 0.006682890001684427, 0.01008548028767109, -0.00481614051386714, 0.030616123229265213, -0.003010537475347519, -0.014790119603276253, -0.006780004128813744, -0.028055189177393913, 0.017940932884812355, -0.01424340344965458, 0.0034511478152126074, 0.04736291244626045, -0.012301120907068253, -0.016502205282449722, -0.0018685475224629045, -0.009999156929552555, -0.004460055846720934, 0.03487475588917732, -0.009790541604161263, -0.019422823563218117, -0.03389642387628555, 0.009991963393986225, 0.0016770168440416455, -0.005945541895925999, -0.0014899822417646646, 0.04517604783177376, 0.008596397936344147, -0.021134909242391586, -0.03263034299015999, -0.025407928973436356, 0.012257959693670273, 0.026127293705940247, 0.05098850652575493, 0.004111164249479771, 0.021149296313524246, -0.021278781816363335, 0.00727636506780982, -0.013380167074501514, -0.014171467162668705, 0.03737814351916313, -0.019653018563985825, 0.018861718475818634, 0.012991710565984249, -0.0012795684160664678, -0.016358332708477974, 0.032112400978803635, 0.017120858654379845, 0.032054852694272995, -0.006474274210631847, -0.01131559256464243, -0.014667828567326069, -0.012689577415585518, 0.01907752826809883, 0.022012531757354736, 0.01740860380232334, 0.0033450417686253786, 0.02322106435894966, 0.03893196955323219, 0.033666227012872696, 0.025566190481185913, -0.017825834453105927, -0.02571006305515766, 0.011258043348789215, 0.023465648293495178, 0.012315508909523487, -0.027235113084316254, 0.020415544509887695, 0.005945541895925999, 0.0029961501713842154, 0.02083277516067028, -0.004118357785046101, 0.012236378155648708, -0.012941354885697365, 0.008776238188147545, 0.009661056101322174, 0.009078371338546276, 0.0009558546589687467, -0.005672183819115162, 0.008869756013154984, 0.0011509821051731706, 0.021883046254515648, 0.008085649460554123, -0.012135667726397514, -0.027896929532289505, -0.03406906872987747, 0.027824992313981056, 0.029724113643169403, -0.02762356959283352, -0.023034028708934784, -0.027321437373757362, -0.02746530994772911, -0.02409868687391281, -0.016372719779610634, -0.0012382050044834614, -0.005837637465447187, 0.001647343160584569, 0.016257621347904205, -0.01657414250075817, 0.0007503863889724016, -0.02595464698970318, 0.01595548912882805, -0.0030051423236727715, -0.002246213611215353, -0.02493315003812313, 0.015610194765031338, 0.008546042256057262, -0.0008038890664465725, 0.03818383067846298, -0.004553572740405798, 0.00098912522662431, -0.0023703037295490503, -0.01996953971683979, -0.042931631207466125, -0.02591148391366005, -0.03473088517785072, 0.010941523127257824, 0.0009603506769053638, 0.021307555958628654, -0.008920111693441868, -0.011142945848405361, 0.022688735276460648, 0.016444656997919083, -0.006197319366037846, 0.020660128444433212, 0.016516592353582382, -0.01410672441124916, 0.006949054542928934, 0.014682215638458729, 0.01007109321653843, 0.009200663305819035, 0.009488408453762531, 0.023623907938599586, 0.028573131188750267, -0.005967122968286276, -0.014675022102892399, -0.013185938820242882, 0.008323038928210735, -0.0018883299781009555, 0.010193385183811188, 0.006582179106771946, -0.0028288981411606073, 0.02163846418261528, -0.005456374492496252, 0.012783095240592957, 0.01589794084429741, 0.030501024797558784, 0.0026544525753706694, 0.035392697900533676, 0.012610447593033314, -0.0026112906634807587, -0.023810941725969315, 0.0035212859511375427, -0.01579722948372364, 0.03263034299015999, -0.025465479120612144, 0.006096608471125364, 0.009207856841385365, -0.009790541604161263, -0.02926371991634369, -0.003773063188418746, 0.023868491873145103, -0.01750931516289711, 0.016171298921108246, 0.033004410564899445, -0.00790580827742815, -0.00728355860337615, 0.0021077359560877085, -0.017955319955945015, -0.0032533227931708097, -0.0004282462759874761, -0.006204512901604176, 0.014430438168346882, -0.03985275700688362, 0.01591232791543007, -0.03519127890467644, -0.03749324008822441, -0.025379154831171036, -0.019552309066057205, -0.008078455924987793, 0.042874082922935486, 0.018502037972211838, -0.010545873083174229, -0.009085564874112606, -0.026069745421409607, -0.0023127547465264797, 0.00957473274320364, -0.02929249405860901, -0.03487475588917732, -0.01132997963577509, 0.03188220411539078, 0.009107145480811596, 0.030644899234175682, -0.013459296897053719, 0.04224104434251785, -0.016789952293038368, 0.0027731475420296192, 0.002381094265729189, 0.01130120549350977, 0.008934498764574528, -0.012337089516222477, 0.007517351768910885, 0.0005318796029314399, -0.005711748730391264, -0.01535122375935316, 0.0017957119271159172, -0.004132745321840048, 0.02592587098479271, -0.026285553351044655, -0.0012876612599939108, -0.01759563945233822, -0.029839210212230682, -0.003095062915235758, -0.0052477591671049595, -0.015567032620310783, 0.01664607785642147, -0.028098350390791893, 0.009984769858419895, 0.04307550564408302, 0.0010817432776093483, 0.00710731465369463, 0.016041813418269157, 0.010437969118356705, -0.028573131188750267, 0.010704133659601212, 0.005229774862527847, 0.002433248097077012, 0.012229184620082378, -0.0018793379422277212, -0.006312417332082987, 0.01743737794458866, -0.016948211938142776, 0.009502795524895191, 0.017782673239707947, -0.00690589239820838, -0.010553067550063133, -0.01595548912882805, 0.020228510722517967, -0.028558744117617607, -0.012186023406684399, 0.024443982169032097, -0.03150813654065132, 0.006003091111779213, -0.03320583328604698, -0.024659791961312294, -0.013876527547836304, 0.007312333211302757, 0.00689869886264205, 0.0004842217604164034, 0.020991036668419838, -0.007797903846949339, -0.014437631703913212, 0.0003288841398898512, -0.01674678921699524, 0.02819906175136566, -0.006826762575656176, 0.03763711452484131, 0.03332093358039856, 0.0006676595658063889, -0.046960070729255676, 0.01986882835626602, 0.03254402056336403, 0.005783685017377138, 0.013142776675522327, 0.02589709684252739, 0.009905640035867691, 0.006765616592019796, -0.00791300181299448, -2.810014905207936e-07, -0.03228504955768585, -0.018904881551861763, 0.01572529226541519, 0.02008463814854622, 0.014732571318745613, -0.009279793128371239, -0.011768791824579239, -0.021998144686222076, -0.011135751381516457, -0.01209969911724329, 0.005643409211188555, -0.008459718897938728, -0.0033558320719748735, -0.012826256453990936, 0.03844280168414116, -0.01502031646668911, 0.009776154533028603, 0.020458707585930824, -0.009704218246042728, -0.012373057194054127, -0.021839885041117668, -0.030587349086999893, -0.005729732569307089, -0.026199230924248695, 0.04943468049168587, 0.0026490571908652782, -0.02658768743276596, 0.002050186973065138, -0.010668165050446987, -0.016200073063373566, -0.046413354575634, -0.008589203469455242, 0.04224104434251785, -0.019336499273777008, -0.009661056101322174, 0.01995515264570713, 0.013509652577340603, -0.0022803833708167076, -0.005157838575541973, -0.016890661790966988, -0.006258465349674225, -0.03591063991189003, 0.003055497771129012, 0.010761682875454426, 0.004546379204839468, -0.003618400078266859, 0.009402085095643997, -0.016473431140184402, -0.018832944333553314, -0.015624581836163998, -0.035392697900533676, 0.009862477891147137, -0.04411138966679573, 0.01038761343806982, -0.030213279649615288, -0.01329384371638298, 0.030414702370762825, 0.02084716409444809, -0.03421294316649437, 0.013444909825921059, 0.04914693534374237, -0.031134065240621567, 0.009286986663937569, 0.00023739006428513676, 0.012811869382858276, -0.03507617861032486, -0.0007841065526008606, -0.020286059007048607, -0.013027678243815899, 0.025249669328331947, -0.009028015658259392, -0.023839715868234634, -0.017164019867777824, 0.02319229021668434, -0.0019171045860275626, 0.01906314119696617, 0.0008443532860837877, 0.05933312699198723, 0.012596060521900654, 0.01253131777048111, -0.034673336893320084, -0.004103970713913441, 0.03812628239393234, -0.02589709684252739, 0.0006658611237071455, 0.002077162964269519, -0.008301458321511745, 0.008596397936344147, -0.003589625470340252, -0.005057127680629492, 0.009358922950923443, -0.03746446594595909, 0.003701126901432872, -0.0028666649013757706, 0.00605344632640481, -0.023667069151997566, -0.016401495784521103, -0.021969370543956757, 0.007841065526008606, -0.0040859864093363285, 0.014703796245157719, -0.016243234276771545, -0.02001270093023777, -0.005798072554171085, 0.015279287472367287, -0.0018577571026980877, 0.011373141780495644, -0.01164649985730648, -0.0021329137962311506, -0.0023667069617658854, -0.025508640334010124, -0.0201277993619442, -0.007071346510201693, -0.01904875412583351, 0.03755079209804535, -0.011042234487831593, 0.005898783449083567, -0.011883890256285667, -0.023595133796334267, -0.02179672382771969, -0.00870430190116167, -0.02490437589585781, 0.011387528851628304, 0.02986798621714115, -0.0021958579309284687, -0.008265490643680096, 0.025235282257199287, 0.001056565553881228, 0.020688902586698532, -0.004470846150070429, 0.0018919268622994423, 0.015509483404457569, -0.004175907000899315, 0.025307219475507736, 0.012272346764802933, -0.019653018563985825, -0.02491876296699047, 0.004352150950580835, 0.029839210212230682, 0.041233934462070465, -0.011984600685536861, -0.0008515469380654395, -0.0076684183441102505, 0.0021868660114705563, -0.016099361702799797, -0.012682383880019188, -0.005197403486818075, 0.0012750723399221897, -0.018545199185609818, 0.0014576109824702144, 0.017681961879134178, 0.00019715064263436943, -0.024760503321886063, -0.017955319955945015, -0.011351561173796654, -0.019465984776616096, 0.009869671426713467, 0.005078708287328482, 0.010754489339888096, 0.024789277464151382, -0.02332177571952343, 0.0173510555177927, 0.0037838537245988846, -0.001575406757183373, 0.0241562370210886, -0.004233456216752529, 0.006970635149627924, -0.01333700492978096, 0.014653440564870834, -0.02986798621714115, 0.008517267182469368, 0.009200663305819035, -0.0011563773732632399, 0.026299940422177315, -0.008092842996120453, -0.01424340344965458, -0.018559586256742477, -0.02594025991857052, -0.009970382787287235, -0.0010502712102606893, -0.033666227012872696, -0.017739512026309967, -0.006596566177904606, -0.027868153527379036, -0.00879781972616911, -0.024458369240164757, -0.010178998112678528, -0.004575153812766075, -0.018948042765259743, -0.02156652696430683, -0.04434158653020859, 0.002605895511806011, 0.0007724168826825917, -0.0012507938081398606, -0.047880854457616806, -0.017825834453105927, -0.010164611041545868, 0.012804675847291946, 0.027249502018094063, 0.18795537948608398, 0.006362773012369871, 0.003310872009024024, 0.01674678921699524, -0.011970213614404202, -0.0014054570347070694, 0.008840980939567089, -0.010963104665279388, -0.037953633815050125, 0.04327692836523056, -0.011142945848405361, 0.010308483615517616, -0.008617978543043137, -0.0027335823979228735, -0.008085649460554123, 0.010301290079951286, -0.043823644518852234, 0.00564700597897172, -0.05435512959957123, -0.009905640035867691, 0.033608678728342056, -0.004006856586784124, -0.006643324624747038, -0.02178233675658703, 0.02596903406083584, -0.03418416902422905, 0.010704133659601212, 0.009761766530573368, 0.014085143804550171, 0.005226178094744682, -0.023710230365395546, 0.0055750696919858456, -0.000936971337068826, -0.026961755007505417, -0.01926456205546856, -0.037953633815050125, 0.012315508909523487, 0.008466912433505058, 0.017077697440981865, 0.0058843959122896194, 0.002050186973065138, -0.0251057967543602, 0.015178576111793518, -0.01746615394949913, -0.00251237815245986, 0.032831765711307526, -0.008956079371273518, 0.0013020484475418925, -0.012480962090194225, 0.03896074369549751, -0.04566521570086479, 0.009955994784832, 0.03170955553650856, 0.011121364310383797, 0.016789952293038368, -0.009452440775930882, -0.01335139200091362, -0.0038342091720551252, 0.022314665839076042, 0.009438052773475647, -0.023940427228808403, 0.033666227012872696, 0.0005417708889581263, 0.007790710311383009, -0.018861718475818634, 0.011696855537593365, -0.02926371991634369, 0.005657796282321215, 0.02402675151824951, -0.04578031226992607, 0.009509989991784096, -0.024875599890947342, 0.004308989271521568, -0.007182847708463669, -0.013200325891375542, -0.00787703413516283, 0.0005876303184777498, 0.00913592055439949, 0.009840897284448147, 0.014437631703913212, -0.0050103687681257725, -0.04721904173493385, -0.007344704587012529, -0.0003257369389757514, -0.042010847479104996, -0.03510495275259018, 0.028832102194428444, 0.022314665839076042, -0.00880501326173544, -0.006744035519659519, 0.0007535336189903319, 0.011351561173796654, 0.011847921647131443, -0.005021159537136555, 0.005154241807758808, 0.014286565594375134, -0.00790580827742815, 0.00566139305010438, -0.022976480424404144, 0.004111164249479771, -0.013502459041774273, 0.05501694604754448, -0.011704049073159695, -0.044600557535886765, -0.00791300181299448, 0.009646669030189514, 0.012236378155648708, 0.006909489631652832, 0.010596228763461113, -0.027954477816820145, -0.00710731465369463, -0.0499238483607769, -0.011991795152425766, -0.007524545304477215, -0.00540961604565382, 0.025307219475507736, 0.0064634839072823524, -0.033004410564899445, 0.002731784014031291, -0.013193132355809212, 0.02581077441573143, -0.0014908815501257777, 0.02258802391588688, 0.018775396049022675, -0.0019998312927782536, 0.0010511704022064805, -0.010790457017719746, -0.008517267182469368, -0.00603186571970582, -0.04321937635540962, 0.015653356909751892, -0.0073806727305054665, 0.010452356189489365, -0.013099615462124348, -0.007869839668273926, -0.00067800038959831, 0.017279118299484253, 6.013431993778795e-05, 0.007096523884683847, -0.008653946220874786, -0.003602214390411973, -0.016833113506436348, -0.011797566898167133, -0.011344367638230324, 0.0015385393053293228, -0.005121870432049036, 0.023868491873145103, 0.0026652428787201643, -0.021422654390335083, 0.007395059801638126, -0.019336499273777008, -0.004528395365923643, 0.014272177591919899, 0.009783348068594933, 0.004319779574871063, -0.010617810301482677, -0.030443476513028145, -0.041406579315662384, 0.005366453900933266, -0.007157669868320227, -0.02171039953827858, 0.018919268622994423, 0.037780988961458206, 0.0038342091720551252, -0.03165200725197792, 0.003003343939781189, -0.18473263084888458, -0.0065066455863416195, -0.018631523475050926, -0.034845981746912, 0.027120016515254974, -0.011761598289012909, 0.031018966808915138, 0.0005489644827321172, -0.006830359343439341, -0.01999831385910511, 0.011416303925216198, 0.0192501749843359, -0.049578554928302765, -0.014660634100437164, -0.004258633591234684, -0.015192964114248753, 0.014732571318745613, -0.0006928372895345092, 0.03691774979233742, 0.017739512026309967, 0.018214290961623192, -0.0349898561835289, 0.010553067550063133, 0.009215050376951694, 0.00868272129446268, 0.03240014612674713, -0.004478039685636759, -0.017897771671414375, -0.009438052773475647, -0.014962767250835896, -0.003569843014702201, 0.018530812114477158, 0.02924933284521103, 0.0018083007307723165, 0.0029098265804350376, -0.0023361339699476957, -0.007431028410792351, -3.0151459213811904e-05, 0.015178576111793518, -0.004916851874440908, 0.018185516819357872, 0.018142355605959892, -0.0067943911999464035, 0.026472589001059532, -0.01988321542739868, 0.017926545813679695, 0.04998139664530754, -0.0366012305021286, 0.01128681842237711, -0.021106135100126266, 0.02832854725420475, -0.025278443470597267, -0.009682636708021164, 0.004154325928539038, 0.021897435188293457, 0.005571472924202681, 0.01586916483938694, 0.021868659183382988, -0.03099019266664982, -6.738415686413646e-05, -0.0005327787948772311, -0.018631523475050926, 0.015293674543499947, -0.00353387463837862, 0.0028648662846535444, -0.004891674034297466, -0.010668165050446987, -0.00961789395660162, -0.00768999895080924, 0.017797060310840607, -0.020631354302167892, -0.011660887859761715, 0.005085902288556099, 0.012171635404229164, 0.008121617138385773, 0.028932811692357063, -0.031076516956090927, 0.009229437448084354, 0.003350436920300126, -0.005826846696436405, -0.004902464337646961, 0.042989183217287064, -0.009891252033412457, -0.008855368942022324, -0.00239368318580091, -0.01040200050920248, -0.010660971514880657, -0.012495349161326885, 0.008071262389421463, -0.02008463814854622, 0.01421462930738926, -0.018228679895401, -0.014862056821584702, -0.011768791824579239, -0.01502031646668911, 0.004028437193483114, -0.0033971955999732018, -0.008164779283106327, 0.01254570484161377, -0.008006519638001919, -0.016789952293038368, -0.020314833149313927, -0.003051901003345847, -0.003602214390411973, -0.012747126631438732, 0.0015700114890933037, -0.03668755292892456, -0.0027929299976676702, 0.020616967231035233, 0.011761598289012909, -0.011222075670957565, 0.03196852654218674, 0.009876864962279797, 0.02583954855799675, -0.009862477891147137, 0.0027821394614875317, -0.0038665805477648973, 0.006823165807873011, 0.017063310369849205, -0.020199736580252647, 0.041406579315662384, -0.0116033386439085, -0.011761598289012909, 0.023494422435760498, -0.011876696720719337, -0.023839715868234634, -0.09668249636888504, -0.031076516956090927, -0.00787703413516283, 0.022789444774389267, -0.014164273627102375, 0.030242053791880608, 0.003294686321169138, 0.033579904586076736, -0.007956163957715034, 0.0083446204662323, -0.0047190263867378235, -0.04445668309926987, -0.0008317644242197275, 0.012351476587355137, -0.0026076938956975937, 0.011027847416698933, -0.02244415134191513, -0.017048921436071396, 0.0200558640062809, 0.03429926559329033, 0.02419939823448658, -0.021839885041117668, 0.011970213614404202, -0.013207519426941872, -0.028012026101350784, -0.003449349431321025, -0.017782673239707947, 0.0173510555177927, 0.005963526200503111, 0.019523533061146736, 0.0380687341094017, 0.003035715315490961, 0.04742046073079109, -0.012588866986334324, -0.03691774979233742, -0.027925703674554825, -0.018199903890490532, 0.003346840152516961, 0.02432888373732567, -0.042845308780670166, 0.01912068948149681, 0.00871868897229433, 0.009286986663937569, 0.005618231371045113, -0.023091578856110573, 0.005111079663038254, -0.02248731255531311, 0.03752201795578003, 0.04575153812766075, -0.01504909060895443, -0.0402555987238884, 0.02737898752093315, -0.00015803524001967162, -0.021019810810685158, 0.003553657326847315, -0.007510158233344555, 0.03435681387782097, 0.006114592310041189, -0.00917188823223114, -0.007315929979085922, 0.008136005140841007, -0.0019332902738824487, 0.015192964114248753, 0.009653862565755844, -0.026961755007505417, 0.009898446500301361, 0.0005602045566774905, -0.03196852654218674, 0.03087509423494339, -0.01756686344742775, -0.017048921436071396, 0.022329052910208702, -0.025666899979114532, 0.009416472166776657, -0.01910630241036415, 0.0036201984621584415, -1.7815495084505528e-05, -0.02760918252170086, 0.0330907367169857, -0.010898361913859844, -0.021192457526922226, -0.018084805458784103, -0.021437041461467743, 0.01674678921699524, 0.0108336191624403, 0.014077950268983841, 0.009445247240364552, 0.021264394745230675, 0.04825492575764656, -0.018746620044112206, -0.0029637787956744432, 0.0055462950840592384, 0.004226262215524912, -0.004301795735955238, 0.010977491736412048, 0.03979520499706268, 0.00434855418279767, 0.014876443892717361, 0.012466575019061565, -0.01500592939555645, -0.05447022616863251, 0.0012391041964292526, -0.08776238560676575, 0.04744923859834671, 0.011761598289012909, 0.0018631522543728352, 0.0025213700719177723, 0.008898530155420303, 0.0028504792135208845, -0.029436366632580757, -0.013416134752333164, 0.023710230365395546, -0.019796893000602722, 0.02408429980278015, 0.0043737320229411125, -0.000470733706606552, -0.0023667069617658854, -0.02506263554096222, 0.026904206722974777, 0.011704049073159695, 0.03904706612229347, -0.0019566696137189865, -0.0036148030776530504, -0.015538258478045464, -0.04005417600274086, 0.032227497547864914, 0.00518661318346858, -0.0316232331097126, 0.006064237095415592, -0.001987242605537176, 0.00768280541524291, -0.014387276023626328, 0.009718605317175388, -0.045435018837451935, 0.014185854233801365, 0.02838609553873539, 0.0023325372021645308, -0.02509140968322754, 0.005607441067695618, 0.0038342091720551252, 0.037867311388254166, 0.008934498764574528, -0.019537921994924545, -0.015192964114248753, -0.008848174475133419, 0.003163402434438467, 0.01165369339287281, -0.013092420995235443, -0.02582516148686409, 0.009768960997462273, 0.013660718686878681, 0.003801837796345353, 0.024846825748682022, 0.009028015658259392, -0.02255924977362156, 0.0010457751341164112, -0.0034835191909223795, -0.01376143004745245, 0.005657796282321215, 0.01582600362598896, -0.012459381483495235, -0.027911316603422165, 0.037032850086688995, -0.012905387207865715, 0.021293168887495995, -0.027767444029450417, 0.013711074367165565, 0.0028217046055942774, 0.00022333998640533537, 0.004111164249479771, 0.02421378530561924, 0.02842925861477852, -0.020257284864783287, -0.013689493760466576, 0.02169601246714592, 0.008215134963393211, 1.9108101696474478e-05, 0.010934329591691494, -0.018674684688448906, -0.006970635149627924, -0.010596228763461113, 0.0012220193166285753, 0.017868997529149055, -0.03596819192171097, -0.013833366334438324, -0.021422654390335083, 0.01211408618837595, -0.010437969118356705, 0.003111248603090644, -0.0020879535004496574, -0.014905218034982681, -0.013020484708249569, -0.03919094055891037, 0.010999072343111038, -0.012452187947928905, -0.0074238344095647335, 0.004161519464105368, 0.016228847205638885, 0.0009432657971046865, 0.013847753405570984, 0.006326804868876934, -0.004686655011028051, 0.006736841984093189, -0.004014050122350454, -0.02337932400405407, -0.025494253262877464, -0.022918930277228355, -0.007625256199389696, -0.019782504066824913, -0.020616967231035233, -0.007546126376837492, -0.008840980939567089, 0.022213954478502274, 0.01412830501794815, -0.003382808296009898, 0.026184841990470886, -0.021278781816363335, 0.0027929299976676702, 0.022199567407369614, -0.00439531309530139, -0.004470846150070429, 0.023882878944277763, 0.0055966502986848354, 0.01752370223402977, 0.03401152044534683, 0.007524545304477215, 0.023739006370306015, 0.011214882135391235, 0.029666563495993614, -0.02242976427078247, 0.0077187735587358475, -0.030673673376441002, 0.011869503185153008, -0.005143451038748026, -0.013545620255172253, -0.012624834664165974, -0.02844364568591118, -0.0009243824752047658, -0.037867311388254166, 0.020214123651385307, -0.005165032111108303, 0.07366285473108292, 0.0038521932438015938, -0.020559417083859444, -0.03855790197849274, 0.018070418387651443, -0.007218815851956606, 0.01831500232219696, -0.0014189451467245817, 0.028760164976119995, -0.027666732668876648, 0.022890156134963036, 0.0019530727295204997, 0.011862309649586678, 0.0022030516993254423, -0.012243571691215038, -0.008661140687763691, 0.0015169584657996893, 0.02845803275704384, -0.010711327195167542, 0.007319526746869087, 0.022170793265104294, 0.009186276234686375, 0.019422823563218117, 0.018948042765259743, 0.0028810519725084305, -0.005330485757440329, 0.03004063293337822, -0.002987158251926303, -0.0058700088411569595, -0.04316182807087898, -0.0008155787363648415, -0.012286733835935593, 0.011552982963621616, -0.00666850246489048, 0.0076612248085439205, -0.022026920691132545, -0.001176159828901291, 0.0009864276507869363, 0.00605344632640481, 0.010013544000685215, -0.011912664398550987, 0.010466743260622025, -0.0481685996055603, -0.020458707585930824, 0.016818726435303688, -0.02411307580769062, 0.017912158742547035, 0.002213842235505581, -0.022026920691132545]

将文档和元数据加载至 TimescaleVector 向量存储库¶

现在我们已经完成了节点准备并为其添加了嵌入向量，接下来将这些节点存入 TimescaleVector 向量存储库。

我们将基于已创建的节点列表构建 Timescale Vector 实例。

首先需要定义集合名称（即 PostgreSQL 数据库中的表名）。

同时需设置时间增量参数，该值将传递给 time_partition_interval 参数，作为按时间分区的时间间隔。每个分区将包含指定时长范围内的数据。为简化操作我们采用7天作为间隔，但您可根据实际需求灵活调整——例如若需频繁查询近期向量，可选用更小的时间增量（如1天）；若需查询跨越十年周期的向量，则可选用更大的时间增量（如6个月或1年）。

最后将节点添加至 Timescale Vector 向量存储库。

In [ ]:

# Create a timescale vector store and add the newly created nodes to it
ts_vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="li_commit_history",
    time_partition_interval=timedelta(days=7),
)
_ = ts_vector_store.add(nodes)

# Create a timescale vector store and add the newly created nodes to it ts_vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="li_commit_history", time_partition_interval=timedelta(days=7), ) _ = ts_vector_store.add(nodes)

按时间和相似度查询向量¶

现在我们已经将文档加载到 TimescaleVector 中，可以通过时间和相似度进行查询。

TimescaleVector 提供了多种方法，通过结合相似性搜索与基于时间的过滤来查询向量。下面我们逐一查看这些方法。

首先我们定义一个查询字符串，并获取该查询字符串的向量嵌入。

In [ ]:

# Define query and generate embedding for it
query_str = "What's new with TimescaleDB functions?"
embed_model = OpenAIEmbedding()
query_embedding = embed_model.get_query_embedding(query_str)

# Define query and generate embedding for it query_str = "What's new with TimescaleDB functions?" embed_model = OpenAIEmbedding() query_embedding = embed_model.get_query_embedding(query_str)

然后我们设置一些将在时间过滤器中使用的变量。

In [ ]:

# Time filter variables for query
start_dt = datetime(
    2023, 8, 1, 22, 10, 35
)  # Start date = 1 August 2023, 22:10:35
end_dt = datetime(
    2023, 8, 30, 22, 10, 35
)  # End date = 30 August 2023, 22:10:35
td = timedelta(days=7)  # Time delta = 7 days

# Time filter variables for query start_dt = datetime( 2023, 8, 1, 22, 10, 35 ) # Start date = 1 August 2023, 22:10:35 end_dt = datetime( 2023, 8, 30, 22, 10, 35 ) # End date = 30 August 2023, 22:10:35 td = timedelta(days=7) # Time delta = 7 days

方法一：在提供的起始日期和结束日期范围内进行筛选。

In [ ]:

# Query the vector database
vector_store_query = VectorStoreQuery(
    query_embedding=query_embedding, similarity_top_k=5
)

# return most similar vectors to query between start date and end date date range
# returns a VectorStoreQueryResult object
query_result = ts_vector_store.query(
    vector_store_query, start_date=start_dt, end_date=end_dt
)
query_result

# Query the vector database vector_store_query = VectorStoreQuery( query_embedding=query_embedding, similarity_top_k=5 ) # return most similar vectors to query between start date and end date date range # returns a VectorStoreQueryResult object query_result = ts_vector_store.query( vector_store_query, start_date=start_dt, end_date=end_dt ) query_result

Out[ ]:

VectorStoreQueryResult(nodes=[TextNode(id_='22747180-31f1-11ee-bd8e-101e36c28c91', embedding=None, metadata={'commit': ' 7aeed663b9c0f337b530fd6cad47704a51a9b2ec', 'author': 'Dmitry Simonenko', 'date': '2023-08-3 14:30:23+0500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='3273f20a98f02c75847896b929888b05e8751ae5e258d7feb8605bd5290ef8ca', text='Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='faa8ea00-4686-11ee-b933-c2c7df407c25', embedding=None, metadata={'commit': ' e4facda540286b0affba47ccc63959fefe2a7b26', 'author': 'Sven Klemm', 'date': '2023-08-29 18:13:24+0320'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='6f45ab1cccf673ddf75c625983b6cf2f4a66bbf865a4c1c65025997a470f3bb3', text='Tue Aug 29 18:13:24 2023 +0200 Sven Klemm Add compatibility layer for _timescaledb_internal functions With timescaledb 2.12 all the functions present in _timescaledb_internal were moved into the _timescaledb_functions schema to improve schema security. This patch adds a compatibility layer so external callers of these internal functions will not break and allow for more flexibility when migrating. ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='d7080180-40d2-11ee-af6f-f43e81a0925a', embedding=None, metadata={'commit': ' cf04496e4b4237440274eb25e4e02472fc4e06fc', 'author': 'Sven Klemm', 'date': '2023-08-22 12:01:19+0320'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='d5a20dc83ae04f44aa901ba2f654e80ca68cb21f6a313bd91afcd91e404b471e', text='Tue Aug 22 12:01:19 2023 +0200 Sven Klemm Move utility functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - generate_uuid() - get_git_commit() - get_os_info() - tsl_loaded() ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='01b10780-4649-11ee-a375-5719b2881af3', embedding=None, metadata={'commit': ' a9751ccd5eb030026d7b975d22753f5964972389', 'author': 'Sven Klemm', 'date': '2023-08-29 10:49:47+0320'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='8fde14d147def41808d82bf2ffa35e1e0ed78b0331962907cee856af34a34e44', text='Tue Aug 29 10:49:47 2023 +0200 Sven Klemm Move partitioning functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - get_partition_for_key(val anyelement) - get_partition_hash(val anyelement) ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='e7ba7f80-36af-11ee-9479-6c18a6a65db1', embedding=None, metadata={'commit': ' 44eab9cf9bef34274c88efd37a750eaa74cd8044', 'author': 'Konstantina Skovola', 'date': '2023-08-9 15:26:03+0500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='f0db9c719928ecc16653bbf0a44f1eaeb221ac79dae16fd36710044d1561dbfa', text='Wed Aug 9 15:26:03 2023 +0300 Konstantina Skovola Release 2.11.2 This release contains bug fixes since the 2.11.1 release. We recommend that you upgrade at the next available opportunity.  **Features** * #5923 Feature flags for TimescaleDB features  **Bugfixes** * #5680 Fix DISTINCT query with JOIN on multiple segmentby columns * #5774 Fixed two bugs in decompression sorted merge code * #5786 Ensure pg_config --cppflags are passed * #5906 Fix quoting owners in sql scripts. * #5912 Fix crash in 1-step integer policy creation  **Thanks** * @mrksngl for submitting a PR to fix extension upgrade scripts * @ericdevries for reporting an issue with DISTINCT queries using segmentby columns of compressed hypertable ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n')], similarities=[0.18141598590707553, 0.1821951378700205, 0.1948705199438009, 0.19657938500765504, 0.19664154042725346], ids=[UUID('22747180-31f1-11ee-bd8e-101e36c28c91'), UUID('faa8ea00-4686-11ee-b933-c2c7df407c25'), UUID('d7080180-40d2-11ee-af6f-f43e81a0925a'), UUID('01b10780-4649-11ee-a375-5719b2881af3'), UUID('e7ba7f80-36af-11ee-9479-6c18a6a65db1')])

让我们检查通过相似性搜索返回的节点：

In [ ]:

# for each node in the query result, print the node metadata date
for node in query_result.nodes:
    print("-" * 80)
    print(node.metadata["date"])
    print(node.get_content(metadata_mode="all"))

# for each node in the query result, print the node metadata date for node in query_result.nodes: print("-" * 80) print(node.metadata["date"]) print(node.get_content(metadata_mode="all"))

--------------------------------------------------------------------------------
2023-08-3 14:30:23+0500
commit:  7aeed663b9c0f337b530fd6cad47704a51a9b2ec
author: Dmitry Simonenko
date: 2023-08-3 14:30:23+0500

Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create
--------------------------------------------------------------------------------
2023-08-29 18:13:24+0320
commit:  e4facda540286b0affba47ccc63959fefe2a7b26
author: Sven Klemm
date: 2023-08-29 18:13:24+0320

Tue Aug 29 18:13:24 2023 +0200 Sven Klemm Add compatibility layer for _timescaledb_internal functions With timescaledb 2.12 all the functions present in _timescaledb_internal were moved into the _timescaledb_functions schema to improve schema security. This patch adds a compatibility layer so external callers of these internal functions will not break and allow for more flexibility when migrating.
--------------------------------------------------------------------------------
2023-08-22 12:01:19+0320
commit:  cf04496e4b4237440274eb25e4e02472fc4e06fc
author: Sven Klemm
date: 2023-08-22 12:01:19+0320

Tue Aug 22 12:01:19 2023 +0200 Sven Klemm Move utility functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - generate_uuid() - get_git_commit() - get_os_info() - tsl_loaded()
--------------------------------------------------------------------------------
2023-08-29 10:49:47+0320
commit:  a9751ccd5eb030026d7b975d22753f5964972389
author: Sven Klemm
date: 2023-08-29 10:49:47+0320

Tue Aug 29 10:49:47 2023 +0200 Sven Klemm Move partitioning functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - get_partition_for_key(val anyelement) - get_partition_hash(val anyelement)
--------------------------------------------------------------------------------
2023-08-9 15:26:03+0500
commit:  44eab9cf9bef34274c88efd37a750eaa74cd8044
author: Konstantina Skovola
date: 2023-08-9 15:26:03+0500

Wed Aug 9 15:26:03 2023 +0300 Konstantina Skovola Release 2.11.2 This release contains bug fixes since the 2.11.1 release. We recommend that you upgrade at the next available opportunity.  **Features** * #5923 Feature flags for TimescaleDB features  **Bugfixes** * #5680 Fix DISTINCT query with JOIN on multiple segmentby columns * #5774 Fixed two bugs in decompression sorted merge code * #5786 Ensure pg_config --cppflags are passed * #5906 Fix quoting owners in sql scripts. * #5912 Fix crash in 1-step integer policy creation  **Thanks** * @mrksngl for submitting a PR to fix extension upgrade scripts * @ericdevries for reporting an issue with DISTINCT queries using segmentby columns of compressed hypertable

请注意查询结果仅返回指定日期范围内的数据。

方法 2：在给定的起始日期和时间增量范围内进行筛选。

In [ ]:

vector_store_query = VectorStoreQuery(
    query_embedding=query_embedding, similarity_top_k=5
)

# return most similar vectors to query from start date and a time delta later
query_result = ts_vector_store.query(
    vector_store_query, start_date=start_dt, time_delta=td
)

for node in query_result.nodes:
    print("-" * 80)
    print(node.metadata["date"])
    print(node.get_content(metadata_mode="all"))

vector_store_query = VectorStoreQuery( query_embedding=query_embedding, similarity_top_k=5 ) # return most similar vectors to query from start date and a time delta later query_result = ts_vector_store.query( vector_store_query, start_date=start_dt, time_delta=td ) for node in query_result.nodes: print("-" * 80) print(node.metadata["date"]) print(node.get_content(metadata_mode="all"))

--------------------------------------------------------------------------------
2023-08-3 14:30:23+0500
commit:  7aeed663b9c0f337b530fd6cad47704a51a9b2ec
author: Dmitry Simonenko
date: 2023-08-3 14:30:23+0500

Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create
--------------------------------------------------------------------------------
2023-08-7 19:49:47+-500
commit:  5bba74a2ec083728f8e93e09d03d102568fd72b5
author: Fabrízio de Royes Mello
date: 2023-08-7 19:49:47+-500

Mon Aug 7 19:49:47 2023 -0300 Fabrízio de Royes Mello Relax strong table lock when refreshing a CAGG When refreshing a Continuous Aggregate we take a table lock on _timescaledb_catalog.continuous_aggs_invalidation_threshold when processing the invalidation logs (the first transaction of the refresh Continuous Aggregate procedure). It means that even two different Continuous Aggregates over two different hypertables will wait each other in the first phase of the refreshing procedure. Also it lead to problems when a pg_dump is running because it take an AccessShareLock on tables so Continuous Aggregate refresh execution will wait until the pg_dump finish.  Improved it by relaxing the strong table-level lock to a row-level lock so now the Continuous Aggregate refresh procedure can be executed in multiple sessions with less locks.  Fix #3554
--------------------------------------------------------------------------------
2023-08-3 14:36:39+0500
commit:  2863daf3df83c63ee36c0cf7b66c522da5b4e127
author: Dmitry Simonenko
date: 2023-08-3 14:36:39+0500

Thu Aug 3 14:36:39 2023 +0300 Dmitry Simonenko Support CREATE INDEX ONLY ON main table This PR adds support for CREATE INDEX ONLY ON clause which allows to create index only on the main table excluding chunks.  Fix #5908
--------------------------------------------------------------------------------
2023-08-2 20:24:14+0140
commit:  3af0d282ea71d9a8f27159a6171e9516e62ec9cb
author: Lakshmi Narayanan Sreethar
date: 2023-08-2 20:24:14+0140

Wed Aug 2 20:24:14 2023 +0100 Lakshmi Narayanan Sreethar PG16: ExecInsertIndexTuples requires additional parameter PG16 adds a new boolean parameter to the ExecInsertIndexTuples function to denote if the index is a BRIN index, which is then used to determine if the index update can be skipped. The fix also removes the INDEX_ATTR_BITMAP_ALL enum value.  Adapt these changes by updating the compat function to accomodate the new parameter added to the ExecInsertIndexTuples function and using an alternative for the removed INDEX_ATTR_BITMAP_ALL enum value.  postgres/postgres@19d8e23
--------------------------------------------------------------------------------
2023-08-7 16:36:17+0500
commit:  373c55662ca5f8a2993abf9b2aa7f5f4006b3229
author: Konstantina Skovola
date: 2023-08-7 16:36:17+0500

Mon Aug 7 16:36:17 2023 +0300 Konstantina Skovola Fix ordered append for partially compressed chunks In the exclusive presence of partially compressed chunks, this optimization was not applied because no pathkeys were supplied. Additionally, this patch makes sure that if applicable, the `enable_decompression_sorted_merge` optimization is chosen for the path, since it is more beneficial due to the ability to push down the sort below DecompressChunk.

再次注意，只有位于开始日期（8月1日）与定义的时间增量后（7天后）之间的节点会被返回。

方法 3：在提供的结束日期和时间增量范围内进行筛选。

In [ ]:

vector_store_query = VectorStoreQuery(
    query_embedding=query_embedding, similarity_top_k=5
)

# return most similar vectors to query from end date and a time delta earlier
query_result = ts_vector_store.query(
    vector_store_query, end_date=end_dt, time_delta=td
)

for node in query_result.nodes:
    print("-" * 80)
    print(node.metadata["date"])
    print(node.get_content(metadata_mode="all"))

vector_store_query = VectorStoreQuery( query_embedding=query_embedding, similarity_top_k=5 ) # return most similar vectors to query from end date and a time delta earlier query_result = ts_vector_store.query( vector_store_query, end_date=end_dt, time_delta=td ) for node in query_result.nodes: print("-" * 80) print(node.metadata["date"]) print(node.get_content(metadata_mode="all"))

--------------------------------------------------------------------------------
2023-08-29 18:13:24+0320
commit:  e4facda540286b0affba47ccc63959fefe2a7b26
author: Sven Klemm
date: 2023-08-29 18:13:24+0320

Tue Aug 29 18:13:24 2023 +0200 Sven Klemm Add compatibility layer for _timescaledb_internal functions With timescaledb 2.12 all the functions present in _timescaledb_internal were moved into the _timescaledb_functions schema to improve schema security. This patch adds a compatibility layer so external callers of these internal functions will not break and allow for more flexibility when migrating.
--------------------------------------------------------------------------------
2023-08-29 10:49:47+0320
commit:  a9751ccd5eb030026d7b975d22753f5964972389
author: Sven Klemm
date: 2023-08-29 10:49:47+0320

Tue Aug 29 10:49:47 2023 +0200 Sven Klemm Move partitioning functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - get_partition_for_key(val anyelement) - get_partition_hash(val anyelement)
--------------------------------------------------------------------------------
2023-08-28 23:26:23+0320
commit:  b2a91494a11d8b82849b6f11f9ea6dc26ef8a8cb
author: Sven Klemm
date: 2023-08-28 23:26:23+0320

Mon Aug 28 23:26:23 2023 +0200 Sven Klemm Move ddl_internal functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - chunk_constraint_add_table_constraint(_timescaledb_catalog.chunk_constraint) - chunk_drop_replica(regclass,name) - chunk_index_clone(oid) - chunk_index_replace(oid,oid) - create_chunk_replica_table(regclass,name) - drop_stale_chunks(name,integer[]) - health() - hypertable_constraint_add_table_fk_constraint(name,name,name,integer) - process_ddl_event() - wait_subscription_sync(name,name,integer,numeric)
--------------------------------------------------------------------------------
2023-08-29 14:47:57+0320
commit:  08231c8aacd17152f315ad36d95c031fb46073aa
author: Jan Nidzwetzki
date: 2023-08-29 14:47:57+0320

Tue Aug 29 14:47:57 2023 +0200 Jan Nidzwetzki Export is_decompress_chunk_path / is_gapfill_path This patch adds the 'ts_' prefix to the function names of is_decompress_chunk_path and is_gapfill_path and makes them available for use by other parts of TimescaleDB.
--------------------------------------------------------------------------------
2023-08-28 15:32:54+0320
commit:  6576d969b319dac8e7fd08a9cf4cfc8197b34d1d
author: Sven Klemm
date: 2023-08-28 15:32:54+0320

Mon Aug 28 15:32:54 2023 +0200 Sven Klemm Move log invalidation functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - cagg_watermark(integer) - cagg_watermark_materialized(integer) - hypertable_invalidation_log_delete(integer) - invalidation_cagg_log_add_entry(integer,bigint,bigint) - invalidation_hyper_log_add_entry(integer,bigint,bigint) - invalidation_process_cagg_log(integer,integer,regtype,bigint,bigint,integer[],bigint[],bigint[]) - invalidation_process_cagg_log(integer,integer,regtype,bigint,bigint,integer[],bigint[],bigint[],text[]) - invalidation_process_hypertable_log(integer,integer,regtype,integer[],bigint[],bigint[]) - invalidation_process_hypertable_log(integer,integer,regtype,integer[],bigint[],bigint[],text[]) - materialization_invalidation_log_delete(integer)

主要结论是：在上述每个查询结果中，仅返回指定时间范围内的向量。这类查询非常高效，因为它们只需搜索相关的数据分区。

4. 将 TimescaleVector 存储用作检索器与查询引擎¶

在探索了基础相似性搜索和基于时间过滤的相似性搜索后，现在让我们看看如何结合 Timescale Vector 的这些特性与 LLamaIndex 的检索器和查询引擎。

首先我们将了解如何将 TimescaleVector 作为检索器使用，具体来说是向量存储检索器。

为了将检索到的节点限制在相关时间范围内，我们可以使用 TimescaleVector 的时间过滤器。在创建检索器时，只需将时间过滤器参数作为 vector_strored_kwargs 传入即可。

In [ ]:

from llama_index.core import VectorStoreIndex
from llama_index.core import StorageContext

index = VectorStoreIndex.from_vector_store(ts_vector_store)
retriever = index.as_retriever(
    vector_store_kwargs=({"start_date": start_dt, "time_delta": td})
)
retriever.retrieve("What's new with TimescaleDB functions?")

from llama_index.core import VectorStoreIndex from llama_index.core import StorageContext index = VectorStoreIndex.from_vector_store(ts_vector_store) retriever = index.as_retriever( vector_store_kwargs=({"start_date": start_dt, "time_delta": td}) ) retriever.retrieve("What's new with TimescaleDB functions?")

Out[ ]:

[NodeWithScore(node=TextNode(id_='22747180-31f1-11ee-bd8e-101e36c28c91', embedding=None, metadata={'commit': ' 7aeed663b9c0f337b530fd6cad47704a51a9b2ec', 'author': 'Dmitry Simonenko', 'date': '2023-08-3 14:30:23+0500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='3273f20a98f02c75847896b929888b05e8751ae5e258d7feb8605bd5290ef8ca', text='Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.1813839050209377),
 NodeWithScore(node=TextNode(id_='b5583780-3574-11ee-871a-5a8c45d660c8', embedding=None, metadata={'commit': ' 5bba74a2ec083728f8e93e09d03d102568fd72b5', 'author': 'Fabrízio de Royes Mello', 'date': '2023-08-7 19:49:47+-500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='ec25a09b9dd34ed2650aefc2ce71e1b11fa471ffc43683715de788d202c6cdc8', text='Mon Aug 7 19:49:47 2023 -0300 Fabrízio de Royes Mello Relax strong table lock when refreshing a CAGG When refreshing a Continuous Aggregate we take a table lock on _timescaledb_catalog.continuous_aggs_invalidation_threshold when processing the invalidation logs (the first transaction of the refresh Continuous Aggregate procedure). It means that even two different Continuous Aggregates over two different hypertables will wait each other in the first phase of the refreshing procedure. Also it lead to problems when a pg_dump is running because it take an AccessShareLock on tables so Continuous Aggregate refresh execution will wait until the pg_dump finish.  Improved it by relaxing the strong table-level lock to a row-level lock so now the Continuous Aggregate refresh procedure can be executed in multiple sessions with less locks.  Fix #3554 ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.23511557892997959)]

接下来我们将探讨如何将 TimescaleVector 作为查询引擎使用。

同样地，我们通过创建查询引擎时在 vector_strored_kwargs 中传入时间过滤参数，利用 TimescaleVector 的时间过滤器将搜索范围限定在相关时间段内。

In [ ]:

index = VectorStoreIndex.from_vector_store(ts_vector_store)
query_engine = index.as_query_engine(
    vector_store_kwargs=({"start_date": start_dt, "end_date": end_dt})
)

# query_str = "What's new with TimescaleDB? List 3 new features"
query_str = (
    "What's new with TimescaleDB functions? When were these changes made and"
    " by whom?"
)
response = query_engine.query(query_str)
print(str(response))

index = VectorStoreIndex.from_vector_store(ts_vector_store) query_engine = index.as_query_engine( vector_store_kwargs=({"start_date": start_dt, "end_date": end_dt}) ) # query_str = "What's new with TimescaleDB? List 3 new features" query_str = ( "What's new with TimescaleDB functions? When were these changes made and" " by whom?" ) response = query_engine.query(query_str) print(str(response))

TimescaleDB functions have undergone changes recently. These changes were made by Sven Klemm on August 29, 2023. The changes involve adding a compatibility layer for _timescaledb_internal functions. This layer ensures that external callers of these internal functions will not break and allows for more flexibility when migrating.