Nomic 嵌入模型¶

Nomic 已发布 v1.5 版本 🪆🪆🪆，该版本通过套娃学习（matryoshka learning）技术实现了可变尺寸嵌入，支持 8192 上下文长度，嵌入维度可在 64 至 768 之间灵活调整。

本笔记本将探索在不同维度下使用 Nomic v1.5 嵌入模型的方法。

安装¶

In [ ]:

%pip install -U llama-index llama-index-embeddings-nomic

%pip install -U llama-index llama-index-embeddings-nomic

设置 API 密钥¶

In [ ]:

nomic_api_key = "<NOMIC API KEY>"

nomic_api_key = ""

In [ ]:

import nest_asyncio

nest_asyncio.apply()

from llama_index.embeddings.nomic import NomicEmbedding

import nest_asyncio nest_asyncio.apply() from llama_index.embeddings.nomic import NomicEmbedding

维度设为 128¶

In [ ]:

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=128, model_name="nomic-embed-text-v1.5", ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]:

print(len(embedding))

print(len(embedding))

128

In [ ]:

embedding[:5]

embedding[:5]

Out[ ]:

[0.05569458, 0.057922363, -0.30126953, -0.09832764, 0.05947876]

维度设为 256¶

In [ ]:

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=256,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=256, model_name="nomic-embed-text-v1.5", ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]:

print(len(embedding))

print(len(embedding))

256

In [ ]:

embedding[:5]

embedding[:5]

Out[ ]:

[0.044708252, 0.04650879, -0.24182129, -0.07897949, 0.04776001]

维度设置为 768¶

In [ ]:

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=768,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=768, model_name="nomic-embed-text-v1.5", ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]:

print(len(embedding))

print(len(embedding))

768

In [ ]:

embedding[:5]

embedding[:5]

Out[ ]:

[0.027282715, 0.028381348, -0.14758301, -0.048187256, 0.029144287]

您仍可使用 v1 版 Nomic 嵌入模型¶

该版本采用固定的 768 维嵌入向量

In [ ]:

embed_model = NomicEmbedding(
    api_key=nomic_api_key, model_name="nomic-embed-text-v1"
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, model_name="nomic-embed-text-v1" ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]:

print(len(embedding))

print(len(embedding))

768

In [ ]:

embedding[:5]

embedding[:5]

Out[ ]:

[0.0059013367, 0.03744507, 0.0035305023, -0.047180176, 0.0154418945]

使用 Nomic v1.5 嵌入模型构建端到端 RAG 流水线¶

我们将采用 OpenAI 完成生成环节。

设置嵌入模型与LLM¶

In [ ]:

from llama_index.core import settings
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI

import os

os.environ["OPENAI_API_KEY"] = "<YOUR OPENAI API KEY>"

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)

llm = OpenAI(model="gpt-3.5-turbo")

settings.llm = llm
settings.embed_model = embed_model

from llama_index.core import settings from llama_index.core import VectorStoreIndex, SimpleDirectoryReader from llama_index.llms.openai import OpenAI import os os.environ["OPENAI_API_KEY"] = "" embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=128, model_name="nomic-embed-text-v1.5", ) llm = OpenAI(model="gpt-3.5-turbo") settings.llm = llm settings.embed_model = embed_model

下载数据¶

In [ ]:

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

--2024-02-16 18:37:03--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8003::154, 2606:50c0:8000::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: 'data/paul_graham/paul_graham_essay.txt'

data/paul_graham/pa 100%[===================>]  73.28K  --.-KB/s    in 0.02s   

2024-02-16 18:37:03 (3.87 MB/s) - 'data/paul_graham/paul_graham_essay.txt' saved [75042/75042]

加载数据¶

In [ ]:

documents = SimpleDirectoryReader("./data/paul_graham").load_data()

documents = SimpleDirectoryReader("./data/paul_graham").load_data()

索引创建¶

In [ ]:

index = VectorStoreIndex.from_documents(documents)

index = VectorStoreIndex.from_documents(documents)

查询引擎¶

In [ ]:

query_engine = index.as_query_engine()

query_engine = index.as_query_engine()

In [ ]:

response = query_engine.query("what did author do growing up?")
print(response)

response = query_engine.query("what did author do growing up?") print(response)

The author, growing up, worked on writing and programming. They wrote short stories and also tried writing programs on an IBM 1401 computer. Later, they got a microcomputer and started programming more extensively, writing simple games and a word processor.