1、使用 sentence-transformers 将文本编码为向量

安装 sentence-transformers：

pip install -U sentence-transformers

在 huggingface 下载 all-MiniLM-L6-v2 模型权重（1_Pooling 是文件夹，里面包含一个 config.json 文件）：

~$ ls
1_Pooling    config_sentence_transformers.json  model.safetensors  sentence_bert_config.json  tokenizer_config.json  train_script.py
config.json  data_config.json                   modules.json       special_tokens_map.json    tokenizer.json         vocab.txt

运行下面的示例脚本，将一句话编码为一个向量：

from sentence_transformers import SentenceTransformer

model_path = "/hub/weights/all-MiniLM-L6-v2"
model = SentenceTransformer(model_path)
sentence = ['This framework generates embeddings for each input sentence']
embedding = model.encode(sentence)
print(len(embedding), len(embedding[0]))  # 1 384

2、使用SQuAD-explorer数据集构建向量数据库

下载 SQuAD-explorer 数据集，这个数据集分为 Training Set 和 Dev Set ，Dev Set 更小更方便格式化预览数据集的结构，也更方便调试。

也可以使用其他的数据集，像第一节演示的那样，只需要是模型支持的语言的句子就可以编码成向量。

.json文件加载后的第一层是一个Python dict，包含两个key："version"和"data"，"data"对应的值是一个list，可以看一下这个list的长度：

import json

with open("dev-v2.0.json", "r") as f:
    data = json.load(f)

data = data["data"]
print(len(data))  # 35

dev数据集中有35条，train数据集中有442条，对于每一条数据，也是包含两个key："title"和"paragraphs"，"paragraphs"对应的值是一个list，"paragraphs"里的每一个元素是dict，我们只需要关注里面的"qas"，即QA pairs，下面使用这些QA pairs来构建向量数据库。

下面先对数据进行读取，获取到数据集中包含的全部QA，由于有些问题含有多个答案，有些问题没有答案，这里统一排除掉没有答案的问题，对于包含多个答案的问题仅获取第一条答案，形成一对一的QA映射关系：

import json

with open("dev-v2.0.json", "r") as f:
    dataset = json.load(f)

qas = [
    (qas["question"], qas["answers"][0]["text"])
    for data in dataset["data"]
    for paragraphs in data["paragraphs"]
    for qas in paragraphs["qas"]
    if qas["answers"]
]

print(len(qas))
print(qas[0])

可以看到第一组问答已经被提取出来了：

5928
('In what country is Normandy located?', 'France')

接下来我们查询一个问题的答案，如果问题刚好和数据中存在的问题完全一致，就可以匹配到答案，例如：

qas_dict = dict(qas)
q = 'In what country is Normandy located?'
a = qas_dict.get(q)
print(a)

把 qas_dict 看作一个简易的 key-value 数据库，用精准的问题去查询可以得到问题的答案

France

但如果问题和数据库中的 key 有点偏差（字符串不相等），就无法检索到这个问题的答案，这也是普通的数据库和向量数据库的最主要的差别之一，为了保证相似的问题也可以检索到正确的答案，我们可以使用向量数据库。

# 这里使用 CPU 版本的 faiss
pip install faiss-cpu

下面是完整的示例代码

import json

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

model_path = "/path/to/all-MiniLM-L6-v2"
model = SentenceTransformer(model_path)


def load_qa_data(data_path):
    with open(data_path, "r") as f:
        dataset = json.load(f)
    qas = [
        (qas["question"], qas["answers"][0]["text"])
        for data in dataset["data"]
        for paragraphs in data["paragraphs"]
        for qas in paragraphs["qas"]
        if qas["answers"]
    ]
    return np.array(qas)


def str_to_vec(sentence_list):
    embedding = model.encode(sentence_list)
    return embedding


def build_faiss_index(vectors, nlist=100, pq_m=8):
    d = vectors.shape[1]
    quantizer = faiss.IndexFlatL2(d)
    index = faiss.IndexIVFPQ(quantizer, d, nlist, 8, pq_m)
    index.train(vectors)
    index.add(vectors)
    return index

k = 10

qas = load_qa_data("dev-v2.0.json")
q, a = [qa[0] for qa in qas], [qa[1] for qa in qas]
q_vec = str_to_vec(q)
print(q_vec.shape)
index = build_faiss_index(q_vec)

query_vector = str_to_vec(["What country does Normandy belong to?"])
distances, indices = index.search(query_vector, k)

for i in range(k):
    print(f"==> distance: {distances[0][i]:.4f}, indice: {indices[0][i]}, {q[i]}")

运行的结果：

==> distance: 0.5488, indice: 0, In what country is Normandy located?
==> distance: 0.6171, indice: 10, When were the Normans in Normandy?
==> distance: 0.6253, indice: 6, From which countries did the Norse originate?
==> distance: 0.6960, indice: 5255, Who was the Norse leader?
==> distance: 0.7582, indice: 12, What century did the Normans first gain their separate identity?
==> distance: 0.7674, indice: 4725, Who was the duke in the battle of Hastings?
==> distance: 0.7722, indice: 22, Who ruled the duchy of Normandy
==> distance: 0.7759, indice: 4488, What religion were the Normans
==> distance: 0.7777, indice: 5259, What is the original meaning of the word Norman?
==> distance: 0.8038, indice: 4706, When was the Latin version of the word Norman first recorded?

可以发现即使是把问题换成了：

“What country does Normandy belong to?”

也仍然能够匹配到在384维的空间内与它最接近的一个句子（L2距离为0.5488）：

“In what country is Normandy located?”

我们就可以通过它的索引 0 找到对应的答案了。