spacy sentence bert下载 - spacy sentence bert源代码下载

spacy sentence bert

其他源码

v0.1.2

下载

spaCy 的句子 BERT

该软件包将句子转换器（也称为句子 BERT）直接包装在 spaCy 中。您可以将任何 spaCy 模型中提供的向量替换为专门针对语义相似性进行调整的向量。

正如 STS 基准所示，建议使用以下模型来分析句子相似性。请记住， sentence-transformers最大序列长度配置为 128。因此，对于较长的文本，它可能更适合与其他模型（例如通用句子编码器）一起使用。

安装

兼容性：

蟒蛇3.7/3.8/3.9/3.10
spaCy>=3.0.0,<4.0.0，上次测试版本为 3.5
句子转换器：在 2.2.2 版本上测试

要安装此软件包，您可以运行以下命令之一：

pip install spacy-sentence-bert
pip install git+https://github.com/MartinoMensio/spacy-sentence-bert.git

您可以使用 pip 从 GitHub 安装独立的 spaCy 包。如果安装独立包，则可以使用spacy.load API 直接加载语言模型，无需添加管道阶段。此表采用 Sentence Transformers 文档中列出的模型，并显示一些统计数据以及安装独立模型的说明。如果您不想安装独立模型，您仍然可以通过添加管道阶段来使用它们（见下文）。

句子-BERT名称	模型名称	方面	语言	STS基准测试	独立安装
`paraphrase-distilroberta-base-v1`	`en_paraphrase_distilroberta_base_v1`	第768章	zh	81.81	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_paraphrase_distilroberta_base_v1-0.1.2.tar.gz#en_paraphrase_distilroberta_base_v1-0.1.2`
`paraphrase-xlm-r-multilingual-v1`	`xx_paraphrase_xlm_r_multilingual_v1`	第768章	50+	83.50	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/xx_paraphrase_xlm_r_multilingual_v1-0.1.2.tar.gz#xx_paraphrase_xlm_r_multilingual_v1-0.1.2`
`stsb-roberta-large`	`en_stsb_roberta_large`	1024	zh	86.39	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_stsb_roberta_large-0.1.2.tar.gz#en_stsb_roberta_large-0.1.2`
`stsb-roberta-base`	`en_stsb_roberta_base`	第768章	zh	85.44	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_stsb_roberta_base-0.1.2.tar.gz#en_stsb_roberta_base-0.1.2`
`stsb-bert-large`	`en_stsb_bert_large`	1024	zh	85.29	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_stsb_bert_large-0.1.2.tar.gz#en_stsb_bert_large-0.1.2`
`stsb-distilbert-base`	`en_stsb_distilbert_base`	第768章	zh	85.16	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_stsb_distilbert_base-0.1.2.tar.gz#en_stsb_distilbert_base-0.1.2`
`stsb-bert-base`	`en_stsb_bert_base`	第768章	zh	85.14	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_stsb_bert_base-0.1.2.tar.gz#en_stsb_bert_base-0.1.2`
`nli-bert-large`	`en_nli_bert_large`	1024	zh	79.19	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_bert_large-0.1.2.tar.gz#en_nli_bert_large-0.1.2`
`nli-distilbert-base`	`en_nli_distilbert_base`	第768章	zh	78.69	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_distilbert_base-0.1.2.tar.gz#en_nli_distilbert_base-0.1.2`
`nli-roberta-large`	`en_nli_roberta_large`	1024	zh	78.69	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_roberta_large-0.1.2.tar.gz#en_nli_roberta_large-0.1.2`
`nli-bert-large-max-pooling`	`en_nli_bert_large_max_pooling`	1024	zh	78.41	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_bert_large_max_pooling-0.1.2.tar.gz#en_nli_bert_large_max_pooling-0.1.2`
`nli-bert-large-cls-pooling`	`en_nli_bert_large_cls_pooling`	1024	zh	78.29	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_bert_large_cls_pooling-0.1.2.tar.gz#en_nli_bert_large_cls_pooling-0.1.2`
`nli-distilbert-base-max-pooling`	`en_nli_distilbert_base_max_pooling`	第768章	zh	77.61	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_distilbert_base_max_pooling-0.1.2.tar.gz#en_nli_distilbert_base_max_pooling-0.1.2`
`nli-roberta-base`	`en_nli_roberta_base`	第768章	zh	77.49	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_roberta_base-0.1.2.tar.gz#en_nli_roberta_base-0.1.2`
`nli-bert-base-max-pooling`	`en_nli_bert_base_max_pooling`	第768章	zh	77.21	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_bert_base_max_pooling-0.1.2.tar.gz#en_nli_bert_base_max_pooling-0.1.2`
`nli-bert-base`	`en_nli_bert_base`	第768章	zh	77.12	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_bert_base-0.1.2.tar.gz#en_nli_bert_base-0.1.2`
`nli-bert-base-cls-pooling`	`en_nli_bert_base_cls_pooling`	第768章	zh	76.30	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nli_bert_base_cls_pooling-0.1.2.tar.gz#en_nli_bert_base_cls_pooling-0.1.2`
`average_word_embeddings_glove.6B.300d`	`en_average_word_embeddings_glove.6B.300d`	第768章	zh	61.77	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_average_word_embeddings_glove.6B.300d-0.1.2.tar.gz#en_average_word_embeddings_glove.6B.300d-0.1.2`
`average_word_embeddings_komninos`	`en_average_word_embeddings_komninos`	第768章	zh	61.56	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_average_word_embeddings_komninos-0.1.2.tar.gz#en_average_word_embeddings_komninos-0.1.2`
`average_word_embeddings_levy_dependency`	`en_average_word_embeddings_levy_dependency`	第768章	zh	59.22	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_average_word_embeddings_levy_dependency-0.1.2.tar.gz#en_average_word_embeddings_levy_dependency-0.1.2`
`average_word_embeddings_glove.840B.300d`	`en_average_word_embeddings_glove.840B.300d`	第768章	zh	52.54	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_average_word_embeddings_glove.840B.300d-0.1.2.tar.gz#en_average_word_embeddings_glove.840B.300d-0.1.2`
`quora-distilbert-base`	`en_quora_distilbert_base`	第768章	zh	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_quora_distilbert_base-0.1.2.tar.gz#en_quora_distilbert_base-0.1.2`
`quora-distilbert-multilingual`	`xx_quora_distilbert_multilingual`	第768章	50+	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/xx_quora_distilbert_multilingual-0.1.2.tar.gz#xx_quora_distilbert_multilingual-0.1.2`
`msmarco-distilroberta-base-v2`	`en_msmarco_distilroberta_base_v2`	第768章	zh	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_msmarco_distilroberta_base_v2-0.1.2.tar.gz#en_msmarco_distilroberta_base_v2-0.1.2`
`msmarco-roberta-base-v2`	`en_msmarco_roberta_base_v2`	第768章	zh	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_msmarco_roberta_base_v2-0.1.2.tar.gz#en_msmarco_roberta_base_v2-0.1.2`
`msmarco-distilbert-base-v2`	`en_msmarco_distilbert_base_v2`	第768章	zh	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_msmarco_distilbert_base_v2-0.1.2.tar.gz#en_msmarco_distilbert_base_v2-0.1.2`
`nq-distilbert-base-v1`	`en_nq_distilbert_base_v1`	第768章	zh	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_nq_distilbert_base_v1-0.1.2.tar.gz#en_nq_distilbert_base_v1-0.1.2`
`distiluse-base-multilingual-cased-v2`	`xx_distiluse_base_multilingual_cased_v2`	第512章	50+	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/xx_distiluse_base_multilingual_cased_v2-0.1.2.tar.gz#xx_distiluse_base_multilingual_cased_v2-0.1.2`
`stsb-xlm-r-multilingual`	`xx_stsb_xlm_r_multilingual`	第768章	50+	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/xx_stsb_xlm_r_multilingual-0.1.2.tar.gz#xx_stsb_xlm_r_multilingual-0.1.2`
`T-Systems-onsite/cross-en-de-roberta-sentence-transformer`	`xx_cross_en_de_roberta_sentence_transformer`	第768章	恩、德	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/xx_cross_en_de_roberta_sentence_transformer-0.1.2.tar.gz#xx_cross_en_de_roberta_sentence_transformer-0.1.2`
`LaBSE`	`xx_LaBSE`	第768章	109	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/xx_LaBSE-0.1.2.tar.gz#xx_LaBSE-0.1.2`
`allenai-specter`	`en_allenai_specter`	第768章	zh	不适用	`pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_allenai_specter-0.1.2.tar.gz#en_allenai_specter-0.1.2`

如果您的模型不在此列表中（例如xlm-r-base-en-ko-nli-ststb ），您仍然可以将其与此库一起使用，但不能作为独立语言。您将需要添加正确配置的管道阶段（请参阅下面的nlp.add_pipe API）。

用法

有不同的方法来加载sentence-bert的模型。

spacy.load API：您需要安装上表中的模型之一
spacy_sentence_bert.load_model ：您可以加载上表中的模型之一，而无需安装独立包
nlp.add_pipe API：您可以在nlp对象之上加载任何sentence-bert模型

`spacy.load` API

从 GitHub 安装的独立模型（例如，从上表中， pip install https://github.com/MartinoMensio/spacy-sentence-bert/releases/download/v0.1.2/en_stsb_roberta_large-0.1.2.tar.gz#en_stsb_roberta_large-0.1.2 ），您可以使用 spaCy API 直接加载模型：

 import spacy
nlp = spacy . load ( 'en_stsb_roberta_large' )

`spacy_sentence_bert.load_model` API

使用以下方法，无需安装独立模型即可获得相同的结果：

 import spacy_sentence_bert
nlp = spacy_sentence_bert . load_model ( 'en_stsb_roberta_large' )

`nlp.add_pipe` API

如果您想在现有 Language 对象上使用句子嵌入之一，可以使用nlp.add_pipe方法。如果您想使用上表中未列出的语言模型，这也适用。只要确保句子转换器支持它即可。

 import spacy
nlp = spacy . blank ( 'en' )
nlp . add_pipe ( 'sentence_bert' , config = { 'model_name' : 'allenai-specter' })
nlp . pipe_names

首次使用模型时，会将句子 BERT 下载到环境变量中TORCH_HOME定义的文件夹（默认~/.cache/torch ）。

加载模型后，通过 spaCy 的vector属性和similarity方法使用它：

 # get two documents
doc_1 = nlp ( 'Hi there, how are you?' )
doc_2 = nlp ( 'Hello there, how are you doing today?' )
# get the vector of the Doc, Span or Token
print ( doc_1 . vector . shape )
print ( doc_1 [ 3 ]. vector . shape )
print ( doc_1 [ 2 : 4 ]. vector . shape )
# or use the similarity method that is based on the vectors, on Doc, Span or Token
print ( doc_1 . similarity ( doc_2 [ 0 : 7 ]))

实用程序

构建并上传

VERSION=0.1.2
# build the standalone models (17)
./build_models.sh
# build the archive at dist/spacy_sentence_bert-${VERSION}.tar.gz
python setup.py sdist
# upload to pypi
twine upload dist/spacy_sentence_bert- ${VERSION} .tar.gz

展开

附加信息