transformers ruby
1.0.0
？ルビー用の最先端の変圧器
高速推論については、情報提供者をご覧ください
まず、torch.rbをインストールします。
次に、この行をアプリケーションのGemfileに追加します。
gem "transformers-rb" 埋め込み
まばらな埋め込み
再ランキング
ドキュメント
sentences = [ "This is an example sentence" , "Each sentence is converted" ]
model = Transformers . pipeline ( "embedding" , "sentence-transformers/all-MiniLM-L6-v2" )
embeddings = model . ( sentences )ドキュメント
query = "How many people live in London?"
docs = [ "Around 9 Million people live in London" , "London is known for its financial district" ]
model = Transformers . pipeline ( "embedding" , "sentence-transformers/multi-qa-MiniLM-L6-cos-v1" )
query_embedding = model . ( query )
doc_embeddings = model . ( docs )
scores = doc_embeddings . map { | e | e . zip ( query_embedding ) . sum { | d , q | d * q } }
doc_score_pairs = docs . zip ( scores ) . sort_by { | d , s | - s }ドキュメント
sentences = [ "This is an example sentence" , "Each sentence is converted" ]
model = Transformers . pipeline ( "embedding" , "sentence-transformers/all-mpnet-base-v2" )
embeddings = model . ( sentences )ドキュメント
sentences = [ "This is an example sentence" , "Each sentence is converted" ]
model = Transformers . pipeline ( "embedding" , "sentence-transformers/paraphrase-MiniLM-L6-v2" )
embeddings = model . ( sentences )ドキュメント
query_prefix = "Represent this sentence for searching relevant passages: "
input = [
"The dog is barking" ,
"The cat is purring" ,
query_prefix + "puppy"
]
model = Transformers . pipeline ( "embedding" , "mixedbread-ai/mxbai-embed-large-v1" )
embeddings = model . ( input )ドキュメント
sentences = [ "That is a happy person" , "That is a very happy person" ]
model = Transformers . pipeline ( "embedding" , "thenlper/gte-small" )
embeddings = model . ( sentences )ドキュメント
doc_prefix = "passage: "
query_prefix = "query: "
input = [
doc_prefix + "Ruby is a programming language created by Matz" ,
query_prefix + "Ruby creator"
]
model = Transformers . pipeline ( "embedding" , "intfloat/e5-base-v2" )
embeddings = model . ( input )ドキュメント
query_prefix = "Represent this sentence for searching relevant passages: "
input = [
"The dog is barking" ,
"The cat is purring" ,
query_prefix + "puppy"
]
model = Transformers . pipeline ( "embedding" , "BAAI/bge-base-en-v1.5" )
embeddings = model . ( input )ドキュメント
query_prefix = "Represent this sentence for searching relevant passages: "
input = [
"The dog is barking" ,
"The cat is purring" ,
query_prefix + "puppy"
]
model = Transformers . pipeline ( "embedding" , "Snowflake/snowflake-arctic-embed-m-v1.5" )
embeddings = model . ( input , pooling : "cls" )ドキュメント
docs = [ "The dog is barking" , "The cat is purring" , "The bear is growling" ]
model_id = "opensearch-project/opensearch-neural-sparse-encoding-v1"
model = Transformers :: AutoModelForMaskedLM . from_pretrained ( model_id )
tokenizer = Transformers :: AutoTokenizer . from_pretrained ( model_id )
special_token_ids = tokenizer . special_tokens_map . map { | _ , token | tokenizer . vocab [ token ] }
feature = tokenizer . ( docs , padding : true , truncation : true , return_tensors : "pt" , return_token_type_ids : false )
output = model . ( ** feature ) [ 0 ]
values , _ = Torch . max ( output * feature [ :attention_mask ] . unsqueeze ( - 1 ) , dim : 1 )
values = Torch . log ( 1 + Torch . relu ( values ) )
values [ 0 .. , special_token_ids ] = 0
embeddings = values . to_aドキュメント
query = "How many people live in London?"
docs = [ "Around 9 Million people live in London" , "London is known for its financial district" ]
model = Transformers . pipeline ( "reranking" , "mixedbread-ai/mxbai-rerank-base-v1" )
result = model . ( query , docs )ドキュメント
query = "How many people live in London?"
docs = [ "Around 9 Million people live in London" , "London is known for its financial district" ]
model = Transformers . pipeline ( "reranking" , "BAAI/bge-reranker-base" )
result = model . ( query , docs ) 埋め込み
embed = Transformers . pipeline ( "embedding" )
embed . ( "We are very happy to show you the ? Transformers library." )再ランキング
rerank = Informers . pipeline ( "reranking" )
rerank . ( "Who created Ruby?" , [ "Matz created Ruby" , "Another doc" ] )指名されたエンティティ認識
ner = Transformers . pipeline ( "ner" )
ner . ( "Ruby is a programming language created by Matz" )感情分析
classifier = Transformers . pipeline ( "sentiment-analysis" )
classifier . ( "We are very happy to show you the ? Transformers library." )質問に答える
qa = Transformers . pipeline ( "question-answering" )
qa . ( question : "Who invented Ruby?" , context : "Ruby is a programming language created by Matz" )特徴抽出
extractor = Transformers . pipeline ( "feature-extraction" )
extractor . ( "We are very happy to show you the ? Transformers library." )画像分類
classifier = Transformers . pipeline ( "image-classification" )
classifier . ( "image.jpg" )画像機能抽出
extractor = Transformers . pipeline ( "image-feature-extraction" )
extractor . ( "image.jpg" ) このライブラリは、トランスフォーマーPython APIに従います。現在、次のモデルアーキテクチャがサポートされています。
Changelogを表示します
誰もがこのプロジェクトの改善を支援することをお勧めします。ここにあなたが助けることができるいくつかの方法があります：
開発を始めるには：
git clone https://github.com/ankane/transformers-ruby.git
cd transformers-ruby
bundle install
bundle exec rake download:files
bundle exec rake test
