perfusion pytorch下载 - perfusion pytorch源码下载

perfusion pytorch

Ai源码

0.1.23

下载

灌注 - Pytorch

锁键一级编辑的实现。项目页面

本文的卖点是每个添加概念的额外参数极低，低至 100kb。

他们似乎成功地应用了 LLM 记忆编辑论文中的 Rank-1 编辑技术，并进行了一些改进。他们还发现键决定新概念的“位置”，而值决定“什么”，并提出将本地/全局键锁定到超类概念（同时学习值）。

对于那里的研究人员来说，如果这篇论文通过，这个存储库中的工具应该适用于任何其他使用交叉注意调节的文本到<insert modality>网络。只是一个想法

欣赏

StabilityAI 以及我的其他赞助商的慷慨赞助
Yoad Tewel 负责多次代码审查和澄清电子邮件
Brad Vidler 预先计算稳定扩散 1.5 中使用的 CLIP 的协方差矩阵！
OpenClip 的所有维护者，感谢他们的 SOTA 开源对比学习文本图像模型

安装

$ pip install perfusion-pytorch

用法

 import torch
from torch import nn

from perfusion_pytorch import Rank1EditModule

to_keys = nn . Linear ( 768 , 320 , bias = False )
to_values = nn . Linear ( 768 , 320 , bias = False )

wrapped_to_keys = Rank1EditModule (
    to_keys ,
    is_key_proj = True
)

wrapped_to_values = Rank1EditModule (
    to_values
)

text_enc = torch . randn ( 4 , 77 , 768 )                  # regular input
text_enc_with_superclass = torch . randn ( 4 , 77 , 768 )  # init_input in algorithm 1, for key-locking
concept_indices = torch . randint ( 0 , 77 , ( 4 ,))        # index where the concept or superclass concept token is in the sequence
key_pad_mask = torch . ones ( 4 , 77 ). bool ()

keys = wrapped_to_keys (
    text_enc ,
    concept_indices = concept_indices ,
    text_enc_with_superclass = text_enc_with_superclass ,
)

values = wrapped_to_values (
    text_enc ,
    concept_indices = concept_indices ,
    text_enc_with_superclass = text_enc_with_superclass ,
)

# after much training ...

wrapped_to_keys . eval ()
wrapped_to_values . eval ()

keys = wrapped_to_keys ( text_enc )

values = wrapped_to_values ( text_enc )

该存储库还包含一个EmbeddingWrapper ，可以轻松训练新概念（以及最终对多个概念进行推理）

 import torch
from torch import nn

from perfusion_pytorch import EmbeddingWrapper

embed = nn . Embedding ( 49408 , 512 ) # open clip embedding, somewhere in the module tree of stable diffusion

# wrap it, and will automatically create a new concept for learning, based on the superclass embed string

wrapped_embed = EmbeddingWrapper (
    embed ,
    superclass_string = 'dog'
)

# now just pass in your prompts with the superclass id

embeds_with_new_concept , embeds_with_superclass , embed_mask , concept_indices = wrapped_embed ([
    'a portrait of dog' ,
    'dog running through a green field' ,
    'a man walking his dog'
]) # (3, 77, 512), (3, 77, 512), (3, 77), (3,)

# now pass both embeds through clip text transformer
# the embed_mask needs to be passed to the cross attention as key padding mask

如果您可以识别稳定扩散实例中的CLIP实例，您还可以将其直接传递到OpenClipEmbedWrapper以获得交叉注意层向前所需的一切

前任。

 from perfusion_pytorch import OpenClipEmbedWrapper

texts = [
    'a portrait of dog' ,
    'dog running through a green field' ,
    'a man walking his dog'
]

wrapped_clip_with_new_concept = OpenClipEmbedWrapper (
    stable_diffusion . path . to . clip ,
    superclass_string = 'dog'
)

text_enc , superclass_enc , mask , indices = wrapped_clip_with_new_concept ( texts )

# (3, 77, 512), (3, 77, 512), (3, 77), (3,)

托多

引文

 @article { Tewel2023KeyLockedRO ,
    title   = { Key-Locked Rank One Editing for Text-to-Image Personalization } ,
    author  = { Yoad Tewel and Rinon Gal and Gal Chechik and Yuval Atzmon } ,
    journal = { ACM SIGGRAPH 2023 Conference Proceedings } ,
    year    = { 2023 } ,
    url     = { https://api.semanticscholar.org/CorpusID:258436985 }
}

 @inproceedings { Meng2022LocatingAE ,
    title   = { Locating and Editing Factual Associations in GPT } ,
    author  = { Kevin Meng and David Bau and Alex Andonian and Yonatan Belinkov } ,
    booktitle = { Neural Information Processing Systems } ,
    year    = { 2022 } ,
    url     = { https://api.semanticscholar.org/CorpusID:255825985 }
}