flamingo pytorch下载 - flamingo pytorch源码下载

？火烈鸟 - Pytorch

在 Pytorch 中实现 Flamingo，这是最先进的少镜头视觉问答注意力网络。它将包括感知器重采样器（除了媒体嵌入之外，还包括学习的查询贡献要关注的键/值的方案）、专门的屏蔽交叉注意块，以及最后交叉注意末端的 tanh 门控 +相应的前馈块

雅尼克·基尔彻演讲

安装

$ pip install flamingo-pytorch

用法

 import torch
from flamingo_pytorch import PerceiverResampler

perceive = PerceiverResampler (
    dim = 1024 ,
    depth = 2 ,
    dim_head = 64 ,
    heads = 8 ,
    num_latents = 64 ,    # the number of latents to shrink your media sequence to, perceiver style
    num_time_embeds = 4  # say you have 4 images maximum in your dialogue
)

medias = torch . randn ( 1 , 2 , 256 , 1024 ) # (batch, time, sequence length, dimension)
perceived = perceive ( medias ) # (1, 2, 64, 1024) - (batch, time, num latents, dimension)

然后，在巨型语言模型中以不同的间隔插入GatedCrossAttentionBlock 。然后，您的文本将关注上方感知到的媒体

导出media_locations布尔张量的推荐方法是为媒体分配一个特殊的令牌 id，然后在大型语言模型的开头执行media_locations = text_id == media_token_id

 import torch
from flamingo_pytorch import GatedCrossAttentionBlock

cross_attn = GatedCrossAttentionBlock (
    dim = 1024 ,
    dim_head = 64 ,
    heads = 8
)

text = torch . randn ( 1 , 512 , 1024 )
perceived = torch . randn ( 1 , 2 , 64 , 1024 )

media_locations = torch . randint ( 0 , 2 , ( 1 , 512 )). bool ()

text = cross_attn (
    text ,
    perceived ,
    media_locations = media_locations
)

就是这样！

您所需要的就是关注。

Flamingo + PaLM 的完整工作示例？？？

与PaLM集成

首先为视觉编码器安装vit-pytorch

$ pip install vit-pytorch

然后

 from vit_pytorch . vit import ViT
from vit_pytorch . extractor import Extractor

vit = ViT (
    image_size = 256 ,
    patch_size = 32 ,
    num_classes = 1000 ,
    dim = 1024 ,
    depth = 6 ,
    heads = 16 ,
    mlp_dim = 2048 ,
    dropout = 0.1 ,
    emb_dropout = 0.1
)

vit = Extractor ( vit , return_embeddings_only = True )

# first take your trained image encoder and wrap it in an adapter that returns the image embeddings
# here we use the ViT from the vit-pytorch library

import torch
from flamingo_pytorch import FlamingoPaLM

# a PaLM language model, the 540 billion parameter model from google that shows signs of general intelligence

flamingo_palm = FlamingoPaLM (
    num_tokens = 20000 ,          # number of tokens
    dim = 1024 ,                  # dimensions
    depth = 12 ,                  # depth
    heads = 8 ,                   # attention heads
    dim_head = 64 ,               # dimension per attention head
    img_encoder = vit ,           # plugin your image encoder (this can be optional if you pass in the image embeddings separately, but probably want to train end to end given the perceiver resampler)
    media_token_id = 3 ,          # the token id representing the [media] or [image]
    cross_attn_every = 3 ,        # how often to cross attend
    perceiver_num_latents = 64 ,  # perceiver number of latents, should be smaller than the sequence length of the image tokens
    perceiver_depth = 2          # perceiver resampler depth
)

# train your PaLM as usual

text = torch . randint ( 0 , 20000 , ( 2 , 512 ))

palm_logits = flamingo_palm ( text )

# after much training off the regular PaLM logits
# now you are ready to train Flamingo + PaLM
# by passing in images, it automatically freezes everything but the perceiver and cross attention blocks, as in the paper

dialogue = torch . randint ( 0 , 20000 , ( 4 , 512 ))
images = torch . randn ( 4 , 2 , 3 , 256 , 256 )

flamingo_logits = flamingo_palm ( dialogue , images )

# do your usual cross entropy loss

如果你不仅仅考虑图像，那么这一切的发展方向就很明显了。

盗梦空间

为了事实的正确性，想象一下如果要使用最先进的检索语言模型作为基础，这个系统会是什么样子。

引文

 @article { Alayrac2022Flamingo ,
    title   = { Flamingo: a Visual Language Model for Few-Shot Learning } ,
    author  = { Jean-Baptiste Alayrac et al } ,
    year    = { 2022 }
}

 @inproceedings { Chowdhery2022PaLMSL ,
    title   = { PaLM: Scaling Language Modeling with Pathways } ,
    author  = {Aakanksha Chowdhery and Sharan Narang and Jacob Devlin and Maarten Bosma and Gaurav Mishra and Adam Roberts and Paul Barham and Hyung Won Chung and Charles Sutton and Sebastian Gehrmann and Parker Schuh and Kensen Shi and Sasha Tsvyashchenko and Joshua Maynez and Abhishek Rao and Parker Barnes and Yi Tay and Noam M. Shazeer and Vinodkumar Prabhakaran and Emily Reif and Nan Du and Benton C. Hutchinson and Reiner Pope and James Bradbury and Jacob Austin and Michael Isard and Guy Gur-Ari and Pengcheng Yin and Toju Duke and Anselm Levskaya and Sanjay Ghemawat and Sunipa Dev and Henryk Michalewski and Xavier Garc{'i}a and Vedant Misra and Kevin Robinson and Liam Fedus and Denny Zhou and Daphne Ippolito and David Luan and Hyeontaek Lim and Barret Zoph and Alexander Spiridonov and Ryan Sepassi and David Dohan and Shivani Agrawal and Mark Omernick and Andrew M. Dai and Thanumalayan Sankaranarayana Pillai and Marie Pellat and Aitor Lewkowycz and Erica Oliveira Moreira and Rewon Child and Oleksandr Polozov and Katherine Lee and Zongwei Zhou and Xuezhi Wang and Brennan Saeta and Mark Diaz and Orhan Firat and Michele Catasta and Jason Wei and Kathleen S. Meier-Hellstern and Douglas Eck and Jeff Dean and Slav Petrov and Noah Fiedel},
    year    = { 2022 }
}

展开