axial attention

軸向注意力

Pytorch 中軸向注意力的實現。一種簡單但強大的技術，可以有效地處理多維資料。它為我和許多其他研究人員創造了奇蹟。

只需在資料中新增一些位置編碼並將其傳遞到這個方便的類別中，指定哪個維度被視為嵌入，以及要旋轉多少個軸向維度。所有的排列、重塑，都會為你處理好。

這篇論文其實被拒絕了，理由是過於簡單。然而，它已經成功地應用於許多應用中，其中包括天氣預報、全注意力影像分割。只是去表演。

安裝

$ pip install axial_attention

用法

影像

 import torch
from axial_attention import AxialAttention

img = torch . randn ( 1 , 3 , 256 , 256 )

attn = AxialAttention (
    dim = 3 ,               # embedding dimension
    dim_index = 1 ,         # where is the embedding dimension
    dim_heads = 32 ,        # dimension of each head. defaults to dim // heads if not supplied
    heads = 1 ,             # number of heads for multi-head attention
    num_dimensions = 2 ,    # number of axial dimensions (images is 2, video is 3, or more)
    sum_axial_out = True   # whether to sum the contributions of attention on each axis, or to run the input through them sequentially. defaults to true
)

attn ( img ) # (1, 3, 256, 256)

通道最後影像潛伏

 import torch
from axial_attention import AxialAttention

img = torch . randn ( 1 , 20 , 20 , 512 )

attn = AxialAttention (
    dim = 512 ,           # embedding dimension
    dim_index = - 1 ,      # where is the embedding dimension
    heads = 8 ,           # number of heads for multi-head attention
    num_dimensions = 2 ,  # number of axial dimensions (images is 2, video is 3, or more)
)

attn ( img ) # (1, 20, 20 ,512)

影片

 import torch
from axial_attention import AxialAttention

video = torch . randn ( 1 , 5 , 128 , 256 , 256 )

attn = AxialAttention (
    dim = 128 ,           # embedding dimension
    dim_index = 2 ,       # where is the embedding dimension
    heads = 8 ,           # number of heads for multi-head attention
    num_dimensions = 3 ,  # number of axial dimensions (images is 2, video is 3, or more)
)

attn ( video ) # (1, 5, 128, 256, 256)

影像轉換器，具有可逆網絡

 import torch
from torch import nn
from axial_attention import AxialImageTransformer

conv1x1 = nn . Conv2d ( 3 , 128 , 1 )

transformer = AxialImageTransformer (
    dim = 128 ,
    depth = 12 ,
    reversible = True
)

img = torch . randn ( 1 , 3 , 512 , 512 )

transformer ( conv1x1 ( img )) # (1, 3, 512, 512)

帶軸向定位嵌入

 import torch
from axial_attention import AxialAttention , AxialPositionalEmbedding

img = torch . randn ( 1 , 512 , 20 , 20 )

attn = AxialAttention (
    dim = 512 ,
    heads = 8 ,
    dim_index = 1
)

pos_emb = AxialPositionalEmbedding (
    dim = 512 ,
    shape = ( 20 , 20 )
)

img = pos_emb ( img )  # (1, 512, 20, 20)  - now positionally embedded
img = attn ( img )     # (1, 512, 20, 20) 

引文

 @misc { ho2019axial ,
    title  = { Axial Attention in Multidimensional Transformers } ,
    author = { Jonathan Ho and Nal Kalchbrenner and Dirk Weissenborn and Tim Salimans } ,
    year   = { 2019 } ,
    archivePrefix = { arXiv }
}

 @misc { wang2020axialdeeplab ,
    title   = { Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation } ,
    author  = { Huiyu Wang and Yukun Zhu and Bradley Green and Hartwig Adam and Alan Yuille and Liang-Chieh Chen } ,
    year    = { 2020 } ,
    eprint  = { 2003.07853 } ,
    archivePrefix = { arXiv } ,
    primaryClass = { cs.CV }
}

 @inproceedings { huang2019ccnet ,
    title   = { Ccnet: Criss-cross attention for semantic segmentation } ,
    author  = { Huang, Zilong and Wang, Xinggang and Huang, Lichao and Huang, Chang and Wei, Yunchao and Liu, Wenyu } ,
    booktitle = { Proceedings of the IEEE/CVF International Conference on Computer Vision } ,
    pages   = { 603--612 } ,
    year    = { 2019 }
}