video diffusion pytorch下載 - video diffusion pytorch原始碼下載

機器想像的煙火

這些煙火不存在

視訊擴散 - Pytorch

文字轉視頻，它正在發生！官方項目頁面

視訊擴散模型的實現，Jonathan Ho 的新論文，將 DDPM 擴展到視訊生成 - 在 Pytorch 中。它使用特殊的時空因子 U 網，將生成從 2D 影像擴展到 3D 視頻

地位

14k 用於困難移動 mnist（比 NUWA 收斂得更快更好）- wip

上述實驗之所以成為可能，全歸功於 Stability.ai 提供的資源

任何文本到視訊合成的新開發都將集中在 Imagen-pytorch

安裝

$ pip install video-diffusion-pytorch

用法

 import torch
from video_diffusion_pytorch import Unet3D , GaussianDiffusion

model = Unet3D (
    dim = 64 ,
    dim_mults = ( 1 , 2 , 4 , 8 )
)

diffusion = GaussianDiffusion (
    model ,
    image_size = 32 ,
    num_frames = 5 ,
    timesteps = 1000 ,   # number of steps
    loss_type = 'l1'    # L1 or L2
)

videos = torch . randn ( 1 , 3 , 5 , 32 , 32 ) # video (batch, channels, frames, height, width) - normalized from -1 to +1
loss = diffusion ( videos )
loss . backward ()
# after a lot of training

sampled_videos = diffusion . sample ( batch_size = 4 )
sampled_videos . shape # (4, 3, 5, 32, 32)

為了對文本進行調節，他們首先透過 BERT-large 傳遞標記化文本來導出文本嵌入。然後你只需要像這樣訓練它

 import torch
from video_diffusion_pytorch import Unet3D , GaussianDiffusion

model = Unet3D (
    dim = 64 ,
    cond_dim = 64 ,
    dim_mults = ( 1 , 2 , 4 , 8 )
)

diffusion = GaussianDiffusion (
    model ,
    image_size = 32 ,
    num_frames = 5 ,
    timesteps = 1000 ,   # number of steps
    loss_type = 'l1'    # L1 or L2
)

videos = torch . randn ( 2 , 3 , 5 , 32 , 32 ) # video (batch, channels, frames, height, width)
text = torch . randn ( 2 , 64 )             # assume output of BERT-large has dimension of 64

loss = diffusion ( videos , cond = text )
loss . backward ()
# after a lot of training

sampled_videos = diffusion . sample ( cond = text )
sampled_videos . shape # (2, 3, 5, 32, 32)

如果您打算使用 BERT-base 進行文字調節，您也可以直接將視訊描述作為字串傳遞

 import torch
from video_diffusion_pytorch import Unet3D , GaussianDiffusion

model = Unet3D (
    dim = 64 ,
    use_bert_text_cond = True ,  # this must be set to True to auto-use the bert model dimensions
    dim_mults = ( 1 , 2 , 4 , 8 ),
)

diffusion = GaussianDiffusion (
    model ,
    image_size = 32 ,    # height and width of frames
    num_frames = 5 ,     # number of video frames
    timesteps = 1000 ,   # number of steps
    loss_type = 'l1'    # L1 or L2
)

videos = torch . randn ( 3 , 3 , 5 , 32 , 32 ) # video (batch, channels, frames, height, width)

text = [
    'a whale breaching from afar' ,
    'young girl blowing out candles on her birthday cake' ,
    'fireworks with blue and green sparkles'
]

loss = diffusion ( videos , cond = text )
loss . backward ()
# after a lot of training

sampled_videos = diffusion . sample ( cond = text , cond_scale = 2 )
sampled_videos . shape # (3, 3, 5, 32, 32)

訓練

該儲存庫還包含一個方便的Trainer類，用於在gifs資料夾上進行訓練。每個gif必須具有正確的尺寸image_size和num_frames 。

 import torch
from video_diffusion_pytorch import Unet3D , GaussianDiffusion , Trainer

model = Unet3D (
    dim = 64 ,
    dim_mults = ( 1 , 2 , 4 , 8 ),
)

diffusion = GaussianDiffusion (
    model ,
    image_size = 64 ,
    num_frames = 10 ,
    timesteps = 1000 ,   # number of steps
    loss_type = 'l1'    # L1 or L2
). cuda ()

trainer = Trainer (
    diffusion ,
    './data' ,                         # this folder path needs to contain all your training data, as .gif files, of correct image size and number of frames
    train_batch_size = 32 ,
    train_lr = 1e-4 ,
    save_and_sample_every = 1000 ,
    train_num_steps = 700000 ,         # total training steps
    gradient_accumulate_every = 2 ,    # gradient accumulation steps
    ema_decay = 0.995 ,                # exponential moving average decay
    amp = True                        # turn on mixed precision
)

trainer . train ()

樣本影片（如gif檔案）將定期保存到./results中，擴散模型參數也是如此。

協同訓練圖像和視頻

論文中的主張之一是，透過進行因子時空注意力，可以迫使網路同時關注當前的圖像和視頻，從而獲得更好的結果。

目前尚不清楚他們是如何實現這一目標的，但我進一步猜測。

為了將一定比例的批量視訊樣本的注意力吸引到當前時刻，只需在擴散前向方法上傳遞prob_focus_present = <prob>

 loss = diffusion ( videos , cond = text , prob_focus_present = 0.5 ) # for 50% of videos, focus on the present during training
loss . backward ()

如果您對如何完成此操作有更好的了解，只需打開 github 問題即可。

托多

引文

 @misc { ho2022video ,
  title   = { Video Diffusion Models } , 
  author  = { Jonathan Ho and Tim Salimans and Alexey Gritsenko and William Chan and Mohammad Norouzi and David J. Fleet } ,
  year    = { 2022 } ,
  eprint  = { 2204.03458 } ,
  archivePrefix = { arXiv } ,
  primaryClass = { cs.CV }
}

 @misc { Saharia2022 ,
    title   = { Imagen: unprecedented photorealism × deep level of language understanding } ,
    author  = { Chitwan Saharia*, William Chan*, Saurabh Saxena†, Lala Li†, Jay Whang†, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho†, David Fleet†, Mohammad Norouzi* } ,
    year    = { 2022 }
}