PaLM rlhf pytorch下载 - PaLM rlhf pytorch源码下载

官方 chatgpt 博客文章

PaLM + RLHF - Pytorch（正在开发中）

在 PaLM 架构之上实现 RLHF（带有人类反馈的强化学习）。也许我也会添加检索功能，就像复古一样

如果您有兴趣公开复制 ChatGPT 等内容，请考虑加入 Laion

潜在的后继者：直接偏好优化 - 此存储库中的所有代码都变成〜二进制交叉熵损失，< 5 loc。奖励模型和 PPO 就这么多了

常问问题

这包含推理模型吗？

没有经过训练的模型。这只是船和总体地图。我们仍然需要数百万美元的计算+数据才能航行到高维参数空间中的正确点。即便如此，你也需要专业的水手（比如因稳定扩散而闻名的罗宾·罗姆巴赫）来实际引导船只度过动荡的时期，到达这一点。

社区

在 ChatGPT 发布之前的几个月里，CarperAI 一直致力于大型语言模型的 RLHF 框架的开发。

Yannic Kilcher 也在致力于开源实现

与 Letitia 一起喝咖啡 | AI代码商场 |代码商场第 2 部分

欣赏

Stability.ai 慷慨赞助前沿人工智能研究
？ Hugging Face 和 CarperAI 撰写了博客文章《说明人类反馈的强化学习》(RLHF)，前者还提供了他们的加速库
@kisseternity 和 @taynoel84 用于代码审查和查找错误
Enrico 从 Pytorch 2.0 集成 Flash Attention

安装

$ pip install palm-rlhf-pytorch

用法

首先训练PaLM ，就像任何其他自回归变压器一样

 import torch
from palm_rlhf_pytorch import PaLM

palm = PaLM (
    num_tokens = 20000 ,
    dim = 512 ,
    depth = 12 ,
    flash_attn = True # https://arxiv.org/abs/2205.14135
). cuda ()

seq = torch . randint ( 0 , 20000 , ( 1 , 2048 )). cuda ()

loss = palm ( seq , return_loss = True )
loss . backward ()

# after much training, you can now generate sequences

generated = palm . generate ( 2048 ) # (1, 2048)

然后利用精心策划的人类反馈来训练您的奖励模型。在最初的论文中，他们无法在不过度拟合的情况下从预训练的 Transformer 中获得奖励模型进行微调，但我无论如何都给出了使用LoRA进行微调的选项，因为它仍然是开放研究。

 import torch
from palm_rlhf_pytorch import PaLM , RewardModel

palm = PaLM (
    num_tokens = 20000 ,
    dim = 512 ,
    depth = 12 ,
    causal = False
)

reward_model = RewardModel (
    palm ,
    num_binned_output = 5 # say rating from 1 to 5
). cuda ()

# mock data

seq = torch . randint ( 0 , 20000 , ( 1 , 1024 )). cuda ()
prompt_mask = torch . zeros ( 1 , 1024 ). bool (). cuda () # which part of the sequence is prompt, which part is response
labels = torch . randint ( 0 , 5 , ( 1 ,)). cuda ()

# train

loss = reward_model ( seq , prompt_mask = prompt_mask , labels = labels )
loss . backward ()

# after much training

reward = reward_model ( seq , prompt_mask = prompt_mask )

然后你将把你的变压器和奖励模型传递给RLHFTrainer

 import torch
from palm_rlhf_pytorch import PaLM , RewardModel , RLHFTrainer

# load your pretrained palm

palm = PaLM (
    num_tokens = 20000 ,
    dim = 512 ,
    depth = 12
). cuda ()

palm . load ( './path/to/pretrained/palm.pt' )

# load your pretrained reward model

reward_model = RewardModel (
    palm ,
    num_binned_output = 5
). cuda ()

reward_model . load ( './path/to/pretrained/reward_model.pt' )

# ready your list of prompts for reinforcement learning

prompts = torch . randint ( 0 , 256 , ( 50000 , 512 )). cuda () # 50k prompts

# pass it all to the trainer and train

trainer = RLHFTrainer (
    palm = palm ,
    reward_model = reward_model ,
    prompt_token_ids = prompts
)

trainer . train ( num_episodes = 50000 )

# then, if it succeeded...
# generate say 10 samples and use the reward model to return the best one

answer = trainer . generate ( 2048 , prompt = prompts [ 0 ], num_samples = 10 ) # (<= 2048,)

托多

引文

 @article { Stiennon2020LearningTS ,
    title   = { Learning to summarize from human feedback } ,
    author  = { Nisan Stiennon and Long Ouyang and Jeff Wu and Daniel M. Ziegler and Ryan J. Lowe and Chelsea Voss and Alec Radford and Dario Amodei and Paul Christiano } ,
    journal = { ArXiv } ,
    year    = { 2020 } ,
    volume  = { abs/2009.01325 }
}

 @inproceedings { Chowdhery2022PaLMSL ,
    title   = { PaLM: Scaling Language Modeling with Pathways } ,
    author  = {Aakanksha Chowdhery and Sharan Narang and Jacob Devlin and Maarten Bosma and Gaurav Mishra and Adam Roberts and Paul Barham and Hyung Won Chung and Charles Sutton and Sebastian Gehrmann and Parker Schuh and Kensen Shi and Sasha Tsvyashchenko and Joshua Maynez and Abhishek Rao and Parker Barnes and Yi Tay and Noam M. Shazeer and Vinodkumar Prabhakaran and Emily Reif and Nan Du and Benton C. Hutchinson and Reiner Pope and James Bradbury and Jacob Austin and Michael Isard and Guy Gur-Ari and Pengcheng Yin and Toju Duke and Anselm Levskaya and Sanjay Ghemawat and Sunipa Dev and Henryk Michalewski and Xavier Garc{'i}a and Vedant Misra and Kevin Robinson and Liam Fedus and Denny Zhou and Daphne Ippolito and David Luan and Hyeontaek Lim and Barret Zoph and Alexander Spiridonov and Ryan Sepassi and David Dohan and Shivani Agrawal and Mark Omernick and Andrew M. Dai and Thanumalayan Sankaranarayana Pillai and Marie Pellat and Aitor Lewkowycz and Erica Oliveira Moreira and Rewon Child and Oleksandr Polozov and Katherine Lee and Zongwei Zhou and Xuezhi Wang and Brennan Saeta and Mark Diaz and Orhan Firat and Michele Catasta and Jason Wei and Kathleen S. Meier-Hellstern and Douglas Eck and Jeff Dean and Slav Petrov and Noah Fiedel},
    year    = { 2022 }
}

 @article { Hu2021LoRALA ,
    title   = { LoRA: Low-Rank Adaptation of Large Language Models } ,
    author  = { Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Weizhu Chen } ,
    journal = { ArXiv } ,
    year    = { 2021 } ,
    volume  = { abs/2106.09685 }
}

 @inproceedings { Sun2022ALT ,
    title     = { A Length-Extrapolatable Transformer } ,
    author    = { Yutao Sun and Li Dong and Barun Patra and Shuming Ma and Shaohan Huang and Alon Benhaim and Vishrav Chaudhary and Xia Song and Furu Wei } ,
    year      = { 2022 }
}

 @misc { gilmer2023intriguing
    title  = { Intriguing Properties of Transformer Training Instabilities } ,
    author = { Justin Gilmer, Andrea Schioppa, and Jeremy Cohen } ,
    year   = { 2023 } ,
    status = { to be published - one attention stabilization technique is circulating within Google Brain, being used by multiple teams }
}

 @inproceedings { dao2022flashattention ,
    title   = { Flash{A}ttention: Fast and Memory-Efficient Exact Attention with {IO}-Awareness } ,
    author  = { Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{'e}, Christopher } ,
    booktitle = { Advances in Neural Information Processing Systems } ,
    year    = { 2022 }
}

 @misc { Rubin2024 ,
    author  = { Ohad Rubin } ,
    url     = { https://medium.com/ @ ohadrubin/exploring-weight-decay-in-layer-normalization-challenges-and-a-reparameterization-solution-ad4d12c24950 }
}

 @inproceedings { Yuan2024FreePR ,
    title   = { Free Process Rewards without Process Labels } ,
    author  = { Lifan Yuan and Wendi Li and Huayu Chen and Ganqu Cui and Ning Ding and Kaiyan Zhang and Bowen Zhou and Zhiyuan Liu and Hao Peng } ,
    year    = { 2024 } ,
    url     = { https://api.semanticscholar.org/CorpusID:274445748 }
}