Vicuna LoRA RLHF PyTorch
1.0.0
a full pipeline to finetune Vicuna LLM with LoRA and RLHF on consumer hardware
穷人卡:2080Ti 12G
torch==2.0.0
cuda==11.8
python apply_delta.py --base 'decapoda-research/llama-7b-hf' --target './weights/vicuna-7b' --delta lmsys/vicuna-7b-delta-v1.1
check src/peft/utils/save_and_load.py first, Only comment the line 52 to
# #to_return = {k: v for k, v in to_return.items() if (("lora_" in k and adapter_name in k) or ("bias" in k))}
then run
python supervised_finetune.py --data_path './data/merge_sample.json' --output_path 'lora-Vicuna' --model_path './weights/vicuna-7b' --eval_steps 200 --save_steps 200 --test_size 1
check peft version first, if peft not 0.2.0, should install peft==0.2.0
pip uninstall peft -y
pip install peft==0.2.0 # 0.3.0.dev0 has many errors
python merge_peft_adapter.py --model_name 'lora-Vicuna'
pip uninstall peft -y
pip install git+https://github.com/huggingface/peft.git # then comments peft/utis/save_and_load.py line 52.
python train_reward_model.py --model_name './weights/vicuna-7b' --gradient_accumulation_steps 32 --per_device_train_batch_size 1 --train_subset 100 --eval_subset 10 --local_rank 0 --bf16 False
python merge_peft_adapter.py --model_name ./reward_model_vicuna-7b
python tuning_lm_with_rl.py --model_name './lora-Vicuna-adapter-merged' --reward_model_name './reward_model_vicuna-7b-adapter-merged' --adafactor False --tokenizer_name 'decapoda-research/llama-7b-hf' --save_freq 100 --output_max_length 128 --batch_size 1 --gradient_accumulation_steps 1 --batched_gen True --ppo_epochs 1 --seed 0 --learning_rate 1.4e-5 --early_stopping True --output_dir './tuning_llama_rl_checkpoints'
value
to put in on 0. 需要参看 transformer 在github上的最新代码,我在发现这个问题的时候,隔天发现在transformer的github上 8小时前才刚刚修复了这个问题。apply_delta.py 来自 FastChat 。
requirements 主要是按照 alpaca-lora 来配环境。
If this project help you reduce time to develop, you can give me a cup of coffee :)
AliPay(支付宝)
WechatPay(微信)
MIT © Kun