MedicalGPT
v2.2.0
??中文| English | Documents/Docs | ?Models/Models
MedicalGPT training medical GPT model with ChatGPT training pipeline, implementation of Pretraining, Supervised Finetuning, RLHF(Reward Modeling and Reinforcement Learning) and DPO(Direct Preference Optimization).
MedicalGPT trains large medical models and implements incremental pre-training, supervised fine-tuning, RLHF (reward modeling, reinforcement learning training) and DPO (direct preference optimization).
[2024/09/21] v2.3 version: Supports Qwen-2.5 series models, see Release-v2.3 for details
[2024/08/02] v2.2 version: Supports role-playing model training, and adds the doctor-patient dialogue SFT data generation script role_play_data. For details, see Release-v2.2
[2024/06/11] v2.1 version: Supports Qwen-2 series models, see Release-v2.1 for details
[2024/04/24] v2.0 version: Supports Llama-3 series models, see Release-v2.0 for details
[2024/04/17] v1.9 version: ORPO is supported. For detailed usage, please refer to run_orpo.sh . See Release-v1.9 for details
[2024/01/26] v1.8 version: Support fine-tuning Mixtral hybrid expert MoE model Mixtral 8x7B . See Release-v1.8 for details
[2024/01/14] v1.7 version: Added retrieval enhanced generation (RAG)-based file Q&A ChatPDF function, code chatpdf.py , which can improve industry Q&A accuracy based on fine-tuned LLM combined with knowledge base file Q&A. See Release-v1.7 for details
[2023/10/23] v1.6 version: Added RoPE interpolation to extend the context length of the GPT model; supports FlashAttention-2 and LongLoRA for the LLaMA model
[2023/08/28] Version v1.5: Added DPO (Direct Preference Optimization) method. DPO achieves precise control of its behavior by directly optimizing the language model, and can effectively learn human preferences. See Release-v1.5 for details
[2023/08/08] v1.4 version: Released the Chinese and English Vicuna-13B model shibing624/vicuna-baichuan-13b-chat based on the fine-tuning of the ShareGPT4 data set, and the corresponding LoRA model shibing624/vicuna-baichuan-13b-chat- lora, see Release-v1.4 for details
[2023/08/02] v1.3 version: Added multi-round dialogue fine-tuning training of LLaMA, LLaMA2, Bloom, ChatGLM, ChatGLM2, and Baichuan models; added domain vocabulary expansion function; added Chinese pre-training data set and Chinese ShareGPT fine-tuning training set, see Release-v1.3 for details
[2023/07/13] v1.1 version: Released the Chinese medical LLaMA-13B model shibing624/ziya-llama-13b-medical-merged, based on the Ziya-LLaMA-13B-v1 model, SFT fine-tuned a version of the medical model, medical The question and answer effect has been improved, and the complete model weights after fine-tuning are released. For details, see Release-v1.1
[2023/06/15] v1.0 version: Released Chinese medical LoRA model shibing624/ziya-llama-13b-medical-lora, based on Ziya-LLaMA-13B-v1 model, SFT fine-tuned a version of the medical model, medical question and answer effect There has been an improvement, and the fine-tuned LoRA weights have been released. For details, see Release-v1.0
[2023/06/05] Version v0.2: Taking medical care as an example, a large model in the training field implements four stages of training: including secondary pre-training, supervised fine-tuning, reward modeling, and reinforcement learning training. See Release-v0.2 for details
Based on the ChatGPT Training Pipeline, this project implements the domain model-the training of a large language model in the medical industry:
| Model | Base Model | Introduction |
|---|---|---|
| shibing624/ziya-llama-13b-medical-lora | IDEA-CCNL/Ziya-LLaMA-13B-v1 | SFT fine-tuned a version of the Ziya-LLaMA-13B model on the 2.4 million Chinese and English medical data set shibing624/medical. The medical question and answer effect has been improved, and the fine-tuned LoRA weights (single-round dialogue) have been released. |
| shibing624/ziya-llama-13b-medical-merged | IDEA-CCNL/Ziya-LLaMA-13B-v1 | SFT fine-tuned a version of the Ziya-LLaMA-13B model on the 2.4 million Chinese and English medical data set shibing624/medical. The medical question and answer effect has been improved, and the complete model weights after fine-tuning are released (single round of dialogue) |
| shibing624/vicuna-baichuan-13b-chat-lora | baichuan-inc/Baichuan-13B-Chat | SFT fine-tuned a version of the baichuan-13b-chat multi-round question and answer model on the 100,000 multi-language ShareGPT GPT4 multi-round conversation data set shibing624/sharegpt_gpt4 and the medical data set shibing624/medical. The effects of daily Q&A and medical Q&A have been improved, and the fine-tuning was released. LoRA weight after |
| shibing624/vicuna-baichuan-13b-chat | baichuan-inc/Baichuan-13B-Chat | SFT fine-tuned a version of the baichuan-13b-chat multi-round question and answer model on the 100,000 multi-language ShareGPT GPT4 multi-round conversation data set shibing624/sharegpt_gpt4 and the medical data set shibing624/medical. The effects of daily Q&A and medical Q&A have been improved, and the fine-tuning was released. The complete model weights after |
| shibing624/llama-3-8b-instruct-262k-chinese | Llama-3-8B-Instruct-262k | The ultra-long text multi-turn dialogue model obtained by fine-tuning the ORPO method on the 20,000 Chinese and English preference data set shibing624/DPO-En-Zh-20k-Preference is suitable for RAG and multi-turn dialogue. |
Demonstrate the effect of shibing624/vicuna-baichuan-13b-chat model: 
For specific cases, see Inference Examples
We provide a simple interactive web interface based on Gradio. After starting the service, you can access it through a browser, enter a question, and the model will return the answer.
Start the service with the following command:
CUDA_VISIBLE_DEVICES=0 python gradio_demo.py --model_type base_model_type --base_model path_to_llama_hf_dir --lora_model path_to_lora_dirParameter description:
--model_type {base_model_type} : Pre-trained model type, such as llama, bloom, chatglm, etc.--base_model {base_model} : Directory to store LLaMA model weights and configuration files in HF format. You can also use the HF Model Hub model call name.--lora_model {lora_model} : The directory where the LoRA file is located. You can also use the HF Model Hub model calling name. If lora weights have been merged into the pre-trained model, delete the --lora_model parameter.--tokenizer_path {tokenizer_path} : The directory where the corresponding tokenizer is stored. If this parameter is not provided, its default value is the same as --base_model--template_name : template name, such as vicuna , alpaca , etc. If this parameter is not provided, its default value is vicuna--only_cpu : Use CPU only for inference--resize_emb : Whether to adjust the embedding size. If not, use the embedding size of the pre-trained model. The default is not to adjust. requirements.txt will be updated from time to time to adapt to the latest features. Use the following command to update dependencies:
git clone https://github.com/shibing624/MedicalGPT
cd MedicalGPT
pip install -r requirements.txt --upgrade* estimate
| training method | Accuracy | 7B | 13B | 30B | 70B | 110B | 8x7B | 8x22B |
|---|---|---|---|---|---|---|---|---|
| Full parameters | AMP (automatic mixed precision) | 120GB | 240GB | 600GB | 1200GB | 2000GB | 900GB | 2400GB |
| Full parameters | 16 | 60GB | 120GB | 300GB | 600GB | 900GB | 400GB | 1200GB |
| LoRA | 16 | 16 GB | 32GB | 64GB | 160GB | 240GB | 120GB | 320GB |
| QLoRA | 8 | 10GB | 20GB | 40GB | 80GB | 140GB | 60GB | 160GB |
| QLoRA | 4 | 6GB | 12GB | 24GB | 48GB | 72GB | 30GB | 96GB |
| QLoRA | 2 | 4GB | 8GB | 16 GB | 24GB | 48GB | 18GB | 48GB |
Training stage:
| Stage | Introduction | Python script | Shell script |
|---|---|---|---|
| Continue Pretraining | Incremental pre-training | pretraining.py | run_pt.sh |
| Supervised Fine-tuning | Supervised fine-tuning | supervised_finetuning.py | run_sft.sh |
| Direct Preference Optimization | Direct preference optimization | dpo_training.py | run_dpo.sh |
| Reward Modeling | Reward model modeling | reward_modeling.py | run_rm.sh |
| Reinforcement Learning | reinforcement learning | ppo_training.py | run_ppo.sh |
| ORPO | Probabilistic preference optimization | orpo_training.py | run_orpo.sh |
| Model Name | Model Size | Target Modules | Template |
|---|---|---|---|
| Baichuan | 7B/13B | W_pack | baichuan |
| Baichuan2 | 7B/13B | W_pack | baichuan2 |
| BLOOMZ | 560M/1.1B/1.7B/3B/7.1B/176B | query_key_value | vicuna |
| ChatGLM | 6B | query_key_value | chatglm |
| ChatGLM2 | 6B | query_key_value | chatglm2 |
| ChatGLM3 | 6B | query_key_value | chatglm3 |
| Cohere | 104B | q_proj,v_proj | cohere |
| DeepSeek | 7B/16B/67B | q_proj,v_proj | deepseek |
| InternLM2 | 7B/20B | wxya | intern2 |
| LLAMA | 7B/13B/33B/65B | q_proj,v_proj | alpaca |
| LLaMA2 | 7B/13B/70B | q_proj,v_proj | llama2 |
| LLaMA3 | 8B/70B | q_proj,v_proj | llama3 |
| Mistral | 7B/8x7B | q_proj,v_proj | mistral |
| Orion | 14B | q_proj,v_proj | orion |
| Qwen | 1.8B/7B/14B/72B | c_attn | qwen |
| Qwen1.5 | 0.5B/1.8B/4B/14B/32B/72B/110B | q_proj,v_proj | qwen |
| Qwen2 | 0.5B/1.5B/7B/72B | q_proj,v_proj | qwen |
| XVERSE | 13B | query_key_value | xverse |
| Yi | 6B/34B | q_proj,v_proj | Yi |
After the training is completed, we now load the trained model and verify the effect of the text generated by the model.
CUDA_VISIBLE_DEVICES=0 python inference.py
--model_type base_model_type
--base_model path_to_model_hf_dir
--tokenizer_path path_to_model_hf_dir
--lora_model path_to_lora
--interactiveParameter description:
--model_type {base_model_type} : Pre-trained model type, such as llama, bloom, chatglm, etc.--base_model {base_model} : Directory to store LLaMA model weights and configuration files in HF format--tokenizer_path {base_model} : Directory to store LLaMA model weights and configuration files in HF format--lora_model {lora_model} : The directory where the LoRA decompressed file is located. You can also use the HF Model Hub model calling name. This parameter does not need to be provided if LoRA weights have been incorporated into the pre-trained model--tokenizer_path {tokenizer_path} : The directory where the corresponding tokenizer is stored. If this parameter is not provided, its default value is the same as --base_model--template_name : template name, such as vicuna , alpaca , etc. If this parameter is not provided, its default value is vicuna--interactive : Start multiple rounds of Q&A interactively, using streaming reasoning--data_file {file_name} : When started in non-interactive mode, read the contents of file_name for batch prediction.--output_file {file_name} : In non-interactive mode, write the predicted results to file_name in jsonl format--resize_emb : Whether to adjust the embedding size. If not, use the embedding size of the pre-trained model. The default is not to adjust.--only_cpu : Use CPU only for inference--gpus {gpu_ids} : Specify the GPU device number used, the default is 0. If multiple GPUs are used, separate them with commas, such as 0,1,2 Multi-card data parallelism, batch inference
CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2 inference_multigpu_demo.py --model_type baichuan --base_model shibing624/vicuna-baichuan-13b-chatbash run_Multi-GPUs-deployment.shshibing624/vicuna-baichuan-13b-chat inference examples:
| Human | Assistant |
|---|---|
| What to do if a child has a fever | Fever is the body's natural response to an infection or other illness. Here are some suggestions to help relieve fever symptoms: 1. Stay hydrated: When you have a fever, your body loses water, so you need to stay hydrated. 2. Use an ice pack: Placing an ice pack on your forehead or neck can help lower your body temperature. 3. Use antipyretics: Antipyretics can help lower your body temperature. 4. Maintain body temperature: Maintaining body temperature can help lower body temperature. 5. Wipe your body with cold or warm water: Wiping your body with cold or warm water can help lower your body temperature. 6. Use antipyretics: Antipyretics can help lower your body temperature. 7. Maintain body temperature: Maintaining body temperature can help lower body temperature. 8. Wipe your body with cold or warm water: Wiping your body with cold or warm water can help lower your body temperature. 9. Use antipyretics: Antipyretics can help lower your body temperature. 10. Maintain body temperature: Maintaining body temperature can help lower body temperature. |
| What medicine can a one-year-old baby take if he has a fever? | When a 1-year-old baby has a fever, the following medications can be considered: 1. Antipyretics: such as Ibuprofen or Paracetamol. These medicines can help lower your body temperature and relieve fever symptoms. 2. Antibiotics: If the fever is caused by a bacterial infection, antibiotics may be considered. But please be sure to use it under the advice of your doctor. 3. Antiviral drugs: If the fever is caused by a viral infection, antiviral drugs may be considered. But please be sure to use it under the advice of your doctor. 4. Antipyretic and analgesic drugs: such as aspirin or paracetamol. These medicines can help relieve fever symptoms. 5. Antihistamines: such as diphenhydramine or fexofenadine (Naproxen). These medicines can help relieve fever symptoms. Be sure to use these medications under your doctor's advice and follow the dosage and usage instructions on the package insert. At the same time, keep your baby well hydrated and make sure your baby gets enough rest. |
| Anal lesions may be symptoms of what diseases | Anal lesions can be caused by a variety of causes, including infection, inflammation, tumors, or other diseases. Some common anal lesions include: 1. Anal Fissure: This is a tear in the skin of the anus, usually caused by constipation or other stress factors. 2. Anal Fistula: This is an anal infection that causes a fistula to form between the skin of the anus and the rectum. 3. Anal wart: This is a wart-like lesion that forms on the skin of the anus, usually caused by the human papilloma virus (HPV). 4. Anal abscess: This is an anal infection that causes an abscess to form under the skin of the anus. 5. Anal tumor: This is a tumor that forms on the skin of the anus or in the rectum. 6. Anal ulcer: This is an ulcer that forms on the skin of the anus, usually caused by infection or inflammation. 7. Anal itching: This is a symptom of itching of the anal skin, which may be caused by infection, inflammation or other diseases. If you have symptoms of anal lesions, consult a health care professional for appropriate diagnosis and treatment. |
| Hello, doctor, why is the leucorrhea increasing and turning yellow? | Hi, Increased and yellowish vaginal discharge can be caused by a variety of reasons, including infection, hormonal changes, vaginal dryness, or other gynecological issues. Here are some possible reasons: 1. Infection: Increased and yellowish vaginal discharge may be caused by vaginal or urinary tract infection. Infections can be caused by bacteria, viruses, or fungi. 2. Hormonal changes: Hormonal changes, such as the menstrual cycle or menopause, may cause increased vaginal discharge and yellowing. 3. Vaginal dryness: Vaginal dryness may lead to increased vaginal discharge and yellowing. 4. Other gynecological problems: Other gynecological problems, such as cervicitis, endometritis, or endometriosis, may also cause increased and yellow vaginal discharge. If you are concerned about your leucorrhea, it is recommended that you consult a gynecologist for examination and diagnosis. Your doctor can determine the cause and provide appropriate treatment based on your symptoms and test results. |
| If there is chocolate leucorrhea after medical abortion, is it because the abortion is not clean? | The occurrence of chocolate leucorrhea after medical abortion does not necessarily mean that the abortion was not clean. Chocolate leucorrhea is caused by the shedding and bleeding of the lining of the uterus, and usually occurs within a period of time after medical abortion. Causes of chocolate leucorrhea after medical abortion may include: 1. Endometrial shedding: After medical abortion, the endometrium will shed, causing bleeding. This bleeding may appear as chocolate leucorrhea. 2. Infection: After medical abortion, there may be trauma to the endometrium and cervix, making them susceptible to bacterial infection. Infection can cause chocolate vaginal discharge. 3. Changes in hormone levels: After medical abortion, hormone levels may change, leading to chocolate leucorrhea. If chocolate leucorrhea lasts for a long time, or is accompanied by other symptoms, such as abdominal pain, fever, peculiar smell of secretions, etc., it is recommended to seek medical treatment in time and conduct relevant examinations, such as gynecological examination, B-ultrasound, etc., to determine whether there is infection or other problems. Carry out corresponding treatment under the guidance of a doctor. |
| What diseases can Vitamin E Cream treat? | Vitamin E cream is a topical drug mainly used to treat skin diseases such as acne and seborrheic dermatitis. It can help reduce skin oil production, reduce inflammation, and promote skin repair. It should be noted that Vitamin E Cream is not a panacea and may not be suitable for certain skin diseases. Before use, it is recommended to consult a doctor or pharmacist to understand the indications, contraindications, usage and dosage of the drug. At the same time, you should follow your doctor's instructions when using drugs to avoid abuse on your own. |
| What causes acne? | Acne, also known as comedones, is caused by clogged pores in the skin. There are many causes of clogged pores, including: 1. Overactive sebaceous glands: The sebaceous glands produce too much oil, causing clogged pores. 2. Clogged pores: Dead skin cells, oil and bacteria on the skin surface clog pores. 3. Inflammation: Acne is often accompanied by inflammation, which can be caused by a skin infection or the immune system's response to acne. 4. Genetic factors: Some people may be more susceptible to acne, possibly due to genetic factors. 5. Hormonal changes: Acne is more common during puberty and menstruation, which may be caused by hormonal changes. 6. Lifestyle factors: Certain lifestyle factors, such as stress, poor diet, and lack of exercise, can also lead to acne. 7. Cosmetics and Skin Care Products: Certain cosmetics and skin care products may cause acne, especially those containing irritating ingredients. 8. Medications: Certain medications, such as oral contraceptive pills, may also cause acne. To prevent acne, you can take steps such as keeping your skin clean, using gentle skin care products, avoiding excessive use of cosmetics and skin care products, maintaining a healthy lifestyle and diet, and avoiding stress and hormonal changes. If your acne is severe, you may need to seek help from a dermatologist. |
| What to do if you have acne on your face, what medicine you can take, and what you need to pay attention to | Acne, also known as acne, is a common skin disease. The treatment of acne needs to comprehensively consider many aspects such as diet, living habits, medications, etc. Here are some suggestions: 1. Diet: * Avoid foods high in sugar, fat and oil, such as fast food, desserts, fried foods, etc. * Eat more fresh vegetables, fruits, and whole grains to increase fiber intake and promote intestinal peristalsis. * Avoid consuming too much caffeine, alcohol, spicy food and other stimulating foods. 2. Living habits: * Maintain a good sleep schedule and ensure adequate sleep. * Avoid using electronic products such as computers and mobile phones for a long time to avoid radiation irritation to the skin. * Keep your skin clean and use a gentle cleanser every morning and evening. * Avoid squeezing pimples to avoid infection and scarring. 3. Drugs: * Topical drugs: You can use skin care products containing salicylic acid, benzoic acid, azelaic acid and other ingredients to help remove dead skin cells on the skin surface and reduce the formation of acne. * Oral drugs: If acne is severe, oral antibiotics, vitamin B complex, zinc and other drugs can be considered, but they need to be used under the guidance of a doctor. 4. Note: * Do not use cosmetics and skin care products randomly to avoid irritating the skin. * Do not over cleanse the skin to avoid damaging the skin barrier. * Keep your mood comfortable and avoid negative emotions such as anxiety and depression. In short, the treatment of acne needs to comprehensively consider many aspects such as diet, living habits, medications, etc. During the treatment process, stay patient and confident, follow the doctor's recommendations, and gradually improve your skin condition. |
| Will blocked vas deferens cause male infertility? Hello, doctor, my colleague has been unhappy these days. I asked her what's wrong. She said that her husband's vas deferens was blocked, so he never had children. It's not that he didn't want to have children. Will the blocked vas deferens cause male infertility? | Blockage of the vas deferens may cause male infertility. The vas deferens is a tube that connects the testicles to the urethra and carries sperm from the testicles to the urethra. If the vas deferens is blocked, sperm cannot pass through the vas deferens to the urethra, causing infertility. There are many reasons for blockage of the vas deferens, including congenital developmental abnormalities, inflammation, tumors, trauma, etc. Treatment methods for vas deferens obstruction include surgical unblocking and drug treatment. If the vas deferens is severely blocked, surgery may be needed to clear it. It's important to note that blockage of the vas deferens does not necessarily lead to infertility, as there are other ways in which sperm can be transported into a woman's body. For example, sperm can reach the female body through tissues such as the epididymis and prostate. Therefore, if the vas deferens is blocked, it is recommended to seek medical treatment promptly and seek help from a professional doctor. |
| Can myocardial infarction be treated? | Myocardial infarction is a serious condition that, if treated promptly, can save lives and prevent long-term disability. The key to treating myocardial infarction is early recognition and treatment. Treatment of myocardial infarction usually involves the following steps: 1. Seek medical attention immediately: If symptoms such as chest pain, shortness of breath, or fainting occur, seek medical attention immediately. 2. Medication: Your doctor may prescribe medications, such as nitroglycerin or aspirin, to help relieve symptoms and prevent further damage. 3. Thrombolytic therapy: If the patient has a myocardial infarction before arriving at the hospital, the doctor may perform thrombolytic therapy, injecting drugs that dissolve blood clots to help restore blood flow. 4. Surgical treatment: If medication or thrombolytic treatment fails, your doctor may recommend surgery, such as coronary artery bypass surgery or cardiac stent implantation. 5. Rehabilitation and prevention: Treatment for myocardial infarction often includes a rehabilitation program to help patients regain physical function and prevent future cardiac events. Additionally, patients should follow their doctor's recommendations to reduce their risk of heart disease. It is important to note that treatment of myocardial infarction depends on the patient's specific condition and symptoms. Therefore, if you or someone you know develops symptoms of a myocardial infarction, seek medical attention immediately. |
| How to keep yourself energetic, here are 5 suggestions | 1. Get enough sleep: Sleep is crucial to keeping your energy levels high. Make sure you get 7-9 hours of quality sleep every night. 2. Eat a balanced diet: A balanced diet is essential to maintain energy levels. Make sure you eat enough protein, carbs, and healthy fats. 3. Exercise regularly: Exercising regularly can help boost energy levels. Try to do at least 30 minutes of aerobic exercise every day, such as walking, running, or biking. 4. Manage Stress: Stress drains energy, so managing it is crucial to keeping your energy levels high. Try to find ways to relax and reduce stress, such as meditation, yoga, or deep breathing exercises. 5. Stay hydrated: Dehydration can lower energy levels. Make sure to drink enough water every day and rehydrate as needed. |
| Write a spring poem | spring, It’s the season when the earth wakes up, It’s the season of revival for all things, It is the season when life blooms. spring, It’s the sunny season, It’s the season of chirping birds and fragrant flowers, It’s the season of lush greenery, It is the season of infinite hope. spring, It’s the season when people laugh, It’s the season for children to play, It’s the season for lovers to embrace each other, It’s the season when people pursue their dreams. |
| Why are there no polar bears in Antarctica? | There are no polar bears in Antarctica because polar bears live in the Arctic region, while Antarctica is located in the southern hemisphere, and the climate and environment are very different from the Arctic regions. Antarctica has a very cold climate and is covered in ice and snow all year round, while polar bears live in the Arctic, where there are abundant marine life and land mammals. In addition, the ecosystem in Antarctica is different from that in the Arctic and does not have the survival conditions required by polar bears. |
This project can only be used for research purposes, and the project developer is not responsible for any harm or loss caused by the use of this project (including but not limited to data, models, codes, etc.). Please refer to the disclaimer for details.
The licensing agreement for the MedicalGPT project code is The Apache License 2.0. The code can be used for free for commercial purposes, and the model weights and data can only be used for research purposes. Please attach the link to MedicalGPT and the license agreement in the product description.
If you use MedicalGPT in your research, please cite it in the following format:
@misc{MedicalGPT,
title={MedicalGPT: Training Medical GPT Model},
author={Ming Xu},
year={2023},
howpublished={ url {https://github.com/shibing624/MedicalGPT}},
}The project code is still very rough. If you have any improvements to the code, you are welcome to submit it back to this project. Before submitting, please pay attention to the following two points:
testspython -m pytest to run all unit tests to ensure that all unit tests passYou can then submit a PR.
Thanks for their great work!
