text generation webui
v1.16
대규모 언어 모델을 위한 Gradio 웹 UI.
목표는 텍스트 생성의 AUTOMATIC1111/stable-diffusion-webui가 되는 것입니다.
 |  |
|---|---|
 |  |
instruct , chat-instruct , chat , chat-instruct 의 자동 프롬프트 템플릿 포함.installer_files 디렉토리에 요구사항이 설치됩니다.start_linux.sh , start_windows.bat , start_macos.sh 또는 start_wsl.bat 를 실행하십시오.http://localhost:7860 으로 이동합니다. 나중에 웹 UI를 다시 시작하려면 동일한 start_ 스크립트를 실행하면 됩니다. 다시 설치해야 하는 경우에는 설치 시 생성된 installer_files 폴더를 삭제하고 스크립트를 다시 실행하세요.
./start_linux.sh --help 와 같은 명령줄 플래그를 사용하거나 이를 CMD_FLAGS.txt 에 추가할 수 있습니다(예: API 사용을 활성화하려면 --api ). 프로젝트를 업데이트하려면 update_wizard_linux.sh , update_wizard_windows.bat , update_wizard_macos.sh 또는 update_wizard_wsl.bat 실행하세요.
이 스크립트는 Miniconda를 사용하여 installer_files 폴더에 Conda 환경을 설정합니다.
installer_files 환경에서 수동으로 설치해야 하는 경우 cmd 스크립트( cmd_linux.sh , cmd_windows.bat , cmd_macos.sh 또는 cmd_wsl.bat 를 사용하여 대화형 셸을 시작할 수 있습니다.
start_ , update_wizard_ 또는 cmd_ )를 admin/root로 실행할 필요가 없습니다.extensions_reqs 스크립트를 사용할 수 있습니다. 마지막으로 이 스크립트는 버전 충돌이 발생할 경우 우선순위를 갖도록 프로젝트의 주요 요구 사항을 설치합니다.GPU_CHOICE , USE_CUDA118 , LAUNCH_AFTER_INSTALL 및 INSTALL_EXTENSIONS 환경 변수를 사용할 수 있습니다. 예: GPU_CHOICE=A USE_CUDA118=FALSE LAUNCH_AFTER_INSTALL=FALSE INSTALL_EXTENSIONS=TRUE ./start_linux.sh .명령줄 사용 경험이 있는 경우 권장됩니다.
https://docs.conda.io/en/latest/miniconda.html
Linux 또는 WSL에서는 다음 두 명령(소스)을 사용하여 자동으로 설치할 수 있습니다.
curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh
conda create -n textgen python=3.11
conda activate textgen
| 체계 | GPU | 명령 |
|---|---|---|
| 리눅스/WSL | 엔비디아 | pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121 |
| 리눅스/WSL | CPU만 | pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cpu |
| 리눅스 | AMD | pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/rocm6.1 |
| 맥OS + MPS | 어느 | pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 |
| 윈도우 | 엔비디아 | pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121 |
| 윈도우 | CPU만 | pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 |
최신 명령은 https://pytorch.org/get-started/locally/에서 찾을 수 있습니다.
NVIDIA의 경우 CUDA 런타임 라이브러리도 설치해야 합니다.
conda install -y -c "nvidia/label/cuda-12.1.1" cuda-runtime
일부 라이브러리를 수동으로 컴파일하기 위해 nvcc 필요한 경우 위 명령을 다음으로 바꾸십시오.
conda install -y -c "nvidia/label/cuda-12.1.1" cuda
git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r <requirements file according to table below>
사용할 요구사항 파일:
| GPU | CPU | 사용할 요구 사항 파일 |
|---|---|---|
| 엔비디아 | AVX2가 있습니다 | requirements.txt |
| 엔비디아 | AVX2 없음 | requirements_noavx2.txt |
| AMD | AVX2가 있습니다 | requirements_amd.txt |
| AMD | AVX2 없음 | requirements_amd_noavx2.txt |
| CPU만 | AVX2가 있습니다 | requirements_cpu_only.txt |
| CPU만 | AVX2 없음 | requirements_cpu_only_noavx2.txt |
| 사과 | 인텔 | requirements_apple_intel.txt |
| 사과 | 애플실리콘 | requirements_apple_silicon.txt |
conda activate textgen
cd text-generation-webui
python server.py
그런 다음 다음을 찾아보세요.
http://localhost:7860/?__theme=dark
위 명령에서 requirements_cpu_only.txt 또는 requirements_cpu_only_noavx2.txt 사용하세요.
하드웨어에 적합한 명령(PyPI에서 설치)을 사용하여 llama-cpp-python을 수동으로 설치합니다.
LLAMA_HIPBLAS=on 토글을 사용하세요.AutoGPTQ를 수동으로 설치합니다: 설치.
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu118
conda install -y -c "nvidia/label/cuda-11.8.0" cuda-runtime
--load-in-8bit 사용하려면 다음과 같이 다운그레이드해야 할 수도 있습니다.pip install bitsandbytes==0.38.1pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl 위의 requirements*.txt 에는 GitHub Actions를 통해 사전 컴파일된 다양한 휠이 포함되어 있습니다. 수동으로 컴파일하고 싶거나 하드웨어에 적합한 휠을 사용할 수 없어 필요한 경우 requirements_nowheels.txt 사용한 다음 원하는 로더를 수동으로 설치할 수 있습니다.
For NVIDIA GPU:
ln -s docker/{nvidia/Dockerfile,nvidia/docker-compose.yml,.dockerignore} .
For AMD GPU:
ln -s docker/{amd/Dockerfile,intel/docker-compose.yml,.dockerignore} .
For Intel GPU:
ln -s docker/{intel/Dockerfile,amd/docker-compose.yml,.dockerignore} .
For CPU only
ln -s docker/{cpu/Dockerfile,cpu/docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
#Create logs/cache dir :
mkdir -p logs cache
# Edit .env and set:
# TORCH_CUDA_ARCH_LIST based on your GPU model
# APP_RUNTIME_GID your host user's group id (run `id -g` in a terminal)
# BUILD_EXTENIONS optionally add comma separated list of extensions to build
# Edit CMD_FLAGS.txt and add in it the options you want to execute (like --listen --cpu)
#
docker compose up --build
요구 requirements*.txt 수시로 변경됩니다. 업데이트하려면 다음 명령을 사용하십시오.
conda activate textgen
cd text-generation-webui
pip install -r <requirements file that you have used> --upgrade
usage: server.py [-h] [--multi-user] [--character CHARACTER] [--model MODEL] [--lora LORA [LORA ...]] [--model-dir MODEL_DIR] [--lora-dir LORA_DIR] [--model-menu] [--settings SETTINGS]
[--extensions EXTENSIONS [EXTENSIONS ...]] [--verbose] [--chat-buttons] [--idle-timeout IDLE_TIMEOUT] [--loader LOADER] [--cpu] [--auto-devices]
[--gpu-memory GPU_MEMORY [GPU_MEMORY ...]] [--cpu-memory CPU_MEMORY] [--disk] [--disk-cache-dir DISK_CACHE_DIR] [--load-in-8bit] [--bf16] [--no-cache] [--trust-remote-code]
[--force-safetensors] [--no_use_fast] [--use_flash_attention_2] [--use_eager_attention] [--load-in-4bit] [--use_double_quant] [--compute_dtype COMPUTE_DTYPE] [--quant_type QUANT_TYPE]
[--flash-attn] [--tensorcores] [--n_ctx N_CTX] [--threads THREADS] [--threads-batch THREADS_BATCH] [--no_mul_mat_q] [--n_batch N_BATCH] [--no-mmap] [--mlock]
[--n-gpu-layers N_GPU_LAYERS] [--tensor_split TENSOR_SPLIT] [--numa] [--logits_all] [--no_offload_kqv] [--cache-capacity CACHE_CAPACITY] [--row_split] [--streaming-llm]
[--attention-sink-size ATTENTION_SINK_SIZE] [--tokenizer-dir TOKENIZER_DIR] [--gpu-split GPU_SPLIT] [--autosplit] [--max_seq_len MAX_SEQ_LEN] [--cfg-cache] [--no_flash_attn]
[--no_xformers] [--no_sdpa] [--cache_8bit] [--cache_4bit] [--num_experts_per_token NUM_EXPERTS_PER_TOKEN] [--triton] [--no_inject_fused_mlp] [--no_use_cuda_fp16] [--desc_act]
[--disable_exllama] [--disable_exllamav2] [--wbits WBITS] [--groupsize GROUPSIZE] [--hqq-backend HQQ_BACKEND] [--cpp-runner] [--deepspeed] [--nvme-offload-dir NVME_OFFLOAD_DIR]
[--local_rank LOCAL_RANK] [--alpha_value ALPHA_VALUE] [--rope_freq_base ROPE_FREQ_BASE] [--compress_pos_emb COMPRESS_POS_EMB] [--listen] [--listen-port LISTEN_PORT]
[--listen-host LISTEN_HOST] [--share] [--auto-launch] [--gradio-auth GRADIO_AUTH] [--gradio-auth-path GRADIO_AUTH_PATH] [--ssl-keyfile SSL_KEYFILE] [--ssl-certfile SSL_CERTFILE]
[--subpath SUBPATH] [--api] [--public-api] [--public-api-id PUBLIC_API_ID] [--api-port API_PORT] [--api-key API_KEY] [--admin-key ADMIN_KEY] [--nowebui]
[--multimodal-pipeline MULTIMODAL_PIPELINE] [--model_type MODEL_TYPE] [--pre_layer PRE_LAYER [PRE_LAYER ...]] [--checkpoint CHECKPOINT] [--monkey-patch] [--no_inject_fused_attention]
Text generation web UI
options:
-h, --help show this help message and exit
Basic settings:
--multi-user Multi-user mode. Chat histories are not saved or automatically loaded. Warning: this is likely not safe for sharing publicly.
--character CHARACTER The name of the character to load in chat mode by default.
--model MODEL Name of the model to load by default.
--lora LORA [LORA ...] The list of LoRAs to load. If you want to load more than one LoRA, write the names separated by spaces.
--model-dir MODEL_DIR Path to directory with all the models.
--lora-dir LORA_DIR Path to directory with all the loras.
--model-menu Show a model menu in the terminal when the web UI is first launched.
--settings SETTINGS Load the default interface settings from this yaml file. See settings-template.yaml for an example. If you create a file called settings.yaml, this
file will be loaded by default without the need to use the --settings flag.
--extensions EXTENSIONS [EXTENSIONS ...] The list of extensions to load. If you want to load more than one extension, write the names separated by spaces.
--verbose Print the prompts to the terminal.
--chat-buttons Show buttons on the chat tab instead of a hover menu.
--idle-timeout IDLE_TIMEOUT Unload model after this many minutes of inactivity. It will be automatically reloaded when you try to use it again.
Model loader:
--loader LOADER Choose the model loader manually, otherwise, it will get autodetected. Valid options: Transformers, llama.cpp, llamacpp_HF, ExLlamav2_HF, ExLlamav2,
AutoGPTQ.
Transformers/Accelerate:
--cpu Use the CPU to generate text. Warning: Training on CPU is extremely slow.
--auto-devices Automatically split the model across the available GPU(s) and CPU.
--gpu-memory GPU_MEMORY [GPU_MEMORY ...] Maximum GPU memory in GiB to be allocated per GPU. Example: --gpu-memory 10 for a single GPU, --gpu-memory 10 5 for two GPUs. You can also set values
in MiB like --gpu-memory 3500MiB.
--cpu-memory CPU_MEMORY Maximum CPU memory in GiB to allocate for offloaded weights. Same as above.
--disk If the model is too large for your GPU(s) and CPU combined, send the remaining layers to the disk.
--disk-cache-dir DISK_CACHE_DIR Directory to save the disk cache to. Defaults to "cache".
--load-in-8bit Load the model with 8-bit precision (using bitsandbytes).
--bf16 Load the model with bfloat16 precision. Requires NVIDIA Ampere GPU.
--no-cache Set use_cache to False while generating text. This reduces VRAM usage slightly, but it comes at a performance cost.
--trust-remote-code Set trust_remote_code=True while loading the model. Necessary for some models.
--force-safetensors Set use_safetensors=True while loading the model. This prevents arbitrary code execution.
--no_use_fast Set use_fast=False while loading the tokenizer (it's True by default). Use this if you have any problems related to use_fast.
--use_flash_attention_2 Set use_flash_attention_2=True while loading the model.
--use_eager_attention Set attn_implementation= eager while loading the model.
bitsandbytes 4-bit:
--load-in-4bit Load the model with 4-bit precision (using bitsandbytes).
--use_double_quant use_double_quant for 4-bit.
--compute_dtype COMPUTE_DTYPE compute dtype for 4-bit. Valid options: bfloat16, float16, float32.
--quant_type QUANT_TYPE quant_type for 4-bit. Valid options: nf4, fp4.
llama.cpp:
--flash-attn Use flash-attention.
--tensorcores NVIDIA only: use llama-cpp-python compiled with tensor cores support. This may increase performance on newer cards.
--n_ctx N_CTX Size of the prompt context.
--threads THREADS Number of threads to use.
--threads-batch THREADS_BATCH Number of threads to use for batches/prompt processing.
--no_mul_mat_q Disable the mulmat kernels.
--n_batch N_BATCH Maximum number of prompt tokens to batch together when calling llama_eval.
--no-mmap Prevent mmap from being used.
--mlock Force the system to keep the model in RAM.
--n-gpu-layers N_GPU_LAYERS Number of layers to offload to the GPU.
--tensor_split TENSOR_SPLIT Split the model across multiple GPUs. Comma-separated list of proportions. Example: 60,40.
--numa Activate NUMA task allocation for llama.cpp.
--logits_all Needs to be set for perplexity evaluation to work. Otherwise, ignore it, as it makes prompt processing slower.
--no_offload_kqv Do not offload the K, Q, V to the GPU. This saves VRAM but reduces the performance.
--cache-capacity CACHE_CAPACITY Maximum cache capacity (llama-cpp-python). Examples: 2000MiB, 2GiB. When provided without units, bytes will be assumed.
--row_split Split the model by rows across GPUs. This may improve multi-gpu performance.
--streaming-llm Activate StreamingLLM to avoid re-evaluating the entire prompt when old messages are removed.
--attention-sink-size ATTENTION_SINK_SIZE StreamingLLM: number of sink tokens. Only used if the trimmed prompt does not share a prefix with the old prompt.
--tokenizer-dir TOKENIZER_DIR Load the tokenizer from this folder. Meant to be used with llamacpp_HF through the command-line.
ExLlamaV2:
--gpu-split GPU_SPLIT Comma-separated list of VRAM (in GB) to use per GPU device for model layers. Example: 20,7,7.
--autosplit Autosplit the model tensors across the available GPUs. This causes --gpu-split to be ignored.
--max_seq_len MAX_SEQ_LEN Maximum sequence length.
--cfg-cache ExLlamav2_HF: Create an additional cache for CFG negative prompts. Necessary to use CFG with that loader.
--no_flash_attn Force flash-attention to not be used.
--no_xformers Force xformers to not be used.
--no_sdpa Force Torch SDPA to not be used.
--cache_8bit Use 8-bit cache to save VRAM.
--cache_4bit Use Q4 cache to save VRAM.
--num_experts_per_token NUM_EXPERTS_PER_TOKEN Number of experts to use for generation. Applies to MoE models like Mixtral.
AutoGPTQ:
--triton Use triton.
--no_inject_fused_mlp Triton mode only: disable the use of fused MLP, which will use less VRAM at the cost of slower inference.
--no_use_cuda_fp16 This can make models faster on some systems.
--desc_act For models that do not have a quantize_config.json, this parameter is used to define whether to set desc_act or not in BaseQuantizeConfig.
--disable_exllama Disable ExLlama kernel, which can improve inference speed on some systems.
--disable_exllamav2 Disable ExLlamav2 kernel.
--wbits WBITS Load a pre-quantized model with specified precision in bits. 2, 3, 4 and 8 are supported.
--groupsize GROUPSIZE Group size.
HQQ:
--hqq-backend HQQ_BACKEND Backend for the HQQ loader. Valid options: PYTORCH, PYTORCH_COMPILE, ATEN.
TensorRT-LLM:
--cpp-runner Use the ModelRunnerCpp runner, which is faster than the default ModelRunner but doesn't support streaming yet.
DeepSpeed:
--deepspeed Enable the use of DeepSpeed ZeRO-3 for inference via the Transformers integration.
--nvme-offload-dir NVME_OFFLOAD_DIR DeepSpeed: Directory to use for ZeRO-3 NVME offloading.
--local_rank LOCAL_RANK DeepSpeed: Optional argument for distributed setups.
RoPE:
--alpha_value ALPHA_VALUE Positional embeddings alpha factor for NTK RoPE scaling. Use either this or compress_pos_emb, not both.
--rope_freq_base ROPE_FREQ_BASE If greater than 0, will be used instead of alpha_value. Those two are related by rope_freq_base = 10000 * alpha_value ^ (64 / 63).
--compress_pos_emb COMPRESS_POS_EMB Positional embeddings compression factor. Should be set to (context length) / (model's original context length). Equal to 1/rope_freq_scale.
Gradio:
--listen Make the web UI reachable from your local network.
--listen-port LISTEN_PORT The listening port that the server will use.
--listen-host LISTEN_HOST The hostname that the server will use.
--share Create a public URL. This is useful for running the web UI on Google Colab or similar.
--auto-launch Open the web UI in the default browser upon launch.
--gradio-auth GRADIO_AUTH Set Gradio authentication password in the format "username:password". Multiple credentials can also be supplied with "u1:p1,u2:p2,u3:p3".
--gradio-auth-path GRADIO_AUTH_PATH Set the Gradio authentication file path. The file should contain one or more user:password pairs in the same format as above.
--ssl-keyfile SSL_KEYFILE The path to the SSL certificate key file.
--ssl-certfile SSL_CERTFILE The path to the SSL certificate cert file.
--subpath SUBPATH Customize the subpath for gradio, use with reverse proxy
API:
--api Enable the API extension.
--public-api Create a public URL for the API using Cloudfare.
--public-api-id PUBLIC_API_ID Tunnel ID for named Cloudflare Tunnel. Use together with public-api option.
--api-port API_PORT The listening port for the API.
--api-key API_KEY API authentication key.
--admin-key ADMIN_KEY API authentication key for admin tasks like loading and unloading models. If not set, will be the same as --api-key.
--nowebui Do not launch the Gradio UI. Useful for launching the API in standalone mode.
Multimodal:
--multimodal-pipeline MULTIMODAL_PIPELINE The multimodal pipeline to use. Examples: llava-7b, llava-13b.https://github.com/oobabooga/text- Generation-webui/wiki
모델은 text-generation-webui/models 폴더에 배치되어야 합니다. 일반적으로 Hugging Face에서 다운로드됩니다.
models 에 직접 배치되어야 합니다. 예: text-generation-webui
└── models
└── llama-2-13b-chat.Q4_K_M.gguf
text-generation-webui
├── models
│ ├── lmsys_vicuna-33b-v1.3
│ │ ├── config.json
│ │ ├── generation_config.json
│ │ ├── pytorch_model-00001-of-00007.bin
│ │ ├── pytorch_model-00002-of-00007.bin
│ │ ├── pytorch_model-00003-of-00007.bin
│ │ ├── pytorch_model-00004-of-00007.bin
│ │ ├── pytorch_model-00005-of-00007.bin
│ │ ├── pytorch_model-00006-of-00007.bin
│ │ ├── pytorch_model-00007-of-00007.bin
│ │ ├── pytorch_model.bin.index.json
│ │ ├── special_tokens_map.json
│ │ ├── tokenizer_config.json
│ │ └── tokenizer.model
두 경우 모두 UI의 "모델" 탭을 사용하여 Hugging Face에서 모델을 자동으로 다운로드할 수 있습니다. 다음 명령줄을 통해 다운로드할 수도 있습니다.
python download-model.py organization/model
모든 옵션을 보려면 python download-model.py --help 실행하세요.
https://colab.research.google.com/github/oobabooga/text- Generation-webui/blob/main/Colab-TextGen-GPU.ipynb
2023년 8월, Andreessen Horowitz(a16z)는 이 프로젝트에 대한 나의 독립적 작업을 격려하고 지원하기 위해 넉넉한 보조금을 제공했습니다. 나는 그들의 신뢰와 인정에 진심 으로 감사드립니다.
