SafeGen_CCS2024

介紹

這是「SafeGen：減少文字到圖像模型中色情內容生成」的官方程式​​碼

SafeGen 將出現在 ACM 電腦和通訊安全會議 ( ACM CCS 2024 ) Core-A*、CCF-A、Big 4 。我們已經在 ArXiv 上發布了相機就緒版本。

我們已經在 Hugging Face 上發布了預訓練模型。請查看如何使用它進行推理？

我們的發布涉及基於僅影像三元組單獨調整穩定擴散的自註意力層。

該實作可以視為可以整合到 Diffusers 庫中的範例。因此，您可以導航到 Examples/text_to_image/ 資料夾，並了解它是如何運作的。

引文

如果您發現我們的論文/代碼/基準有幫助，請考慮透過以下參考文獻引用這項工作：

 @inproceedings{li2024safegen,
  author       = {Li, Xinfeng and Yang, Yuchen and Deng, Jiangyi and Yan, Chen and Chen, Yanjiao and Ji, Xiaoyu and Xu, Wenyuan},
  title        = {{SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models}},
  booktitle    = {Proceedings of the 2024 {ACM} {SIGSAC} Conference on Computer and Communications Security (CCS)},
  year         = {2024},
}

或者

 @article{li2024safegen,
  title={{SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models}},
  author={Li, Xinfeng and Yang, Yuchen and Deng, Jiangyi and Yan, Chen and Chen, Yanjiao and Ji, Xiaoyu and Xu, Wenyuan},
  journal={arXiv preprint arXiv:2404.06666},
  year={2024}
}

環境與安裝

您可以使用單一 A100-40GB (NVIDIA) 並採用我們的預設配置來執行此程式碼。特別是，設定一個小的training_batch_size以避免記憶體不足錯誤。

我們建議您管理兩個 conda 環境以避免依賴項衝突。

• 用於調整穩定擴散模型的自註意力層的Pytorch環境以及與評估相關的庫。

• 反深裸模型資料準備階段所需的Tensorflow環境。

PyTorch + 擴散器的要求

 # You can install the main dependencies by conda/pip
conda create -n text-agnostic-t2i python=3.8.5
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
# Using the official Diffusers package
pip install --upgrade diffuers[torch]
# Or you may use the community maintained version
conda install -c conda-forge diffusers
...

 # Or you can create the env via environment.yaml
conda env create -f environment_pytorch.yaml 

僅影像資料準備的要求

由於反deepnude模型需要TensorFlow 1.13，因此可以安裝Python<=3.7。

 # You can install the dependencies individually
conda create -n anti-deepndue python=3.7
pip install tensorflow-gpu==1.13.1 keras==2.2.4
...

 # Or you can create the env via environment.yaml
conda env create -f environment_tf.yaml 

Anti-DeepNude 資料準備

您可以透過以下方式取得<nude，mosaic>對：

python anti-deepnude.py

模型治理使用

 export MODEL_NAME= " CompVis/stable-diffusion-v1-4 " # export the model you want to protect
export TRAIN_DIR= " <path to the manifest of your training files>/meta.csv " # export your image triplets folder
accumulation_steps=5
learning_rate=1e-05
steps=1000
train_batch_size=1
warmup_steps=200
weight_tune=0.1
weight_preserve=0.9
defend_type= " nudity_advtrain "
current_time= $( date + ' %m%d_%H%M ' )
output_dir= " mos_model_sal_ " ${defend_type} - " step_ " ${steps} - " lr_ " ${learning_rate} - " warm_ " ${warmup_steps} - ${weight_tune} + ${weight_preserve} " gacc_ " ${accumulation_steps} - ${current_time}

export CUDA_VISIBLE_DEVICES=0
accelerate launch --mixed_precision= " fp16 " " --num_processes=1 " train_text_to_image_adjust_sal.py 
  --pretrained_model_name_or_path= $MODEL_NAME 
  --train_data_dir= $TRAIN_DIR 
  --use_ema 
  --resolution=512 --center_crop
  --train_batch_size ${train_batch_size} 
  --gradient_accumulation_steps ${accumulation_steps} 
  --gradient_checkpointing 
  --max_train_steps ${steps} 
  --learning_rate ${learning_rate} 
  --image_column= " nude_image_path " 
  --max_grad_norm=1 
  --lr_scheduler= " constant " 
  --lr_warmup_steps ${warmup_steps} 
  --output_dir ${output_dir} 
  --weight_tune ${weight_tune} 
  --weight_preserve ${weight_preserve} 
  --checkpoints_total_limit 0 
  --validation_prompts " woman, nudity " " a running dog on grass " " a woman standing aside the colorful table " " a nude sexy man, blue eyes "
  
  # In addition, you can set the 'validation_prompts' to visually monitor the training efficacy, such as "a running dog on grass", "a woman standing aside the colorful table", "a nude sexy man, blue eyes", etc.

只需運行腳本即可

run_adjust_SD.sh

如何使用監理模型？

from diffusers import StableDiffusionPipeline
import torch

model_path = ${output_dir} # the save path of your model
pipeline = StableDiffusionPipeline.from_pretrained(model_path, torch_dtype=torch.float16)
pipeline.to( " cuda " )

prompt = " a photo of an astronaut riding a horse on mars "
image = pipeline(prompt).images[0]
image.save( " example.png " )

對抗性文字提示基準

已經開發了超過 50,000 個文本對抗性提示，包括看似無害的自我優化提示，以測試 T2I 模型在生成露骨色情內容方面的潛在利用。由於這些圖像的敏感性，訪問受到限制以確保道德合規性。有興趣將這些圖像用於學術目的的研究人員必須承諾不再進一步分發它們。請與我聯繫以請求訪問並討論必要的保障措施。我的信箱地址是：[email protected]。

致謝

這項工作基於令人驚嘆的研究工作和開源項目，非常感謝所有作者的分享！

• ？擴散器

@misc{von-platen-etal-2022-diffusers,
    author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Thomas Wolf},
    title = {Diffusers: State-of-the-art diffusion models},
    year = {2022},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = { url {https://github.com/huggingface/diffusers}}
}

• 反深裸

• 清潔-Fid

@inproceedings{parmar2021cleanfid,
  title={On Aliased Resizing and Surprising Subtleties in GAN Evaluation},
  author={Parmar, Gaurav and Zhang, Richard and Zhu, Jun-Yan},
  booktitle={CVPR},
  year={2022}
}

• LPIPS分數

@inproceedings{zhang2018perceptual,
  title={The Unreasonable Effectiveness of Deep Features as a Perceptual Metric},
  author={Zhang, Richard and Isola, Phillip and Efros, Alexei A and Shechtman, Eli and Wang, Oliver},
  booktitle={CVPR},
  year={2018}
}