xFinder
v0.2.3 Released!
於慶辰1,* 、鄭子凡1,* 、宋世超2,* 、李志宇1,† 、熊飛宇1 、唐博1 、丁晨1
1先進演算法研究院，上海， 2中國人民大學
誰該關注我們的工作？
重要的
？為我們加星！透過在 GitHub 上為我們的專案加註星標，您將立即收到所有發布通知。感謝您的支持！
我們的主要貢獻總結如下：
如圖所示，展示了 LM Eval Harness 和 OpenCompass 等評估框架未能提取關鍵答案的實例。具體來說，A/T/C/M 分別表示帶有字母/短文字/分類標籤/數學選項的任務。
建立基準資料集：為了簡化使用xFinder的評估流程，我們將各種主流基準資料集標準化為統一的JSON格式。有關實作細節，請參閱create_benchmark_dataset.py。如果您希望使用 xFinder 評估您自己的資料集，請參閱我們提供的腳本範本 benchmark_dataset_template.py 以取得格式轉換指南。
準備 QA 對和 LLM 輸出：收集您想要評估的 LLM 輸出。確保您的資料包含以下元素：
部署 xFinder 模型：選擇下列模型之一進行部署：
部署 xFinder 模型後，請依照下列步驟執行評估：
# Install xfinder
conda create -n xfinder_env python=3.10 -y
conda activate xfinder_env
pip install xfinder
# Perform an evaluation with xFinder (a built-in example)
CUDA_VISIBLE_DEVICES=0 python -m xfinder.eval --run-example --model-name xFinder-qwen1505 --inference-mode local --model-path-or-url /path/to/anonymized/model/xFinder-qwen1505此方法可讓您評估儲存在 JSON 檔案中的多個範例。
# Initialize Evaluator object
evaluator = Evaluator (
model_name = "xFinder-qwen1505" , # Model name
inference_mode = "api" , # Inference mode, 'local' or 'api'
model_path_or_url = "http://your-anonymized-url/generate" , # Anonymized model path or URL
)
# Perform batch evaluation
data_path = "/path/to/your/data/example.json" # User needs to provide their own data path
accuracy = evaluator . evaluate ( data_path )
print ( f"Batch evaluation accuracy: { accuracy } " )此方法可讓您評估單一範例，這些範例可以整合到法學碩士評估框架中。
# Initialize Evaluator object
evaluator = Evaluator (
model_name = "xFinder-qwen1505" , # Model name
inference_mode = "local" , # Inference mode, 'local' or 'api'
model_path_or_url = "IAAR-Shanghai/xFinder-qwen1505" , # Anonymized model path or URL
)
# Define input for a single evaluation
question = "What is the capital of France?"
llm_output = "The capital of France is Paris."
standard_answer_range = "[ " Paris " , " Lyon " , " Marseille " ]"
key_answer_type = "short_text"
correct_answer = "Paris"
# Perform single example evaluation
result = evaluator . evaluate_single_example (
question ,
llm_output ,
standard_answer_range ,
key_answer_type ,
correct_answer
)提示
demo.ipynb 。export HF_ENDPOINT=https://hf-mirror.com以使用中文鏡像。我們示範了四種類型問題的實例，其中 RegEx 無法提取或經常提取錯誤答案，而 xFinder 可以準確地提取關鍵答案。
{
"key_answer_type" : " alphabet option " ,
"question" : " A man is seen playing guitar on a stage with others playing instruments behind him. The man grabs a guitar from the audience and begins playing both one after the other ... " ,
"llm_output" : " Option A is the correct choice as it describes ... " ,
"standard_answer_range" : " [['A', 'strums the guitar in the end, continues playing the guitar with the crowd following him as well as lining up next to him.'], ['B', 'continues playing the instruments and ends by waving to the crowd and walking off stage.'], ['C', 'then turns to the audience and gives a stuffed toy to the audience and continues playing.'], ['D', 'finally stops playing and moves his hands for the crowd to see.']] " ,
"gold_label" : " A " ,
"xFinder_output" : " A " ,
},
{
"key_answer_type" : " short text " ,
"question" : " If you really wanted a grape, where would you go to get it? Answer Choices: winery / fruit stand / field / kitchen / food " ,
"llm_output" : " The answer is winery / fruit stand / field / kitchen / food ... " ,
"standard_answer_range" : " [ " winery " , " fruit stand " , " field " , " kitchen " , " food " ] " ,
"gold_label" : " [No valid answer] " ,
"xFinder_output" : " [No valid answer] " ,
},
{
"key_answer_type" : " categorical label " ,
"question" : " How tall is the Sears Building ? " ,
"llm_output" : " The Sears Building is a specific structure, so the answer would be a Location ... " ,
"standard_answer_range" : " ['Abbreviation', 'Entity', 'Description', 'Person', 'Location', 'Number'] " ,
"gold_label" : " Location " ,
"xFinder_output" : " Location " ,
},
{
"key_answer_type" : " math " ,
"question" : " Mike made 69 dollars mowing lawns over the summer. If he spent 24 dollars buying new mower blades, how many 5 dollar games could he buy with the money he had left? " ,
"llm_output" : " To find out how many 5 dollar ... Let's calculate that: nn $45 / $5 = 9 nn So, Mike could buy 9 5 dollar games with the money he had left. " ,
"standard_answer_range" : " a(n) number / set / vector / matrix / interval / expression / function / equation / inequality " ,
"gold_label" : " 9 " ,
"xFinder_output" : " 9 " ,
}基線：OpenCompass、LM Eval Harness、UltraEval、GPT-4。我們的方法：xFinder-qwen1505、xFinder-qwen1518、xFinder-gemma7、xFinder-chatglm36base、xFinder-llama38、xFinder-llama38it。
我們評估了它們從 KAF 測試集和泛化集提取關鍵答案的準確性。表中的指標是準確性。
@article{xFinder,
title={xFinder: Robust and Pinpoint Answer Extraction for Large Language Models},
author={Qingchen Yu and Zifan Zheng and Shichao Song and Zhiyu Li and Feiyu Xiong and Bo Tang and Ding Chen},
journal={arXiv preprint arXiv:2405.11874},
year={2024},
}
?回到頂部
