xFinder
v0.2.3 Released!
于庆辰1,* 、郑子凡1,* 、宋世超2,* 、李志宇1,† 、熊飞宇1 、唐博1 、丁晨1
1先进算法研究院，上海， 2中国人民大学
谁应该关注我们的工作？
重要的
？为我们加星！通过在 GitHub 上为我们的项目加注星标，您将立即收到所有发布通知。感谢您的支持！
我们的主要贡献总结如下：
如图所示，展示了 LM Eval Harness 和 OpenCompass 等评估框架未能提取关键答案的实例。具体来说，A/T/C/M 分别表示带有字母/短文本/分类标签/数学选项的任务。
创建基准数据集：为了简化使用xFinder的评估过程，我们将各种主流基准数据集标准化为统一的JSON格式。有关实现细节，请参阅create_benchmark_dataset.py。如果您希望使用 xFinder 评估您自己的数据集，请参阅我们提供的脚本模板 benchmark_dataset_template.py 以获取格式转换指南。
准备 QA 对和 LLM 输出：收集您想要评估的 LLM 输出。确保您的数据包含以下元素：
部署 xFinder 模型：选择以下模型之一进行部署：
部署 xFinder 模型后，请按照以下步骤运行评估：
# Install xfinder
conda create -n xfinder_env python=3.10 -y
conda activate xfinder_env
pip install xfinder
# Perform an evaluation with xFinder (a built-in example)
CUDA_VISIBLE_DEVICES=0 python -m xfinder.eval --run-example --model-name xFinder-qwen1505 --inference-mode local --model-path-or-url /path/to/anonymized/model/xFinder-qwen1505此方法允许您评估存储在 JSON 文件中的多个示例。
# Initialize Evaluator object
evaluator = Evaluator (
model_name = "xFinder-qwen1505" , # Model name
inference_mode = "api" , # Inference mode, 'local' or 'api'
model_path_or_url = "http://your-anonymized-url/generate" , # Anonymized model path or URL
)
# Perform batch evaluation
data_path = "/path/to/your/data/example.json" # User needs to provide their own data path
accuracy = evaluator . evaluate ( data_path )
print ( f"Batch evaluation accuracy: { accuracy } " )此方法允许您评估单个示例，这些示例可以集成到法学硕士评估框架中。
# Initialize Evaluator object
evaluator = Evaluator (
model_name = "xFinder-qwen1505" , # Model name
inference_mode = "local" , # Inference mode, 'local' or 'api'
model_path_or_url = "IAAR-Shanghai/xFinder-qwen1505" , # Anonymized model path or URL
)
# Define input for a single evaluation
question = "What is the capital of France?"
llm_output = "The capital of France is Paris."
standard_answer_range = "[ " Paris " , " Lyon " , " Marseille " ]"
key_answer_type = "short_text"
correct_answer = "Paris"
# Perform single example evaluation
result = evaluator . evaluate_single_example (
question ,
llm_output ,
standard_answer_range ,
key_answer_type ,
correct_answer
)提示
demo.ipynb 。export HF_ENDPOINT=https://hf-mirror.com以使用中文镜像。我们演示了四种类型问题的实例，其中 RegEx 无法提取或经常提取错误答案，而 xFinder 可以准确提取关键答案。
{
"key_answer_type" : " alphabet option " ,
"question" : " A man is seen playing guitar on a stage with others playing instruments behind him. The man grabs a guitar from the audience and begins playing both one after the other ... " ,
"llm_output" : " Option A is the correct choice as it describes ... " ,
"standard_answer_range" : " [['A', 'strums the guitar in the end, continues playing the guitar with the crowd following him as well as lining up next to him.'], ['B', 'continues playing the instruments and ends by waving to the crowd and walking off stage.'], ['C', 'then turns to the audience and gives a stuffed toy to the audience and continues playing.'], ['D', 'finally stops playing and moves his hands for the crowd to see.']] " ,
"gold_label" : " A " ,
"xFinder_output" : " A " ,
},
{
"key_answer_type" : " short text " ,
"question" : " If you really wanted a grape, where would you go to get it? Answer Choices: winery / fruit stand / field / kitchen / food " ,
"llm_output" : " The answer is winery / fruit stand / field / kitchen / food ... " ,
"standard_answer_range" : " [ " winery " , " fruit stand " , " field " , " kitchen " , " food " ] " ,
"gold_label" : " [No valid answer] " ,
"xFinder_output" : " [No valid answer] " ,
},
{
"key_answer_type" : " categorical label " ,
"question" : " How tall is the Sears Building ? " ,
"llm_output" : " The Sears Building is a specific structure, so the answer would be a Location ... " ,
"standard_answer_range" : " ['Abbreviation', 'Entity', 'Description', 'Person', 'Location', 'Number'] " ,
"gold_label" : " Location " ,
"xFinder_output" : " Location " ,
},
{
"key_answer_type" : " math " ,
"question" : " Mike made 69 dollars mowing lawns over the summer. If he spent 24 dollars buying new mower blades, how many 5 dollar games could he buy with the money he had left? " ,
"llm_output" : " To find out how many 5 dollar ... Let's calculate that: nn $45 / $5 = 9 nn So, Mike could buy 9 5 dollar games with the money he had left. " ,
"standard_answer_range" : " a(n) number / set / vector / matrix / interval / expression / function / equation / inequality " ,
"gold_label" : " 9 " ,
"xFinder_output" : " 9 " ,
}基线：OpenCompass、LM Eval Harness、UltraEval、GPT-4。我们的方法：xFinder-qwen1505、xFinder-qwen1518、xFinder-gemma7、xFinder-chatglm36base、xFinder-llama38、xFinder-llama38it。
我们评估了它们从 KAF 测试集和泛化集提取关键答案的准确性。表中的指标是准确性。
@article{xFinder,
title={xFinder: Robust and Pinpoint Answer Extraction for Large Language Models},
author={Qingchen Yu and Zifan Zheng and Shichao Song and Zhiyu Li and Feiyu Xiong and Bo Tang and Ding Chen},
journal={arXiv preprint arXiv:2405.11874},
year={2024},
}
?回到顶部
