xFinder
v0.2.3 Released!
Qingchen Yu 1,* , Zifan Zheng 1,* , Lagu Shichao 2,* , Zhiyu Li 1,† , Feiyu Xiong 1 , Bo Tang 1 , Ding Chen 1
1 Institut Penelitian Algoritma Tingkat Lanjut, Shanghai, 2 Universitas Renmin Tiongkok
Siapa yang Harus Memperhatikan Pekerjaan Kita?
Penting
? Bintangi Kami! Dengan membintangi proyek kami di GitHub, Anda akan menerima semua notifikasi rilis secara instan. Kami menghargai dukungan Anda!
Kami merangkum kontribusi utama kami sebagai berikut:
Seperti yang ditunjukkan pada gambar, contoh-contoh di mana kerangka evaluasi seperti LM Eval Harness dan OpenCompass gagal mengekstraksi jawaban-jawaban kunci diilustrasikan. Secara khusus, A/T/C/M mewakili tugas dengan opsi alfabet/teks pendek/label kategori/matematika.
Buat Kumpulan Data Tolok Ukur : Untuk menyederhanakan proses evaluasi menggunakan xFinder, kami telah menstandarkan berbagai kumpulan data tolok ukur utama ke dalam format JSON terpadu. Untuk detail implementasi, lihat create_benchmark_dataset.py. Jika Anda ingin mengevaluasi kumpulan data Anda sendiri menggunakan xFinder, lihat templat skrip benchmark_dataset_template.py yang kami sediakan untuk panduan konversi format.
Siapkan Pasangan QA & Keluaran LLM : Kumpulkan keluaran LLM yang ingin Anda evaluasi. Pastikan data Anda menyertakan elemen berikut:
Terapkan Model xFinder : Pilih salah satu model berikut untuk diterapkan:
Setelah menerapkan model xFinder, ikuti langkah-langkah berikut untuk menjalankan evaluasi:
# Install xfinder
conda create -n xfinder_env python=3.10 -y
conda activate xfinder_env
pip install xfinder
# Perform an evaluation with xFinder (a built-in example)
CUDA_VISIBLE_DEVICES=0 python -m xfinder.eval --run-example --model-name xFinder-qwen1505 --inference-mode local --model-path-or-url /path/to/anonymized/model/xFinder-qwen1505Metode ini memungkinkan Anda mengevaluasi beberapa contoh yang disimpan dalam file JSON.
# Initialize Evaluator object
evaluator = Evaluator (
model_name = "xFinder-qwen1505" , # Model name
inference_mode = "api" , # Inference mode, 'local' or 'api'
model_path_or_url = "http://your-anonymized-url/generate" , # Anonymized model path or URL
)
# Perform batch evaluation
data_path = "/path/to/your/data/example.json" # User needs to provide their own data path
accuracy = evaluator . evaluate ( data_path )
print ( f"Batch evaluation accuracy: { accuracy } " )Metode ini memungkinkan Anda mengevaluasi contoh individual, yang dapat diintegrasikan ke dalam kerangka evaluasi LLM.
# Initialize Evaluator object
evaluator = Evaluator (
model_name = "xFinder-qwen1505" , # Model name
inference_mode = "local" , # Inference mode, 'local' or 'api'
model_path_or_url = "IAAR-Shanghai/xFinder-qwen1505" , # Anonymized model path or URL
)
# Define input for a single evaluation
question = "What is the capital of France?"
llm_output = "The capital of France is Paris."
standard_answer_range = "[ " Paris " , " Lyon " , " Marseille " ]"
key_answer_type = "short_text"
correct_answer = "Paris"
# Perform single example evaluation
result = evaluator . evaluate_single_example (
question ,
llm_output ,
standard_answer_range ,
key_answer_type ,
correct_answer
)Tip
demo.ipynb untuk contoh lebih detail.export HF_ENDPOINT=https://hf-mirror.com untuk menggunakan mirror berbahasa Mandarin jika Anda tidak dapat terhubung ke Hugging Face.Kami mendemonstrasikan contoh di empat jenis pertanyaan di mana RegEx gagal mengekstrak atau sering mengekstrak jawaban yang salah, sedangkan xFinder secara akurat mengekstrak jawaban utama.
{
"key_answer_type" : " alphabet option " ,
"question" : " A man is seen playing guitar on a stage with others playing instruments behind him. The man grabs a guitar from the audience and begins playing both one after the other ... " ,
"llm_output" : " Option A is the correct choice as it describes ... " ,
"standard_answer_range" : " [['A', 'strums the guitar in the end, continues playing the guitar with the crowd following him as well as lining up next to him.'], ['B', 'continues playing the instruments and ends by waving to the crowd and walking off stage.'], ['C', 'then turns to the audience and gives a stuffed toy to the audience and continues playing.'], ['D', 'finally stops playing and moves his hands for the crowd to see.']] " ,
"gold_label" : " A " ,
"xFinder_output" : " A " ,
},
{
"key_answer_type" : " short text " ,
"question" : " If you really wanted a grape, where would you go to get it? Answer Choices: winery / fruit stand / field / kitchen / food " ,
"llm_output" : " The answer is winery / fruit stand / field / kitchen / food ... " ,
"standard_answer_range" : " [ " winery " , " fruit stand " , " field " , " kitchen " , " food " ] " ,
"gold_label" : " [No valid answer] " ,
"xFinder_output" : " [No valid answer] " ,
},
{
"key_answer_type" : " categorical label " ,
"question" : " How tall is the Sears Building ? " ,
"llm_output" : " The Sears Building is a specific structure, so the answer would be a Location ... " ,
"standard_answer_range" : " ['Abbreviation', 'Entity', 'Description', 'Person', 'Location', 'Number'] " ,
"gold_label" : " Location " ,
"xFinder_output" : " Location " ,
},
{
"key_answer_type" : " math " ,
"question" : " Mike made 69 dollars mowing lawns over the summer. If he spent 24 dollars buying new mower blades, how many 5 dollar games could he buy with the money he had left? " ,
"llm_output" : " To find out how many 5 dollar ... Let's calculate that: nn $45 / $5 = 9 nn So, Mike could buy 9 5 dollar games with the money he had left. " ,
"standard_answer_range" : " a(n) number / set / vector / matrix / interval / expression / function / equation / inequality " ,
"gold_label" : " 9 " ,
"xFinder_output" : " 9 " ,
}Dasar : OpenCompass, LM Eval Harness, UltraEval, GPT-4. Metode Kami : xFinder-qwen1505, xFinder-qwen1518, xFinder-gemma7, xFinder-chatglm36base, xFinder-llama38, xFinder-llama38it.
Kami mengevaluasi keakuratannya dalam mengekstraksi jawaban utama dari set pengujian KAF dan set generalisasi. Metrik dalam tabel adalah akurasi.
@article{xFinder,
title={xFinder: Robust and Pinpoint Answer Extraction for Large Language Models},
author={Qingchen Yu and Zifan Zheng and Shichao Song and Zhiyu Li and Feiyu Xiong and Bo Tang and Ding Chen},
journal={arXiv preprint arXiv:2405.11874},
year={2024},
}
?Kembali ke atas
