The scientific research efficiency artifact OpenScholar is released, completely changing the literature review experience of scientific researchers! The editor of Downcodes brings you this AI-driven scientific research tool. It has 450 million open access papers and 237 million article paragraph embeddings. It can quickly and accurately filter out documents related to your scientific research questions and generate complete Answers to references. OpenScholar is not only powerful, it can also learn and improve itself, continuously improve the quality of answers, and ultimately present the most perfect scientific research results. It can also be used to train smaller and more efficient models, bringing revolutionary changes to the field of scientific research!
Staying up late to review literature? Scratching your head and writing a paper? Don’t panic! The scientific research experts of AI2 are here to save you with their latest masterpiece OpenScholar! This scientific research efficiency artifact can make literature review as easy and enjoyable as walking in the park!
The biggest secret weapon of OpenScholar is a system called OpenScholar-Datastore (OSDS) with 450 million open access papers and 237 million embedded article paragraphs. With such a strong knowledge base, OpenScholar can cope with various scientific research problems with ease.
When you encounter a scientific research problem, OpenScholar will first send out its powerful tools - the searcher and reorderer, to quickly filter out the article paragraphs related to your problem from OSDS. Next, a language model (LM) contains the complete answer for the reference. What's even more powerful is that OpenScholar will continue to improve the answers based on your natural language feedback and supplement the missing information until you are satisfied.
OpenScholar is not only powerful on its own, but can also help train smaller and more efficient models. The researchers used OpenScholar's process to generate massive amounts of high-quality training data, and used this data to train an 8 billion parameter language model called OpenScholar-8B, as well as other retrieval models.
In order to comprehensively test the combat effectiveness of OpenScholar, the researchers also specially created a new test arena called SCHOLARQABENCH. A variety of scientific literature review tasks are set up in this arena, including closed classification, multiple choice, and long-form generation, covering multiple fields such as computer science, biomedicine, physics, and neuroscience. In order to ensure the fairness and justice of the competition, SCHOLARQABENCH also uses multi-faceted evaluation methods, including expert review, automatic indicators and user experience testing.
After many rounds of fierce competition, OpenScholar finally stood out! Experimental results showed that it performed well in various tasks, even surpassing human experts! This breakthrough result will surely set off a revolution in the field of scientific research and let scientists say goodbye The hard work of literature review, focusing on exploring the mysteries of science!
The powerful functions of OpenScholar mainly benefit from its unique self-feedback retrieval enhanced reasoning mechanism. To put it simply, it will first ask itself questions, then continuously improve the answers based on its own answers, and finally present the most perfect answer to you. Isn't it amazing?
Specifically, OpenScholar's self-feedback reasoning process is divided into three steps: initial answer generation, feedback generation, and feedback integration. First, the language model generates an initial answer based on the retrieved article passages. Then, like a stern examiner, it will self-criticize its answers, identify shortcomings, and generate some natural language feedback, such as "The answer only contains experimental results on question and answer tasks, please supplement other types of tasks." result". Finally, the language model will re-search the relevant literature based on this feedback and integrate all the information to generate a more complete answer.
In order to train smaller but equally powerful models, the researchers also used OpenScholar's self-feedback inference process to generate large amounts of high-quality training data. They first selected the most cited papers from the database, then generated some information query questions based on the abstracts of these papers, and finally used OpenScholar's inference process to generate high-quality answers. These answers and the feedback information generated in the process constitute valuable training data. The researchers mixed this data with existing general domain instruction fine-tuning data and scientific domain instruction fine-tuning data to train an 8 billion parameter language model called OpenScholar-8B.
To more fully evaluate the performance of OpenScholar and other similar models, the researchers also created a new benchmark called SCHOLARQABENCH. This benchmark contains 2,967 literature review questions written by experts covering four fields: computer science, physics, biomedicine, and neuroscience. Each question has a lengthy answer written by an expert, and on average each answer takes an expert about an hour to complete. SCHOLARQABENCH also employs a multifaceted evaluation approach that combines automated metrics and manual evaluation to provide a more comprehensive measure of the quality of the answers generated by the model.
Experimental results show that OpenScholar's performance on SCHOLARQABENCH far exceeds other models and even surpasses human experts in some aspects! For example, in the field of computer science, OpenScholar-8B's correct rate is 5% higher than GPT-4o, which is 5% higher than that of GPT-4o. PaperQA2 is 7% higher. Moreover, the citation accuracy of the answers generated by OpenScholar is comparable to that of human experts, while GPT-4o is as high as 78-90% fabricated out of thin air.
The emergence of OpenScholar is undoubtedly a great boon to the field of scientific research! It can not only help scientific researchers save a lot of time and energy, but also improve the quality and efficiency of literature reviews. I believe that in the near future, OpenScholar will become an indispensable assistant for scientific researchers!
Paper address: https://arxiv.org/pdf/2411.14199
Project address: https://github.com/AkariAsai/OpenScholar
All in all, with its powerful functions and efficient performance, OpenScholar has brought unprecedented convenience to scientific researchers and greatly improved scientific research efficiency. It is not only a tool, but also a revolution in the field of scientific research. It is worth looking forward to its future development and application.