Source: MIT News
Although the capabilities of large language models are impressive, they are still far from perfect. These AI models can sometimes "hallucinate" and generate incorrect or unfounded information in response to queries.
Because of this illusory problem, model responses often need to be verified by human reviewers, especially in high-stakes environments such as healthcare or finance. However, the validation process often requires people to read long documents referenced by the model, a tedious and error-prone task that may deter some users from using generative AI models.
To aid human verifiers, MIT researchers have created a user-friendly system that enables people to more quickly verify the responses of large language models. The tool, called SymGen, allows LLM to generate responses with references that point directly to specific locations in the source document, such as a cell in a database.
Users can hover over highlighted portions of text responses to see the data the model used to generate specific words or phrases. Meanwhile, the unhighlighted portions show phrases that require additional attention for inspection and verification.
"We give people the ability to selectively focus on the parts of the text they need to pay more attention to. Ultimately, SymGen improves people's confidence in the model's response because they can easily double-check to ensure the information is reliable." Electrical Engineering & Computer Science said graduate student Shannon Shen, co-first author of the SymGen paper.
Through user research, Shen and his collaborators found that verification time using SymGen was reduced by about 20% compared to the manual process. By making the process of validating model output faster and simpler, SymGen helps identify errors in LLMs used in a variety of real-world applications, from generating clinical records to summarizing financial market reports.
Shen’s co-authors on the paper also include co-first author Lucas Torroba Hennigen, an EECS graduate student; EECS graduate student Aniruddha “Ani” Nrusimha; Bernhard Gapp, president of the Good Data Initiative; and senior author David Sontag, an EECS professor and member of the MIT Jameel Clinic, Computer Science with the head of the Clinical Machine Learning Group of the Artificial Intelligence Laboratory (CSAIL); and Assistant Professor yoon Kim, a member of CSAIL. The research was recently presented at a language modeling conference.
Symbol reference
To aid verification, many LLMs are designed to generate references to external documents and provide language-based responses for user inspection. However, these verification systems are often an afterthought and do not take into account the effort required of people to sift through large numbers of citations, Shen said.
“The purpose of generative AI is to reduce the time it takes users to complete a task. If you need to spend hours reading these documents to verify whether the model’s claims are reasonable, the generated content will be less helpful in real-world applications.” Shen said.
The researchers approached this question from the perspective of the person who would be doing the validation work.
SymGen users first provide LLM with data that can be used as a reference, such as a table containing basketball game statistics. The researchers then perform an intermediate step without immediately asking the model to complete a task, such as generating a match summary from this data. They prompt the model to generate responses in symbolic form.
With this prompt, whenever the model wishes to reference a word in a response, it must write out the specific cell in the data table that contains that information. For example, if the model wants to reference the phrase "Portland Trail Blazers" in a response, it will replace that text with the names of cells in the data table that contain those words.
"Because we have this intermediate step where the text is presented in a symbolic format, we can achieve very fine-grained referencing. We can clearly indicate which part of the data each piece of text in the output corresponds to," says Torroba Hennigen.
SymGen then uses rules-based tools to parse each reference, copying the corresponding text from the data table into the model's response.
"That way, we know it's copied verbatim, so we can make sure there's no error in the parts of the text that correspond to the actual data variables," Shen added.
Simplify verification
The model is able to generate symbolic responses due to the way it is trained. Large language models accept large amounts of data from the Internet, some of which is recorded in "placeholder format" with codes replacing actual values.
SymGen uses a similar structure when it prompts the model to generate symbolic responses.
"We designed the prompts in a specific way to unleash the capabilities of LLM," Shen added.
In user studies, most participants stated that SymGen made it easier to verify LLM-generated text. They verified model responses about 20% faster than using standard methods.
However, the effectiveness of SymGen is limited by the quality of the source data. The LLM may reference the wrong variables, and the human verifier may not be aware of this.
Additionally, users must provide source data in a structured format (such as a table) for input into SymGen. Currently, the system only works with tabular data.
Going forward, researchers are enhancing SymGen's capabilities to handle arbitrary text and other data forms. With this capability, it can help validate certain parts of AI-generated legal document summaries. They also plan to test SymGen with doctors to study how it identifies errors in AI-generated clinical summaries.
This work was funded in part by LiBERTy Mutual and the MIT Intelligent Discovery Initiative.