[RAG] LLM-based Query Rewriting 논문 리뷰 (1) Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting

https://arxiv.org/abs/2310.09716

Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting

 

요약

 

본 논문은 LLM을 query rewriters로 활용, well-designed instructions을 통해 유익한 query rewrites를 생성할 것을 제안함. well-formed rewrites를 위해 four essential properties를 정의하고, 모든 속성을 instruction에 통합. 또한 initial query rewrites가 가능할 때 LLM에 대한 rewrite editors의 role을 소개하여 a “rewrite-then-edit” 프로세스를 구성. 또한 rewriting latency을 줄이기 위해 LLM의 rewriting기능을 더 작은 모델로 distilling하는 것을 제안함.

 

 

Keyword

instructions, four essential properties, role - rewrite editors, distilling

 

Info 

EMNLP Findings 2023

 

 

Abstract

 

insruction을 통해 LLM을 query rewriters와 rewrite editors 둘 다로 prompt 하는 것을 제안함.

We propose to prompt LLMs as both query rewriters and rewrite editors by providing clear instructions that incorporate all the desirable properties.

 

 

3.1 Prompting LLM as Query Rewriter

 

• four desirable properties that a well-crafted rewritten query should possess
  • Correctness: The rewritten query should preserve the meaning of the original query, ensuring that the user’s intent remains unchanged.
  • Clarity: The rewritten query should be unambiguous and independent of the conversational context, enabling it to be comprehensible by people outside the conversational context. This clarity can be achieved by addressing coreference and omission issues arising in the original query.
  • Informativeness: The rewritten query should incorporate as much valuable and relevant information from the conversational context as possible, thereby providing more useful information to the off-the-shelf retriever.
  • Nonredundancy: The rewritten query should avoid duplicating any query previously raised in the conversational context, as it is important to ensure that the rewritten query only conveys the intent and meaning of the current query.
• instruction

“Given a question and its context, decontextualize the question by addressing coreference and omission issues. The resulting question should retain its original meaning and be as informative as possible, and should not duplicate any previously asked questions in the context.”

 

 

이를 적용해서, 2가지 셋팅으로 실험

 

 

1. Zero-Shot Learning (ZSL) Setting
    
   • 어떤 human-labeled instances도 없이 current query Qt와 그에 상응하는 conversational context Xt만을 제공
   • Qt와 Xt를 instruction I에 concatenation해서 the rewrite Q′t를 샘플링하도록 LLM에게 prompt로 제공 

• Zero-Shot prompt

 

 

2. Few-Shot Learning (FSL) Setting

 

• instruction과 소량의 demonstrations을 제공, in-context learning을 활용한 Few-Shot Learning 진행
• 앞의 four properties를 명시하고 있는 well-designed demonstrations을 제공해야 함

• Few-Shot prompt for Query Rewriters

 

 

3.2 Prompting LLM as Rewrite Editor

 

This approach draws inspiration from the observation that humans often find it easier to edit existing content than to create it from scratch.

• LLM을 the query rewriter와 rewrite editor 둘 다로 사용하면서, the “rewrite-then-edit” process 가 LLM이 self-correction을 수행하도록 함

• adopt the FSL setting to prompt LLMs as rewrite editors → 각 demonstrations에 initial rewrite Qˆ 추가. 이는 T5QR와 같이 작은 query rewriting model을 이용해서 생성해도 되며, 저자들은 LLM으로 생성하게 함.

 

• Few-Shot prompt for Editors

 

3.3 Distillation: LLM as Rewriting Teacher

• In addition, we employ distillation techniques to condense the rewriting capabilities of LLMs into smaller models to improve rewriting efficiency.
• rewriting efficiency를 개선하기 위해, LLM이 생성한 rewrites를 ground-truth labels로 사용해서 small query rewriting model을 rewriter로 fine-tune하는 것을 추가로 제안.
• LLM을 활용한 knowledge distillation 진행, T5를 the student model로 선정함
• Model input : the concatenation of all utterances in the conversational context Xt and the current user query Qt
• user queries와 system responses를 구분하기 위해 each user query에 special token **<Que>**를, ****each system response에 special token <Ans> 를 추가함.
• standard cross-entropy loss 사용

 

 

4 Experimental Setup

 

4.1 Evaluation Metrics

• mean reciprocal rank (MRR), mean average precision (MAP), Recall@10 (R@10) 사용.
• pytrec_eval toolkit 사용해서 계산

 

4.2 Comparison Methods

• Original : 원래 쿼리
• Human : 사람이 재작성한 쿼리
• T5QR : fine-tunes the T5-base model as a seq2seq query rewriter
• ConQRR
• ConvGQR
• RW(ZSL) : prompts an LLM as a query rewriter in the ZSL setting
• RW(FSL) : prompts an LLM as a query rewriter in the FSL setting
• ED(Self) : prompts an LLM as a rewrite editor, wherein the initial rewrites are generated by RW(FSL) with the same LLM applied.
• ED(T5QR) : also prompts an LLM as a rewrite editor, but the initial rewrites are generated by T5QR.

* Blue : 저자들이 제안한 방식

For simplicity, we only prompt LLMs as rewrite editors in the FSL setting.

 

4.3 Retrieval Systems

• 2개의 off-the-shelf retrievers를 사용해서 성능 평가 진행
  1. BM25를 Sparse retriever로 사용
  2. GTR : Dense retriever로 사용