Demographically-Inspired Query Variants Using an LLM

Abstract

This study proposes a method to diversify queries in existing test collections to reflect some of the diversity of search engine users, aligning with an earlier vision of an `ideal’ test collection. A LLM is used to create query variants: alternative queries that have the same meaning as the original. These variants represent user profiles characterised by different properties, such as language and domain proficiency, which are known in the IR literature to influence query formulation.

The LLM’s ability to generate query variants that align with user profiles is empirically validated, and the variants’ utility is further explored for IR system evaluation. Results demonstrate that the variants impact how systems are ranked and show that user profiles experience significantly different levels of system effectiveness. This method enables an alternative perspective on system evaluation where we can observe both the impact of user profiles on system rankings and how system performance varies across users.

Citation

If you find this paper useful, please cite it using the following BibTeX:

@INPROCEEDINGS{Alaofi-2025-demographic,
  AUTHOR = {Alaofi, Marwah and Ferro, Nicola and Thomas, Paul and Scholer, Falk and Sanderson, Mark},
  TITLE = {Demographically-Inspired Query Variants Using an LLM},
  YEAR = {2025},
  PUBLISHER = {Association for Computing Machinery},
  ADDRESS = {New York, NY, USA},
  URL = {https://doi.org/10.1145/3731120.3744608},
  DOI = {10.1145/3731120.3744608},
  BOOKTITLE = {Proceedings of the 2025 ACM International Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR’25)},
  LOCATION = {Padua, Italy},
  SERIES = {ICTIR '25}
}