Speaker: Krisztian Balog, Professor at the University of Stavanger in Norway
Talk Title: What Does Conversational Information Access Exactly Mean and How to Evaluate It?
Date: Friday, January 15, 2021 - 1:00 - 2:00 PM EST (North American Eastern Standard Time) via Zoom
Zoom Access: Zoom Link and reach out to Alex Taubman for the passcode.
Abstract: In this talk, I'll identify a set of specific tasks and scenarios related to information access within the vast space that is casually referred to as conversational AI. While most of these problems have been identified in the literature for quite some time now, progress has been limited. Apart from the inherently challenging nature of these problems, the lack of progress, in large part, can be attributed to the shortage of appropriate evaluation methodology and resources. I'll present some recent work towards filling this gap.
In one line of research, we investigate the presentation of tabular search results in a conversational setting. Instead of generating a static summary of a result table, we complement brief summaries with clues that invite further exploration, thereby taking advantage of the conversational paradigm. One of the main contributions of this study is the development of a test collection using crowdsourcing.
Another line of work focuses on large-scale evaluation of conversational recommender systems via simulated users. Building on the well-established agenda-based simulation framework from dialogue systems research, we develop interaction and preference models specific to the item recommendation scenario. For evaluation, we compare three existing conversational movie recommender systems with both real and simulated users, and observe high correlation between the two means of evaluation.
Bio: Krisztian Balog is full professor at the University of Stavanger in Norway, where he leads the Information Access & Artificial Intelligence (IAI) research group. He also holds an Adjunct Professor in AI/NLP position at the Norwegian University of Science and Technology (NTNU). In 2018-19, he spent a sabbatical year at Google, London, UK, working as a Staff Visiting Faculty Researcher, then stayed affiliated with Google for an additional year, part-time.
Balog’s general research interests lie in the use and development of information retrieval, natural language processing, and machine learning techniques for intelligent information access tasks. His current research concerns entity-oriented and semantic search, and novel evaluation methodologies. He serves as a senior programme committee member at SIGIR, WSDM, WWW, CIKM and ECIR, and has previously served as an Associate Editor of the ACM Transactions on Information Systems as well as a coordinator of IR benchmarking efforts at TREC and CLEF. He served as Lab co-chair of CLEF’16, as program co-chair of ICTIR’19, and as general co-chair of ICTIR’20 and ECIR'22. Balog is the recipient of the 2018 Karen Spärck Jones Award, and is a member of the Norwegian Academy of Technological Sciences.