Evaluating user search trails in exploratory search tasks

摘要

Exploratory search is characterized by a user’s uncertainty towards a complex information seeking task. A user conducting such a search in an information retrieval (IR) system may need help and recommendations that are beyond mere query suggestions. In this paper we propose a new method for recommending search trails to struggling users. We first use a search process prediction model from the literature to predict whether a user is likely to under-perform in an exploratory search task, and given that case, recommend a search trail based on other users’ search behaviors in a similar context. We then present a method to evaluate the effectiveness of these recommendations that involves two different evaluation criteria. First, we use Open Directory Project (ODP)-based categorization of user-traversed Web pages to evaluate each user’s information coverage. Next, we evaluate the order of users’ search trails while simultaneously incorporating a novel set of metrics that use adjacency of queries issued and Web pages traversed. To evaluate search trails, we incorporated proposed metrics with transactional log data from multiple user studies in which more than 300 users conducted exploratory search tasks on different topics.We demonstrate the effectiveness of the proposed evaluation criteria by measuring how the recommended search trails lead to improvements in both information space coverage and search performance metrics for users across multiple user search datasets. Based on the analysis results, we demonstrate that the order of the recommended search trails plays a significant role and it outperforms the random order of search trails thus being beneficial for the struggling users in improving their overall search effectiveness. We also show that by providing search trail recommendations, users are able to discover more information across multiple facets (in breadth) as well as investigate certain facets in more detail (in depth). These findings provide substantial evidence across multiple datasets to confirm that recommended search trails improve users’ information seeking coverage and overall knowledge acquisition throughout their search processes.