topauthor: Ruthven_I* : Human-Computer Interaction Bibliography Search

[1] Grieving online: the use of search engines in times of grief and bereavement Web search behaviour / Ruthven, Ian Proceedings of the 2012 Symposium on Information Interaction in Context 2012-08-21 p.120-128

ACM Digital Library Link

Summary: In this paper, we study the information goals in searches on the topic of grieving and bereavement. Using log analyses and content analyses we present a categorization system of grief and bereavement-related queries submitted to a major search engine, reflecting the variety of information needs that occur during a period of bereavement. We also present patterns of interaction during the query formulation stage of these searches and results on the success of searches on the topic of grieving and bereavement. Using linguistic style analyses we compare the content of grief-related queries showing differences in content that may be used to classify grief-related queries to help provide tailored support for different types of query.

[2] Looking for genre: the use of structural features during search tasks with Wikipedia Wikipedia and cultural heritage search behaviour session / Clark, Malcolm / Ruthven, Ian / Holt, Patrik O'Brian / Song, Dawei Proceedings of the 2012 Symposium on Information Interaction in Context 2012-08-21 p.145-154

ACM Digital Library Link

Summary: This paper reports on our task-based observational, logged, questionnaire study and analysis of ocular behavior pertaining to the interaction of structural features of text in Wikipedia using eye tracking. We set natural and realistic tasks searching Wikipedia online focusing on examining which features and strategies (skimming or scanning) were the most important for the participants to complete their tasks. Our research, carried out on a group of 30 participants, highlighted their interactions with the structural areas within Wikipedia articles, the visual cues and features perceived during the searching of the Wiki text. We collected questionnaire and ocular behavior (fixation metrics) data to highlight the ways in which people view the features in the articles. We found that our participants' extensively interacted with layout features, such as tables, titles, bullet lists, contents lists, information boxes, and references. The eye tracking results showed that participants used the format and layout features and they also highlighted them as important. They were able to navigate to useful information consistently, and they were an effective means of locating relevant information for the completion of their tasks with some success. This work presents results which contribute to the long-term goals of studying the features for genre and theoretical perception research.

[3] Graphical representation and similarity measurement of relevance judgments on the web Poster session / Balatsoukas, Panos / Ruthven, Ian Proceedings of the 2012 Symposium on Information Interaction in Context 2012-08-21 p.266-269

ACM Digital Library Link

Summary: The purpose of this paper is to present a method for the graphical representation and similarity measurement of relevance judgments on the web. In order to address this objective a Latent Semantic Indexing technique was used. The findings suggest that the proposed method could help researchers in information seeking and retrieval to make methodological decisions about their data, such as the selection of specific subsets of relevance judgments for further examination, the recording of dissimilarities between judgments, or, the identification of possible cognitive shifts and abnormalities in relevance judgment behavior during web searching.

[4] Bayesian latent variable models for collaborative item rating prediction Information filtering / Harvey, Morgan / Carman, Mark J. / Ruthven, Ian / Crestani, Fabio Proceedings of the 2011 ACM Conference on Information and Knowledge Management 2011-10-24 p.699-708

ACM Digital Library Link

Summary: Collaborative filtering systems based on ratings make it easier for users to find content of interest on the Web and as such they constitute an area of much research. In this paper we first present a Bayesian latent variable model for rating prediction that models ratings over each user's latent interests and also each item's latent topics. We describe a Gibbs sampling procedure that can be used to estimate its parameters and show by experiment that it is competitive with the gradient descent SVD methods commonly used in state-of-the-art systems. We then proceed to make an important and novel extension to this model, enhancing it with user-dependent and item-dependant biases to significantly improve rating estimation. We show by experiment on a large set of real ratings data that these models are able to outperform 3 common baselines, including a very competitive and modern SVD-based model. Furthermore we illustrate other advantages of our approach beyond simply its ability to provide more accurate ratings and show that it is able to perform better on the common and important case where the user profile is short.

[5] The fun semantic differential scales / Yusoff, Yusrita Mohd / Ruthven, Ian / Landoni, Monica Proceedings of ACM IDC'11: Interaction Design and Children 2011-06-20 p.221-224

ACM Digital Library Link

Summary: In this paper, we report on our experience developing an evaluation tool called the Fun Semantic Differential Scales (FSDS). The FSDS has been developed for use with and by very young children (3 to 5 years old) to express their feelings when interacting with computer products. We applied an iterative approach in designing and evaluating early versions before finalising the FSDS. A series of small studies have been conducted in one UK local nursery to investigate and understand how young children respond to all the FSDS versions.

[6] What Makes Re-finding Information Difficult? A Study of Email Re-finding Interactive IR / Elsweiler, David / Baillie, Mark / Ruthven, Ian Proceedings of ECIR'11, the 2011 European Conference on Information Retrieval 2011-04-18 p.568-579

Link to Digital Content at Springer

Summary: Re-finding information that has been seen or accessed before is a task which can be relatively straight-forward, but often it can be extremely challenging, time-consuming and frustrating. Little is known, however, about what makes one re-finding task harder or easier than another. We performed a user study to learn about the contextual factors that influence users' perception of task difficulty in the context of re-finding email messages. 21 participants were issued re-finding tasks to perform on their own personal collections. The participants' responses to questions about the tasks combined with demographic data and collection statistics for the experimental population provide a rich basis to investigate the variables that can influence the perception of difficulty. A logistic regression model was developed to examine the relationships between variables and determine whether any factors were associated with perceived task difficulty. The model reveals strong relationships between difficulty and the time lapsed since a message was read, remembering when the sought-after email was sent, remembering other recipients of the email, the experience of the user and the user's filing strategy. We discuss what these findings mean for the design of re-finding interfaces and future re-finding research.

[7] Ranking social bookmarks using topic models Poster session 2: IR track / Harvey, Morgan / Ruthven, Ian / Carman, Mark Proceedings of the 2010 ACM Conference on Information and Knowledge Management 2010-10-26 p.1401-1404

ACM Digital Library Link

Summary: Ranking of resources in social tagging systems is a difficult problem due to the inherent sparsity of the data and the vocabulary problems introduced by having a completely unrestricted lexicon. In this paper we propose to use hidden topic models as a principled way of reducing the dimensionality of this data to provide more accurate resource rankings with higher recall. We first describe Latent Dirichlet Allocation (LDA) and then show how it can be used to rank resources in a social bookmarking system. We test the LDA tagging model and compare it with 3 non-topic model baselines on a large data sample obtained from the Delicious social bookmarking site. Our evaluations show that our LDA-based method significantly outperforms all of the baselines.

[8] Relevance in Technicolor Search in Digital Libraries / Beresi, Ulises Cerviño / Kim, Yunhyong / Song, Dawei / Ruthven, Ian / Baillie, Mark ECDL 2010: Proceedings of the European Conference on Digital Libraries 2010-09-06 p.196-207

Link to Digital Content at Springer

Summary: In this article we propose the concept of relevance criteria profiles, which provide a global view of user behaviour in judging the relevance of retrieved information. We further propose a plotting technique which provides a session based overview of the relevance judgement processes interlaced with interactions that allow the researcher to visualise and quickly detect emerging patterns in both interactions and relevance criteria usage. We discuss by example, using data from a user study conducted between the months of January and August of 2008, how these tools support the better understanding of task based user valuation of documents that is likely to lead to recommendations for improving end-user services in digital libraries.

[9] A New Focus on End Users: Eye-Tracking Analysis for Digital Libraries Posters / Sykes, Jonathan / Dobreva, Milena / Birrell, Duncan / McCulloch, Emma / Ruthven, Ian / Ünal, Yurdagül / Feliciati, Pierluigi ECDL 2010: Proceedings of the European Conference on Digital Libraries 2010-09-06 p.510-513

Keywords: digital libraries; eye-tracking; gaze plots; heat maps; user studies

Link to Digital Content at Springer

Summary: Eye-tracking data was gathered as part of a user and functional evaluation of the Europeana v1.0 prototype, to determine which areas of the interface screen are most heavily used and which areas attract users' attention but are not effectively used in search. Outputs from eye-tracking data can offer insight into how advanced search functions can be made more intuitive for end users with differing interests and abilities, and can be used to inform continued interface development as digital libraries look to the future. Results led to recommendations for the future development of the Europeana digital library.

[10] The emotional impact of search tasks Tasks: emotions, multitasking and support / Poddar, Arti / Ruthven, Ian Proceedings of the 2010 Symposium on Information Interaction in Context 2010-08-18 p.35-44

ACM Digital Library Link

Summary: In this paper we consider the emotional impact of search tasks within Information Retrieval experiments. We study how search tasks of different types lead to different emotional responses by experimental participants and study the interaction between emotions and other subjective search variables. We show that some search tasks can lead to negative emotional responses whilst others are characterised by positive experiences. We discuss these findings with respect to how experiments are designed and conducted in Information Retrieval and how studying emotion within experimentation can lead to improved experimental design.

[11] First impressions: how search engine results contextualise digital identities Poster session / Ruthven, Ian / Clews, Caroline / Dali, Wajihah Haji Md Proceedings of the 2010 Symposium on Information Interaction in Context 2010-08-18 p.311-316

ACM Digital Library Link

Summary: In this paper, we explore how the results of a person search can provide a context for making judgments about other people. Exploring the new aggregated method of search results presentation, we describe two simulated studies to cast light on whether the results from an aggregated search can change opinions on another person. We show that information presentations can modify opinions on a person we do not know but it is far harder to change existing opinions.

[12] What eyes can tell about the use of relevance criteria during predictive relevance judgment? Poster session / Balatsoukas, Panos / Ruthven, Ian Proceedings of the 2010 Symposium on Information Interaction in Context 2010-08-18 p.389-394

ACM Digital Library Link

Summary: This paper reports on the preliminary findings of a user study that explored how searchers fixate on information associated with different relevance criteria during the process of predictive relevance judgment. In order to address this objective a user study was conducted that involved the completion of questionnaires, use of eye tracking technology, talk aloud protocols and post-search interviews. As opposed to previous studies, the present research asked participants to search for real information needs that represented different search contexts (e.g. from searches about personal interest to academic related searches). This permitted the identification of several relevance criteria that naturally occur across different search contexts and the emergence of some fixation patterns, not observed before, associated to the use of these criteria. The paper concludes with a discussion of the impact and implication of this study in the wider context of relevance judgment and information seeking in context research.

[13] Tripartite Hidden Topic Models for Personalised Tag Suggestion Personalization and Recommendation / Harvey, Morgan / Baillie, Mark / Ruthven, Ian / Carman, Mark J. Proceedings of ECIR'10, the 2010 European Conference on Information Retrieval 2010-03-28 p.432-443

Link to Digital Content at Springer

Summary: Social tagging systems provide methods for users to categorise resources using their own choice of keywords (or "tags") without being bound to a restrictive set of predefined terms. Such systems typically provide simple tag recommendations to increase the number of tags assigned to resources. In this paper we extend the latent Dirichlet allocation topic model to include user data and use the estimated probability distributions in order to provide personalised tag suggestions to users. We describe the resulting tripartite topic model in detail and show how it can be utilised to make personalised tag suggestions. Then, using data from a large-scale, real life tagging system, test our system against several baseline methods. Our experiments show a statistically significant increase in performance of our model over all key metrics, indicating that the model could be successfully used to provide further social tagging tools such as resource suggestion and collaborative filtering.

[14] Enabling Interactive Query Expansion through Eliciting the Potential Effect of Expansion Terms User Issues / Sahib, Nuzhah Gooda / Tombros, Anastasios / Ruthven, Ian Proceedings of ECIR'10, the 2010 European Conference on Information Retrieval 2010-03-28 p.532-543

Link to Digital Content at Springer

Summary: Despite its potential to improve search effectiveness, previous research has shown that the uptake of interactive query expansion (IQE) is limited. In this paper, we investigate one method of increasing the uptake of IQE by displaying summary overviews that allow searchers to view the impact of their expansion decisions in real time, engage more with suggested terms, and support them in making good expansion decisions. Results from our user studies show that searchers use system-generated suggested terms more frequently if they know the impact of doing so on their results. We also present evidence that the usefulness of our proposed IQE approach is highest when searchers attempt unfamiliar or difficult information seeking tasks. Overall, our work presents strong evidence that searchers are more likely to engage with suggested terms if they are supported by the search interface.

[15] Colouring the Dimensions of Relevance Posters / Beresi, Ulises Cerviño / Kim, Yunhyong / Baillie, Mark / Ruthven, Ian / Song, Dawei Proceedings of ECIR'10, the 2010 European Conference on Information Retrieval 2010-03-28 p.569-572

Link to Digital Content at Springer

Summary: In this article we introduce a visualisation technique for analysing relevance and interaction data. It allows the researcher to quickly detect emerging patterns in both interactions and relevance criteria usage. The concept of "relevance criteria profile", which provides a global view of user behaviour in judging the relevance of the retrieved information, is developed. We discuss by example, using data from a live search user study, how these tools support the data analysis.

[16] PuppyIR: Designing an Open Source Framework for Interactive Information Services for Children Posters/Demos / Azzopardi, Leif / Glassey, Richard / Lalmas, Mounia / Polajnar, Tamara / Ruthven, Ian Proceedings of the Workshop on Human-Computer Interaction and Information Retrieval 2009-10-23 p.26-29

Summary: One of the main aims of the PuppyIR project is to provide an open source framework for the development of Interactive Information Retrieval Services. The main focus of the project is directed towards developing such services for children, which introduces a number of novel and challenging issues to address (such as language development, security, moderation, etc).
In this poster paper, we outline the preliminary high-level design of the open source framework. The framework uses a layered architecture to minimize dependencies between the user-side concerns of interaction and presentation, and the system-side concerns of aggregating content from multiple sources and processing information appropriately. Each layer will consist of a series of interchangeable components, which can be interconnected to form a complete service. To facilitate the construction of diverse information services, a dataflow language is proposed to enable the assembly of the components in an intuitive and visual manner. One of the design goals of the architecture, and ultimate measures of success, is to provide a "lego" style building block environment in which researchers and developers of any age can build their own information service.
This poster provides the starting point for the design of the framework and aims to seek comments, feedback and suggestions from the community in order to improve and refine the architecture.

[17] The context of the interface Keynote presentations / Ruthven, Ian Proceedings of the 2008 Symposium on Information Interaction in Context 2008-10-14 p.3-5

ACM Digital Library Link

Summary: Our ideas on context (from both a soft and hard laboratory perspective) often manifest themselves at the interface. Interfaces for information seeking range from systems which offer little contextual information or flexibility of use to systems which are highly driven by the searcher's individual interaction. In this talk I will use existing search interfaces to highlight how the nature of information retrieval interfaces has changed in response to research from contextual IS&R. A particular theme will be the commercial nature of web search interfaces and how approaches such as persuasive and emotional design can be used to encourage interaction. I will also consider the nature of specialized versus general purpose search interfaces and the challenges raised in interface design. The (not very) hidden agenda behind this presentation will be to argue for an interaction-centered approach to IR systems.

[18] A Comparison of Named Entity Patterns from a User Analysis and a System Analysis Posters / Mohd, Masnizah / Crestani, Fabio / Ruthven, Ian Proceedings of ECIR'08, the 2008 European Conference on Information Retrieval 2008-03-30 p.679-683

Keywords: Named entity; Topic Detection and Tracking (TDT)

Link to Digital Content at Springer

Summary: This paper investigates the detection of named entity (NE) patterns by comparing the results of NE patterns resulting from a user analysis and a system analysis. Findings revealed that there are difference in NE patterns detected by system and user, something that may affect the performance of a TDT system based on NE detection.

[19] Exploring memory in email refinding / Elsweiler, David / Baillie, Mark / Ruthven, Ian ACM Transactions on Information Systems 2008 v.26 n.4 p.21

ACM Digital Library Link

Summary: Human memory plays an important role in personal information management (PIM). Several scholars have noted that people refind information based on what they remember and it has been shown that people adapt their management strategies to compensate for the limitations of memory. Nevertheless, little is known about what people tend to remember about their personal information and how they use their memories to refind. The aim of this article is to increase our understanding of the role that memory plays in the process of refinding personal information. Concentrating on email re-finding, we report on a user study that investigates what attributes of email messages participants remember when trying to refind. We look at how the attributes change in different scenarios and examine the factors which impact on what is remembered.

[20] Towards task-based personal information management evaluations Personalization / Elsweiler, David / Ruthven, Ian Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2007-07-23 p.23-30

ACM Digital Library Link

Summary: Personal Information Management (PIM) is a rapidly growing area of research concerned with how people store, manage and refind information. A feature of PIM research is that many systems have been designed to assist users manage and refind information, but very few have been evaluated. This has been noted by several scholars and explained by the difficulties involved in performing PIM evaluations. The difficulties include that people re-find information from within unique personal collections; researchers know little about the tasks that cause people to re-find information; and numerous privacy issues concerning personal information. In this paper we aim to facilitate PIM evaluations by addressing each of these difficulties. In the first part, we present a diary study of information re-finding tasks. The study examines the kind of tasks that require users to refind information and produces a taxonomy of refinding tasks for email messages and web pages. In the second part, we propose a task-based evaluation methodology based on our findings and examine the feasibility of the approach using two different methods of task creation.

[21] Intra-assessor consistency in question answering Posters / Ruthven, Ian / Glasgow, Leif Azzopardi / Baillie, Mark / Bierig, Ralf / Nicol, Emma / Sweeney, Simon / Yakici, Murat Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2007-07-23 p.727-728

ACM Digital Library Link

Summary: In this paper we investigate the consistency of answer assessment in a complex question answering task examining features of assessor consistency, types of answers and question type.

[22] Modelling epistemic uncertainty in ir evaluation Posters / Yakici, Murat / Baillie, Mark / Ruthven, Ian / Crestani, Fabio Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2007-07-23 p.769-770

ACM Digital Library Link

Summary: Modern information retrieval (IR) test collections violate the completeness assumption of the Cranfield paradigm. In order to maximise the available resources, only a sample of documents (i.e. the pool) are judged for relevance by a human assessor(s). The subsequent evaluation protocol does not make any distinctions between assessed or unassessed documents, as documents that are not in the pool are assumed to be not relevant for the topic. This is beneficial from a practical point of view, as the relative performance can be compared with confidence if the experimental conditions are fair for all systems. However, given the incompleteness of relevance assessments, two forms of uncertainty emerge during evaluation. The first is Aleatory uncertainty, which refers to variation in system performance across the topic set, which is often addressed through the use of statistical significance tests. The second form of uncertainty is Epistemic, which refers to the amount of knowledge (or ignorance) we have about the estimate of a system's performance. Epistemic uncertainty is a consequence of incompleteness and is not addressed by the current evaluation protocol. In this study, we present a first attempt at modelling both aleatory and epistemic uncertainty associated with IR evaluation. We aim to account for both the variability associated with system performance and the amount of knowledge known about the performance estimate.

[23] A Retrieval Evaluation Methodology for Incomplete Relevance Assessments Evaluation / Baillie, Mark / Azzopardi, Leif / Ruthven, Ian Proceedings of ECIR'07, the 2007 European Conference on Information Retrieval 2007-04-02 p.271-282

Link to Digital Content at Springer

Summary: In this paper we a propose an extended methodology for laboratory based Information Retrieval evaluation under incomplete relevance assessments. This new protocol aims to identify potential uncertainty during system comparison that may result from incompleteness. We demonstrate how this methodology can lead towards a finer grained analysis of systems. This is advantageous, because the detection of uncertainty during the evaluation process can guide and direct researchers when evaluating new systems over existing and future test collections.

[24] Examining assessor attributes at HARD 2005 Posters / Baillie, Mark / Ruthven, Ian Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval 2006-08-06 p.609-610

Keywords: clarification forms, query expansion

ACM Digital Library Link

[25] The Effects on Topic Familiarity on Online Search Behaviour and Use of Relevance Criteria Posters / Wen, Lei / Ruthven, Ian / Borlund, Pia Proceedings of ECIR'06, the 2006 European Conference on Information Retrieval 2006-04-10 p.456-459

Link to Digital Content at Springer

Summary: This paper presents an experimental study on the effect of topic familiarity on the assessment behaviour of online searchers. In particular we investigate the effect of topic familiarity on the resources and relevance criteria used by searchers. Our results indicate that searching on an unfamiliar topic leads to use of more generic and fewer specialised resources and that searchers employ different relevance criteria when searching on less familiar topics.