Publications

Paper

2026

1. 

   What We Talk About When We Talk About Frameworks in HCI

   Shitao Fang, Koji Yatani, and Kasper Hornbæk

   In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems 2026 (CHI’26)

   Abs URL PDF

   In HCI, frameworks function as a type of theoretical contribution, often supporting ideation, design, and evaluation. Yet, little is known about how they are actually used, what functions they serve, and which scholarly practices that shape them. To address this gap, we conducted a systematic review of 615 papers from a decade of CHI proceedings (2015–2024) that prominently featured the term framework. We classified these papers into six engagement types. We then examined the role, form, and essential components of newly proposed frameworks through a functional typology, analyzing how they are constructed, validated, and articulated for reuse. Our results show that enthusiasm for proposing new frameworks exceeds the willingness to iterate on existing ones. They also highlight the ambiguity in the function of frameworks and the scarcity of systematic validation. Based on these insights, we call for more rigorous, reflective, and cumulative practices in the development and use of frameworks in HCI.

2. 

   Unraveling multiparty conversations: From human interaction mechanisms to conversational agent challenges and persona design

   Shitao Fang, Xingyu Liu, Takeo Igarashi, and Koji Yatani

   In International Journal of Human-Computer Studies 2026 (IJHCS)

   Abs URL PDF

   Multiparty conversations are ubiquitous and indispensable in diverse social and collaborative contexts. However, current conversational agents (CAs) face significant challenges in effectively engaging in such interactions, particularly within text-based environments. While earlier limitations were often attributed to the inadequacies of AI models, recent advances in large language models now compel us to revisit both our understanding of multiparty conversation and the way we design CAs. This paper synthesizes findings from two complementary qualitative investigations and proposes a conceptual model for designing CAs that can genuinely participate, rather than merely function as tools or outsiders. The first study, employing retrospective think-aloud sessions (N=30) with users in text-based multiparty settings, uncovers 5 key interactional mechanisms (e.g., Turn-taking Management, Presence Management) that underpin successful human-human multiparty interactions, derived from participants’ articulated perceptions and reasoning. Subsequently, the second study, through semi-structured interviews (N=15), identifies user expectations for CA integration and key traits (e.g., proactivity, social authenticity) that shape an ideal CA persona perceived by users as a genuine participant. Drawing from these human-centric insights, we then derive design considerations, aiming to guide the development of CAs capable of more natural, effective, and socially intelligent participation in multiparty conversation.

2025

1. 

   Proactive Conversational Agents with Inner Thoughts

   Xingyu Bruce Liu,  Shitao Fang, Weiyan Shi, Chien-Sheng Wu, Takeo Igarashi, and Xiang ’Anthony’ Chen

   In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems 2025 (CHI’25)

   Abs URL Code PDF

   One of the long-standing aspirations in conversational AI is to allow them to autonomously take initiatives in conversations, i.e., being proactive. This is especially challenging for multi-party conversations. Prior NLP research focused mainly on predicting the next speaker from contexts like preceding conversations. In this paper, we demonstrate the limitations of such methods and rethink what it means for AI to be proactive in multi-party, human-AI conversations. We propose that just like humans, rather than merely reacting to turn-taking cues, a proactive AI formulates its own inner thoughts during a conversation, and seeks the right moment to contribute. Through a formative study with 24 participants and inspiration from linguistics and cognitive psychology, we introduce the Inner Thoughts framework. Our framework equips AI with a continuous, covert train of thoughts in parallel to the overt communication process, which enables it to proactively engage by modeling its intrinsic motivation to express these thoughts. We instantiated this framework into two real-time systems: an AI playground web app and a chatbot. Through a technical evaluation and user studies with human participants, our framework significantly surpasses existing baselines on aspects like anthropomorphism, coherence, intelligence, and turn-taking appropriateness.

2. 

   Multimodal Silent Speech-based Text Entry with Word-initials Conditioned LLM

   Zixiong Su,  Shitao Fang, and Jun Rekimoto

   In Proceedings of the 7th ACM Conference on Conversational User Interfaces 2025 (CUI’25)

   Abs URL PDF

   Although exhibiting great potential in enabling seamless communication between humans and conversational agents, large vocabulary recognition is still challenging for silent speech interfaces. In this research, we propose a novel interaction technique that combines silent speech and typing to enable more efficient text entry while preserving privacy. This technique allows users to use abbreviated phrase input while still ensuring high accuracy by leveraging visual information. By fine-tuning a large language model with a visual speech encoder, we condition the models to decode the speech content with word initials as hints. Evaluations on existing datasets show that our model can reduce the Word Error Rate from 20.3% to 9.19%, compared to state-of-the-art visual speech recognition models. Results from a user study demonstrated significant improvements in input speed and keystroke saving. Participants reported that our prototype, LipType, leads to an overall lower perceived workload, particularly in the effort and physical demand dimension.

2024

1. 

   Examining Human Perception of Generative Content Replacement in Image Privacy Protection

   Anran Xu,  Shitao Fang, Huan Yang, Simo Hosio, and Koji Yatani

   In Proceedings of the CHI Conference on Human Factors in Computing Systems 2024 (CHI’24)

   Abs URL Code PDF

   The richness of the information in photos can often threaten privacy, thus image editing methods are often employed for privacy protection. Existing image privacy protection techniques, like blurring, often struggle to maintain the balance between robust privacy protection and preserving image usability. To address this, we introduce a generative content replacement (GCR) method in image privacy protection, which seamlessly substitutes privacy-threatening contents with similar and realistic substitutes, using state-of-the-art generative techniques. Compared with four prevalent image protection methods, GCR consistently exhibited low detectability, making the detection of edits remarkably challenging. GCR also performed reasonably well in hindering the identification of specific content and managed to sustain the image’s narrative and visual harmony. This research serves as a pilot study and encourages further innovation on GCR and the development of tools that enable human-in-the-loop image privacy protection using approaches similar to GCR.

2023

1. 

   LipLearner: Customizable Silent Speech Interactions on Mobile Devices

   Zixiong Su,  Shitao Fang, and Jun Rekimoto

   In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems 2023 (CHI’23) 🏆 Best Paper Award

   Abs URL Code PDF

   Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.

Demo & Poster & Workshop

2023

1. 

   Towards Understanding Sense of Inclusion in Social VR Onboarding

   Shitao Fang, and Koji Yatani

   In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems 2023 (CHI ’23 LBW)

   Abs URL PDF

   Being included in social interactions is a fundamental human need in both physical and virtual worlds. However, it is overlooked in the context of social VR user experience. Based on social psychology, we define the sense of inclusion as the degree to which an individual perceives a sense of belonging and authenticity from a group. We initially use non-verbal behavior, which is commonly used in social VR, as an entry point to understanding the role of the sense of inclusion in social VR. We examine how the reactive behaviors of existing community members would influence the sense of inclusion during social VR onboarding. Our between-subject experiment (N=39) with three reactive behavioral conditions confirms that positive responses from existing community members increased the sense of inclusion. And the sense of inclusion positively mediates several user experiences including enjoyment and immersion. We highlight potential design implications and future research for social VR.

2022

1. 

   Customizing Silent Speech Commands from Voice Input Using One-Shot Lipreading

   Zixiong Su,  Shitao Fang, and Jun Rekimoto

   In The Adjunct Publication of the 35th Annual ACM Symposium on User Interface Software and Technology 2022 (UIST’22)

   Abs URL

   We present LipLearner, a lipreading-based silent speech interface that enables in-situ command customization on mobile devices. By leveraging contrastive learning to learn efficient representations from existing datasets, it performs instant fine-tuning for unseen users and words using one-shot learning. To further minimize the labor of command registration, we incorporate speech recognition to automatically learn new commands from voice input. Conventional lipreading systems provide limited pre-defined commands due to the time cost and user burden of data collection. In contrast, our technique provides expressive silent speech interaction with minimal data requirements. We conducted a pilot experiment to investigate the real-time performance of LipLearner, and the result demonstrates that an average accuracy of is achievable with only one training sample for each command.

2021

1. 

   Towards Understanding the Social and Societal Implications of Avatar Designs in Social VR

   Nami Ogawa,  Shitao Fang, and Koji Yatani

   In CHI Social VR Workshop 2021

   Abs URL

   Existing research confirms that the appearance and affordance of avatars can influence the users’ perception, attitudes, and behavior. However, such studies focus on perception and behavioral changes of the user who directly controls the given avatar. As a result, the social and societal implications of avatar designs are still underexplored. We argue that an emerging platform of social VR would enable further explorations on avatar’s effects in the context of interaction with other users, potentially opening up a new research horizon. In this paper, we describe our research direction and discuss potential research questions.