Sensor-enhanced LLM for Smart Health Systems

The widespread adoption of Large Language Models (LLMs) has fundamentally transformed modern AI development. However, current deployments face critical challenges in bridging the gap between digital intelligence and physical environments, particularly in processing heterogeneous multi-modal sensory data, operating within the computational constraints of edge devices, and safeguarding user privacy.

In this talk, I will introduce SensorLLM, a novel framework that seamlessly integrates real-world sensing capabilities with LLMs for smart health applications. SensorLLM features three core innovations: (1) an edge-cloud collaborative architecture that enables open-class classification on resource-constrained devices, (2) cost-efficient encoding schemes ensuring secure and privacy-preserving Transformer inference across distributed systems, and (3) an intelligent orchestration system that coordinates multi-modal sensors to address complex user queries.

I will demonstrate SensorLLM’s transformative impact through three smart health applications. DrHouse functions as an LLM-based virtual consultation system that synthesizes wearable sensor data with medical expertise to deliver personalized clinical recommendations. Nuna, an LLM-powered smart necklace, enables intuitive emotional tracking and continuous health monitoring for everyday users. KoalaFM represents the first LLM-enabled platform for early diagnosis, personalized intervention, and complex cross-disease analysis of aging-related degenerative conditions, currently undergoing validation through a comprehensive five-year clinical trial with 1,000 participants.