Chaoyi Jiang

I am a PhD candidate at the Ming Hsieh Department of Electrical and Computer Engineering at the University of Southern California, advised by Prof. Murali Annavaram. I received my bachelor’s degree in Electronic Engineering from Tsinghua University and my master’s degree in Software Engineering from Carnegie Mellon University. After completing my master’s, I worked at Microsoft as a software engineer on the Bing Ads online ranking platform team.

My research centers on advancing the efficiency and scalability of modern machine learning systems, with a particular focus on efficient training and inference of large language models (LLMs) and large-scale recommendation systems.

News

• 04/30/2025: Our paper DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing has been accepted to ICML 2026.
• 04/07/2025: Our paper DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning has been accepted to ACL 2026.
• 04/02/2025: Our paper [HuffmanEmbed: Using Huffman Coding for Embedding Table Compression in Deep Learning Recommendation Models] has been accepted to SIGIR 2026.
• 02/28/2025: Our paper Fast NF4 Dequantization Kernels for Large Language Model Inference has been accepted to EMC2 Workshop 2026.
• 07/07/2025: Our paper DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding has been accepted to CoLM 2025.
• 07/07/2025: Our paper LEAF: Lightweight, Efficient, Adaptive and Flexible Embedding for Large-Scale Recommendation Models has been accepted to RecSys 2025.
• 05/15/2025: Our paper KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation has been accepted to ACL Findings 2025.
• 12/10/2024: Our paper Efficient LLM Inference with I/O-Aware Partial KV Cache Recomputation has been accepted to the AAAI SEAS Workshop 2025.
• 09/20/2024: Our paper CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Data has been accepted to RecSys Workshop 2024.

Teaching Assistant

• EE599: Systems for Machine Learning (Fall 2023)
• EE109: Introduction to Embedded Systems (Spring 2024)
• EE557: Computer Systems Architecture (Fall 2024, Fall 2025, Spring 2026)
• EE508: Hardware Foundations for Machine Learning (Spring 2025)

Volunteer Service

• Mentorship: USC CURVE (Summer 2025, Fall 2025, Spring 2026)