Skip to main content

Ke-Han Lu


Google Scholar GitHub X

I am Ke-Han Lu, a second-year Ph.D. student at National Taiwan University, working under the supervision of Prof. Hung-Yi Lee. My research interests lie in the field of Multimodal Language Models, with a particular focus on cross-modal alignment and leveraging powerful language models to enhance multi-modal systems.

Publications​

  1. Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks
    ICLR 2025, Paper
    Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Xuanjun Chen, Chi-Yuan Hsiao, Puyuan Peng, Shih-Heng Wang, Chun-Yi Kuan, Ke-Han Lu, Kai-Wei Chang, Chih-Kai Yang, Fabian Ritter-Gutierrez, Ming To Chuang, Kuan-Po Huang, Siddhant Arora, You-Kuan Lin, Eunjung Yeo, Kalvin Chang, Chung-Ming Chien, Kwanghee Choi, Cheng-Hsiu Hsieh, Yi-Cheng Lin, Chee-En Yu, I Chiu, Heitor R Guimarães, Jionghao Han, Tzu-Quan Lin, Tzu-Yuan Lin, Homu Chang, Ting-Wu Chang, Chun Wei Chen, Shou-Jen Chen, Yu-Hua Chen, Hsi-Chun Cheng, Kunal Dhawan, Jia-Lin Fang, Shi-Xin Fang, Kuan-Yu Fang Chiang, Chi An Fu, Hsien-Fu Hsiao, Ching Yu Hsu, Shao-Syuan Huang, Lee Chen Wei, Hsi-Che Lin, Hsuan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Lin, Ting-Chun Liu, Li-Chun Lu, Tsung-Min Pai, Ankita Pasad, Shih-Yun Shan Kuan, Suwon Shon, Yuxun Tang, Yun-Shao Tsai, Jui-Chiang Wei, Tzu-Chieh Wei, Chengxi Wu, Dien-Ruei Wu, Chao-Han Huck Yang, Chieh-Chi Yang, Jia Qi Yip, Shao-Xiang Yuan, Vahid Noroozi, Zhehuai Chen, Haibin Wu, Karen Livescu, David Harwath, Shinji Watanabe, Hung-yi Lee, 
  2. Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
    ICASSP 2025, Paper, GitHub
    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, Hung-yi Lee, 
  3. SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioning
    ICASSP 2025, Paper
    Chien-yu Huang, Min-Han Shih, Ke-Han Lu, Chi-Yuan Hsiao, Hung-yi Lee, 
  4. Building a taiwanese mandarin spoken language model: A first attempt
    ArXiv preprint, Paper
    Chih-Kai Yang, Yu-Kuan Fu, Chen-An Li, Yi-Cheng Lin, Yu-Xiang Lin, Wei-Chih Chen, Ho Lam Chung, Chun-Yi Kuan, Wei-Ping Huang, Ke-Han Lu*, Tzu-Quan Lin, Hsiu-Hsuan Wang, En-Pei Hu, Chan-Jan Hsu, Liang-Hsuan Tseng, I Chiu, Ulin Sanga, Xuanjun Chen, Po-chun Hsu, Shu-wen Yang, Hung-yi Lee, 
  5. Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation
    IEEE SLT 2024, Paper
    Chun-Yi Kuan, Chih-Kai Yang, Wei-Ping Huang, Ke-Han Lu, Hung-yi Lee, 
  6. DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
    InterSpeech 2024, Paper
    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, He Huang, Boris Ginsburg, Yu-Chiang Frank Wang, Hung-yi Lee, 
  7. HypR: A comprehensive study for ASR hypothesis revising with a reference corpus
    InterSpeech 2024, Paper, GitHub
    Yi-Wei Wang, Ke-Han Lu, Kuan-Yu Chen, 
  8. Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
    ICASSP 2024, Paper, GitHub
    Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, Hung-yi Lee, 
  9. Investigating zero-shot generalizability on mandarin-english code-switched asr and speech-to-text translation of recent foundation models with self-supervision and weak supervision
    ICASSP workshop 2024, Paper
    Chih-Kai Yang, Kuan-Po Huang, Ke-Han Lu, Chun-Yi Kuan, Chi-Yuan Hsiao, Hung-yi Lee, 
  10. A Context-aware Knowledge Transferring Strategy for CTC-based ASR
    IEEE SLT 2022, Paper, GitHub
    Ke-Han Lu, Kuan-Yu Chen, 
  11. Non-autoregressive ASR Modeling using Pre-trained Language Models for Chinese Speech Recognition
    IEEE/ACM Transactions on Audio, Speech, and Language Processing, Paper
    Fu-Hao Yu, Kuan-Yu Chen, Ke-Han Lu, 
  12. A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021
    Poster spotlight, VQA workshop, CVPR 2021, Paper, Video, LeaderBoard
    Ke-Han Lu, Bo-Han Fang, Kuan-Yu Chen, 
  13. ntust-nlp-2 at ROCLING-2021 Shared Task: BERT-based semantic analyzer with word-level information
    ROCLING 2021: Conference on Computational Linguistics and Speech Processing, Paper
    Ke-Han Lu, Kuan-Yu Chen, 
  14. A Preliminary Study of Formosa Speech Recognition Challenge 2020 – Taiwanese ASR
    International Journal of Computational Linguistics and Chinese Language Processing, Paper
    Fu-Hao Yu, Ke-Han Lu, Yi-Wei Wang, Wei-Zhe Chang, Wei-Kai Huang, Kuan-Yu Chen, 

Education​

  • National Taiwan University
    • Ph.D. in Communication Engineering
      • Feb 2024 - Present
  • National Taiwan University of Science and Technology
    • M.S. in Computer Science and Information Engineering
      • Sep 2020 - Feb 2023
  • National Taiwan University of Science and Technology
    • B.S. in Computer Science and Information Engineering
      • Sep 2016 - Jun 2020

Award​

  • NSTC Graduate Research Fellowship（NSTC-GRF）
  • 16th TaiwanTech Outstanding Youth Award

Skills​

  • Programming: Python, PyTorch, Javascript, Latex
  • Software and tools: Linux, Docker, Git, NeMo, Megatron-LM, ESPNET, Huggingface Transformers, fairseq
  • Language: Mandarin(native), English(fluent)