Ke-Han Lu

I’m Ke-Han Lu, a second-year Ph.D. student at National Taiwan University, advised by Prof. Hung-Yi Lee. My research focuses on multimodal language models, particularly on cross-modal alignment and utilizing large language models to enhance multimodal understanding.
- Core contributor to Dynamic-SUPERB (Phase 1 & Phase 2)
- Lead author of the DeSTA series: DeSTA2.5-Audio, DeSTA-2, DeSTA-1, Speech-IFEval
Selected Publications
For the full publication list, please refer to my google scholar.
DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data
Building a taiwanese mandarin spoken language model: A first attempt
Chih-Kai Yang, Yu-Kuan Fu, Chen-An Li, Yi-Cheng Lin, Yu-Xiang Lin, Wei-Chih Chen, Ho Lam Chung, Chun-Yi Kuan, Wei-Ping Huang, Ke-Han Lu*, Tzu-Quan Lin, Hsiu-Hsuan Wang, En-Pei Hu, Chan-Jan Hsu, Liang-Hsuan Tseng, I Chiu, Ulin Sanga, Xuanjun Chen, Po-chun Hsu, Shu-wen Yang, Hung-yi Lee,
(Co-first author) Technical Report, ArXiv preprint., Paper, BibTeX
(Co-first author) Technical Report, ArXiv preprint., Paper, BibTeX
DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
A Context-aware Knowledge Transferring Strategy for CTC-based ASR
Non-autoregressive ASR Modeling using Pre-trained Language Models for Chinese Speech Recognition
A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021
Ke-Han Lu, Bo-Han Fang, Kuan-Yu Chen,
Poster spotlight, VQA workshop, CVPR 2021, Paper, Video, LeaderBoard, BibTeX
Poster spotlight, VQA workshop, CVPR 2021, Paper, Video, LeaderBoard, BibTeX
Education
- National Taiwan University
- Ph.D. in Communication Engineering
- Feb 2024 - Present
- Ph.D. in Communication Engineering
- National Taiwan University of Science and Technology
- M.S. in Computer Science and Information Engineering
- Sep 2020 - Feb 2023
- M.S. in Computer Science and Information Engineering
- National Taiwan University of Science and Technology
- B.S. in Computer Science and Information Engineering
- Sep 2016 - Jun 2020
- B.S. in Computer Science and Information Engineering
Honors
- NVIDIA Academic Grant Program
- NSTC Graduate Research Fellowship(NSTC-GRF)
- 16th TaiwanTech Outstanding Youth Award
Skills
- Programming: Python, PyTorch, Javascript, Latex
- Software and tools: Linux, Docker, Git, NeMo, Megatron-LM, ESPNET, Huggingface Transformers, fairseq
- Language: Mandarin(native), English(fluent)