Zhengyan ShiHi, weclome to my personal page. I am a PhD student supervised by Prof Aldo Lipani and Prof Emine Yilmaz at University College London, affiliated with Web Intelligence Group and SpaceTimeLab. Previously, I worked as an Intern of Technical Staff at Cohere in London and completed two internships as an Applied Scientist at Amazon in both London and Seattle offices. Prior to pursuing my PhD, I obtained a Master's degree in Data Science (Statistics) with Distinction from University College London and a Bachelor's degree in Mathematics with First Class Honor from University of Liverpool and Xi'an Jiaotong-Liverpool University. Central to my research is the ambition to leverage language models efficiently and robustly to solve general tasks. To that end, my existing work can be broadly categorized into the following directions:
Google Scholar / Twitter / Github / LinkedIn / Email |
|
Preprint, 2024
In this work, we identify a critical issue of likelihood over-optimisation in state-of-the-art Direct Alignment Algorithms (DAAs) and explore the relationship between completion likelihood and model performance.
Advances in Neural Information Processing Systems (NeurIPS), 2024
Paper / Github / Community Discussion
We show that in certain scenarios, applying loss to instructions rather than outputs only, which we refer to as Instruction Modelling, could largely improve the performance of instruction tuning on both various NLP and open-ended generation benchmarks. Remarkably, in the most advantageous case, our approach boosts model performance on AlpacaEval 1.0 by over 100%.
International Conference on Learning Representations (ICLR), 2024
Paper / Github / Trending in Community
Improves efficiency of Prompt Tuning in both time and memory by over 20% (when T5-Base is used as the backbone), with better performance. DePT grows more efficient as the model size increases.
Advances in Neural Information Processing Systems (NeurIPS), 2023
Paper / Github / Trending in Community
Combines the idea of the instruction tuning and language modelling. Represents the first work to perform instruction tuning via unsupervised objectives. Boosts prompt-based fine-tuning performance by over 20% in absolute.
Association for Computational Linguistics (Findings of ACL), 2023
Shows Task-adaptive Pre-training (TAPT) as a simple yet effective method for semi-supervised learning (often SoTA performance). Highlights the effectiveness of TAPT even with only a few hundred unlabelled samples (in contrary to the common belief that continued pre-training requires a large amount of unlabelled data).
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022
Paper / Github / HuggingFace Dataset
Introduces StepGame, a new benchmark for testing multi-hop spatial reasoning in texts. This dataset challenges models to perform robust spatial reasoning across multiple steps, providing a valuable tool for advancing natural language understanding in complex spatial scenarios.
Program Committee: NeurIPS (2023, 2024), ICML (2024), ICLR (2025), AAAI (2023, 2024), COLM (2024), ACL ARR (Feb. 2023 - Jan. 2024), ACL (2023), EMNLP (2022, 2023), EACL (2023), COLING (2023, 2024), ECML/PKDD (2022), KDD (2023), SIGIR (2022, 2023, 2024), ECIR (2024), SDM (2024)