My paper (collaborated with Prof. Sophie Hao) “ERAS: Evaluating the Robustness of Chinese NLP Models to Morphological Garden Path Errors ” has been accpeted as oral presentation to NAACL25.
Brief Introduction: Our paper evaluate behaviors of language models(LMs) in handling local ambiguity or fake morphological garden paths in Chinese that can be parsed two ways while only one way is grametically correct. (eg. 领导有意见他。 can potentially be parsed as 领导 有 意见 他。 and 领导 有意 见 他。 and the first one is grametically wrong.) We found that LMs perform bad in tasks, like sentiment analysis, required implicitly parsing this kind of ambiguity while co-training with word segmentation will largely mitigate LMs’ errors.
Abstract: In languages without orthographic word boundaries, NLP models perform word segmentation, either as an explicit preprocessing step or as an implicit step in an end-to-end computation. This paper shows that Chinese NLP models are vulnerable to morphological garden path errors: errors caused by a failure to resolve local word segmentation ambiguities using sentence-level morphosyntactic context. We propose a benchmark, ERAS, that tests a model’s vulnerability to morphological garden path errors by comparing its behavior on sentences with and without local segmentation ambiguities. Using ERAS, we show that word segmentation models make garden path errors on locally ambiguous sentences, but do not make equivalent errors on unambiguous sentences. We further show that sentiment analysis models with character-level tokenization make implicit garden path errors, even without an explicit word segmentation step in the pipeline. Our results indicate that models’ segmentation of Chinese text often fails to account for morphosyntactic context.