How to Test ChatGPT’s Performance in Semantic Understanding and Common-Sense Reasoning: Challenges and Opportunities of Linguistics in the Era of Large Language Models

Yuan Yulin

Chinese Journal of Language Policy and Planning ›› 2024, Vol. 9 ›› Issue (1) : 49-63.

Welcome to visit Chinese Journal of Language Policy and Planning!
Chinese Journal of Language Policy and Planning ›› 2024, Vol. 9 ›› Issue (1) : 49-63. DOI: 10.19689/j.cnki.cn10-1361/h.20240105

How to Test ChatGPT’s Performance in Semantic Understanding and Common-Sense Reasoning: Challenges and Opportunities of Linguistics in the Era of Large Language Models

  • Yuan Yulin
Author information +
History +

Abstract

This paper examines the approaches to test the performance of ChatGPT in semantic understanding and common-sense reasoning. It first reviews the remarkable performance of ChatGPT and other large language models (LLM) in semantic understanding and common-sense reasoning, and explains how they use distributional semantics and token vector representations to process language. Then, it critiques the classical “Turing Test” and introduces some alternative new tests such as Winograd Schema Challenge (WSC) and its upgraded version WinoGrande datasets, which are more relevant to linguistic studies. Next, it presents the results of applying ChatGPT to Winograd Schema sentences, showing that LLM have reached human-level performance in semantic understanding and common-sense reasoning. Finally, it argues that these language models can “understand” human natural languages, and the development of tests like the Winograd Schema is an opportunity for linguists to expand their academic field in the era of artificial intelligence.

Key words

ChatGPT / large language models / semantic understanding and common-sense reasoning / (New) Turing Test / Winograd Schema challenge / WinoGrande dataset

Cite this article

Download Citations
Yuan Yulin. How to Test ChatGPT’s Performance in Semantic Understanding and Common-Sense Reasoning: Challenges and Opportunities of Linguistics in the Era of Large Language Models[J]. Chinese Journal of Language Policy and Planning. 2024, 9(1): 49-63 https://doi.org/10.19689/j.cnki.cn10-1361/h.20240105

Accesses

Citation

Detail

Sections
Recommended

/