|
|
How to Test ChatGPT’s Performance in Semantic Understanding and Common-Sense Reasoning: Challenges and Opportunities of Linguistics in the Era of Large Language Models |
Yuan Yulin |
|
|
Abstract This paper examines the approaches to test the performance of ChatGPT in semantic understanding and common-sense reasoning. It first reviews the remarkable performance of ChatGPT and other large language models (LLM) in semantic understanding and common-sense reasoning, and explains how they use distributional semantics and token vector representations to process language. Then, it critiques the classical “Turing Test” and introduces some alternative new tests such as Winograd Schema Challenge (WSC) and its upgraded version WinoGrande datasets, which are more relevant to linguistic studies. Next, it presents the results of applying ChatGPT to Winograd Schema sentences, showing that LLM have reached human-level performance in semantic understanding and common-sense reasoning. Finally, it argues that these language models can “understand” human natural languages, and the development of tests like the Winograd Schema is an opportunity for linguists to expand their academic field in the era of artificial intelligence.
|
|
|
|
|
|
|
|