语言学知识驱动的空间语义理解能力评测数据集研究

詹卫东,孙春晖,肖力铭

语言战略研究 ›› 2024, Vol. 9 ›› Issue (5) : 7-21.

欢迎访问《语言战略研究》官方网站!今天是
语言战略研究 ›› 2024, Vol. 9 ›› Issue (5) : 7-21. DOI: 10.19689/j.cnki.cn10-1361/h.20240501
语言战略研究

语言学知识驱动的空间语义理解能力评测数据集研究

  • 詹卫东,孙春晖,肖力铭
作者信息 +

SpaCE: A Linguistic Knowledge-Driven Benchmark for Spatial Cognition Evaluation

  • Zhan Weidong, Sun Chunhui and Xiao Liming
Author information +
文章历史 +

摘要

近20年来,深度学习技术显著提升了机器的自然语言处理能力,使之在诸多任务上接近甚至超过人类水平。机器学习的对象不再是直接来自人类语言学研究成果(知识),而是人类语言材料(数据)。在靠数据和算力驱动的大语言模型几近建成巴别塔的当下,语言学家通过深挖语言现象总结的语言学知识价值何在?本文提出从知识到数据的研究思路,设计了空间语义理解的6项任务:空间信息正误判别、异常空间信息识别、缺失参照成分补回、空间语义角色标注、空间表达异形同义判别、空间方位关系推理,以构建中文空间语义理解能力评测数据集为例,介绍从SpaCE2021到SpaCE2024数据集的设计思想、数据集制作概况以及机器在空间语义理解任务上的表现。总的来看,参加SpaCE赛事的大语言模型,在依赖表面分布特征(形式线索)的任务上容易获得好成绩,在依赖深层语义理解(认知能力)的任务上容易表现不好。因此,在人工智能高速发展使得语言学知识在计算机信息处理领域被动边缘化的当下,语言学知识的价值需要拓展,即用于指导小而精的高品质语言数据,以提升机器学习的效果和效率。为了计算应用的目的,语法研究应该在观察充分、描写充分、解释充分之上,追求更具挑战性的目标——生成充分。

Abstract

Over the past two decades, deep learning technology has propelled machine natural language processing capabilities to rival or even surpass human levels in many tasks. Machine learning does not directly utilize the outcomes of human linguistic research (knowledge), but rather from human language materials (data). This situation should garner significant attention from linguists. As large language models, driven purely by data and computational power, have nearly constructed a modern Tower of Babel, the question of how to realize the value of linguistic knowledge through in-depth exploration of specific and subtle language phenomena looms large over every linguistic researcher. This paper proposes a research approach that generates text data from linguistic knowledge for evaluating machine understanding of spatial semantics. Over the past four years, we have organized four consecutive competitions on Chinese Spatial Cognition Evaluation (SpaCE): from SpaCE2021 to SpaCE2024, including 6 sub-tasks: Determination of Spatial Information Validity, Detection of Spatial Anomalies, Recovery of Spatial References, Identification of Spatial Semantic Roles, Recognition of Spatial Equivalences, and Spatial Position Reasoning. This paper introduces the design philosophy, dataset creation process, dataset overview, and the performance characteristics of machines in SpaCE tasks. Overall, large language models participating in the SpaCE competitions perform relatively well on tasks that rely on surface distribution features, that is, tasks with formal cues, but poorly on tasks that depend on deep semantic understanding, that is, tasks requiring cognitive abilities. In the current era of rapid AI development, where linguistic knowledge is passively marginalized in the field of natural language processing, the value of linguistic knowledge needs to be redefined. It should be used to guide the production of small, high-quality language data to enhance the effectiveness and efficiency of machine learning. For computational applications, grammatical research should pursue more challenging goals—adequate generation—beyond the objectives of adequate observation, description, and explanation.

关键词

人工智能 / 大语言模型 / 语言学知识 / 空间语义理解 / 数据合成

Key words

artificial intelligence / large language models / linguistic knowledge / spatial semantic understanding / data synthesis

引用本文

导出引用
詹卫东,孙春晖,肖力铭. 语言学知识驱动的空间语义理解能力评测数据集研究[J]. 语言战略研究. 2024, 9(5): 7-21 https://doi.org/10.19689/j.cnki.cn10-1361/h.20240501
Zhan Weidong, Sun Chunhui and Xiao Liming. SpaCE: A Linguistic Knowledge-Driven Benchmark for Spatial Cognition Evaluation[J]. Chinese Journal of Language Policy and Planning. 2024, 9(5): 7-21 https://doi.org/10.19689/j.cnki.cn10-1361/h.20240501

基金

教育部人文社会科学重点研究基地重大项目“面向机器语言能力评测的综合型语言知识库研究”(22JJD740004)。本研究得到北京大学计算语言所穗志方教授和常宝宝副教授课题组、复旦大学计算机学院邱锡鹏教授课题组大力支持;北京大学中文系多位同学参与了本文的工作。SpaCE2021、SpaCE2022、SpaCE2024评测大赛由华为公司提供奖金资助。特此致谢。

Accesses

Citation

Detail

段落导航
相关文章

/