Papers

2024

1. Preprint

   Mission: Impossible Language Models

   Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts

   2024

   Abs arXiv Bib PDF

   Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. However, there is very little published experimental evidence to support such a claim. Here, we develop a set of synthetic impossible languages of differing complexity, each designed by systematically altering English data with unnatural word orders and grammar rules. These languages lie on an impossibility continuum: at one end are languages that are inherently impossible, such as random and irreversible shuffles of English words, and on the other, languages that may not be intuitively impossible but are often considered so in linguistics, particularly those with rules based on counting word positions. We report on a wide range of evaluations to assess the capacity of GPT-2 small models to learn these uncontroversially impossible languages, and crucially, we perform these assessments at various stages throughout training to compare the learning process for each language. Our core finding is that GPT-2 struggles to learn impossible languages when compared to English as a control, challenging the core claim. More importantly, we hope our approach opens up a productive line of inquiry in which different LLM architectures are tested on a variety of impossible languages in an effort to learn more about how LLMs can be used as tools for these cognitive and typological investigations.

   @misc{kallini2024mission,
     title = {Mission: Impossible Language Models},
     author = {Kallini, Julie and Papadimitriou, Isabel and Futrell, Richard and Mahowald, Kyle and Potts, Christopher},
     year = {2024},
     eprint = {2401.06416},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2023

1. GWC

   What to Make of make? Sense Distinctions for Light Verbs

   Julie Kallini and Christiane Fellbaum

   In Global WordNet Conference 2023, Jan 2023

   Abs Bib HTML PDF

   Verbs like make, have, and get present challenges for applications requiring automatic word sense discrimination. These verbs are both highly frequent and polysemous, with semantically “full” readings, as in make dinner, and “light” readings, as in make a request. Lexical resources like WordNet encode dozens of senses, making discrimination difficult and inviting proposals for reducing the number of entries or grouping them into coarser-grained supersenses. We propose a data-driven, linguistically-based approach to establishing a motivated sense inventory, focusing on make to establish a proof of concept. From several large, syntactically annotated corpora, we extract nouns that are complements of the verb make, and group them into clusters based on their Word2Vec semantic vectors. We manually inspect, for each cluster, the words with vectors closest to the centroid as well as a random sample of words within the cluster. The results show that the clusters reflect an intuitively plausible sense discrimination of make. As an evaluation, we test whether words within a given cluster cooccur in coordination phrases, such as apples and oranges, as prior work has shown that such conjoined nouns are semantically related. Conversely, noun complements from different clusters are less likely to be conjoined. Thus, coordination provides a similarity metric independent of the contextual embeddings used for clustering. Our results pave the way for a WordNet sense inventory that, while not inconsistent with the present one, would reduce it significantly and hold promise for improved automatic word sense discrimination.

   @inproceedings{GWC2023,
     title = {What to Make of <em>make</em>? Sense Distinctions for Light Verbs},
     author = {Kallini, Julie and Fellbaum, Christiane},
     booktitle = {Global WordNet Conference 2023},
     month = jan,
     year = {2023},
     address = {Donostia-San Sebastian, Spain},
     publisher = {Global WordNet Association},
   }

2022

1. TSD

   Computational Approaches for Understanding Semantic Constraints on Two-termed Coordination Structures

   Julie Kallini and Christiane Fellbaum

   In Proceedings of the 25th International Conference on Text, Speech and Dialogue, Sep 2022

   Abs Bib HTML PDF Slides

   Coordination is a linguistic phenomenon where two or more terms or phrases, called conjuncts, are conjoined by a coordinating conjunction, such as and, or, or but. Well-formed coordination structures seem to require that the conjuncts are semantically similar or related. In this paper, we utilize English corpus data to examine the semantic constraints on syntactically like coordinations, which link constituents with the same lexical or syntactic categories. We examine the extent to which these semantic constraints depend on the type of conjunction or on the lexical or syntactic category of the conjuncts. We employ two distinct, independent metrics to measure the semantic similarity of conjuncts: WordNet relations and semantic word embeddings. Our results indicate that both measures of similarity have varying distributions depending on the particular conjunction and the conjuncts’ lexical or syntactic categories.

   @inproceedings{TSD2022,
     author = {Kallini, Julie and Fellbaum, Christiane},
     editor = {Sojka, Petr and Hor{\'a}k, Ale{\v{s}} and Kope{\v{c}}ek, Ivan and Pala, Karel},
     title = {Computational Approaches for Understanding Semantic Constraints on Two-termed Coordination Structures},
     booktitle = {Proceedings of the 25th International Conference on Text, Speech and Dialogue},
     month = sep,
     year = {2022},
     publisher = {Springer International Publishing},
     address = {Cham},
     pages = {64--76},
     isbn = {978-3-031-16270-1}
   }

2021

1. EMNLP

   A Corpus-based Syntactic Analysis of Two-termed Unlike Coordination

   Julie Kallini and Christiane Fellbaum

   In Findings of the Association for Computational Linguistics: EMNLP 2021, Nov 2021

   Abs Bib HTML PDF Slides

   Coordination is a phenomenon of language that conjoins two or more terms or phrases using a coordinating conjunction. Although coordination has been explored extensively in the linguistics literature, the rules and constraints that govern its structure are still largely elusive and widely debated amongst linguists. This paper presents a study of two-termed unlike coordinations in particular, where the two conjuncts of the coordination phrase form valid constituents but have distinct categories. We conducted a syntactic analysis of the phrasal categories that can be conjoined in such unlike coordinations through a computational corpus-based approach, utilizing the Corpus of Contemporary American English (COCA) as the main data source, as well as the Penn Treebank (PTB). The results show that the two conjuncts within unlike coordinations display different properties based on their position, supporting an antisymmetric view of the structure of coordination. This research provides new data and perspectives through the use of statistical techniques that can help shape future theories and models of coordination.

   @inproceedings{kallini-fellbaum-2021-corpus-based,
     title = {A Corpus-based Syntactic Analysis of Two-termed Unlike Coordination},
     author = {Kallini, Julie and Fellbaum, Christiane},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
     month = nov,
     year = {2021},
     address = {Punta Cana, Dominican Republic},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2021.findings-emnlp.335},
     doi = {10.18653/v1/2021.findings-emnlp.335},
     pages = {3998--4008},
   }