Back
neurips-2025-less-is-more-local-intrinsic-dimensions-of-contextual-language-models-paper-conference_0
Conference paper (in proceedings)

Less is More: Local Intrinsic Dimensions of Contextual Language Models

BP2-STS

Show more…
  • 2026
Published in:
  • 39th Conference on Neural Information Processing Systems (NeurIPS 2025). - 2026
Topological data analysis
Topological deep learning
Geometric deep learning
Intrinsic dimension; large language models
English Understanding the internal mechanisms of large language models (LLMs) remains a challenging and complex endeavor. Even fundamental questions, such as how fine-tuning affects model behavior, often require extensive empirical evaluation. In this paper, we introduce a novel perspective based on the geometric properties of contextual latent embeddings to study the effects of training and fine-tuning. To that end, we measure the local dimensions of a contextual language model's latent space and analyze their shifts during training and fine-tuning. We show that the local dimensions provide insights into the model's training dynamics and generalization ability. Specifically, the mean of the local dimensions predicts when the model's training capabilities are exhausted, as exemplified in a dialogue state tracking task, overfitting, as demonstrated in an emotion recognition task, and grokking, as illustrated with an arithmetic task. Furthermore, our experiments suggest a practical heuristic: reductions in the mean local dimension tend to accompany and predict subsequent performance gains. Through this exploration, we aim to provide practitioners with a deeper understanding of the implications of fine-tuning on embedding spaces, facilitating informed decisions when configuring models for specific applications. The results of this work contribute to the ongoing discourse on the interpretability, adaptability, and generalizability of LLMs by bridging the gap between intrinsic model mechanisms and geometric properties in the respective embeddings.
Faculty
Faculté des sciences et de médecine
Department
Département d'Informatique
Language
  • English
Classification
Computer science and technology
License
License undefined
Open access status
green
Persistent URL
https://folia.unifr.ch/unifr/documents/336332
Statistics
Document views: 3 File downloads:
  • neurips-2025-less-is-more-local-intrinsic-dimensions-of-contextual-language-models-paper-conference_0: 1