The SIGNAL (Situated Grounding and Natural Language) Lab studies language understanding and cognition using computational methods. We tackle cutting-edge AI problems from a strongly multimodal perspective, welding computational linguistics and natural language processing with computer vision and cognitive systems. We take a hybrid “neurosymbolic” approach that includes machine learning, logical reasoning, and simulation. The SIGNAL Lab was spun out of the Lab for Linguistics and Computation at Brandeis University and we collaborate heavily with other labs in the Colorado State Department of Computer Science, including the Communicating with Computers Lab and the Natural User Interface Lab. The lab is directed by Prof. Nikhil Krishnaswamy.
What is situated grounding?
“Grounding” refers to the process of giving a linguistic reference meaning by linking it to an actual entity (for example, the noun “cat” to an actual cat or the verb “run” to the act of running). “Situated” here refers to the contextual model a person or computational agent creates of the situation/environment it inhabits, including what entities surround it, and what it can do in that environment.
This is a skill that humans excel at but that even the most sophisticated AI systems still struggle with. Try asking your iPhone or Google Home, “What am I pointing at?” Because these devices have no model of the surrounding environment, they can’t answer the question!
“Situated grounding” is the process of giving symbols meaning by linking them to entities in the surrounding environment, It allows for a critical feature of language use: decontextualized reference, or the use of a symbol outside the situation where the linked entity is present. That is, there doesn’t need to be a cat in the room to talk about cats, as long as everyone understands what a cat is and what it does, which they most likely learned by seeing or interacting with a cat somewhere else.
What do we work on?
The term “situated grounding” was first used in the context of “situated grounding problems,” or failures of intelligent systems to link references to entities in context. By merging natural language understanding to sophisticated environmental and simulation models, we develop solutions to these problems, allowing for natural and useful interactions with intelligent agents, large datasets, and robots.
This involves work in AI, including developing and deploying machine learning and neurosymbolic models; theoretical and corpus linguistics, including semantic modeling and annotating data; and software engineering, including building real-time interactive architectures.
What are some projects?
Above is Diana, an intelligent interactive agent who understands spoken English and gestures to collaborate with a human in real time on construction tasks. We also have ongoing work with navigating robots (see image below) and multimodal data exploration.
Our primary platform is VoxWorld, which is built on the Unity game engine and VoxML, a multimodal semantic modeling language that allows us to interpret language, gesture, and other modalities in a simulated environment. Individual implementations built on VoxWorld allow humans to communicate with intelligent agents such as Diana in ways that the agent’s embodiment allows, be that manipulating virtual objects, or navigating a real space and finding items of interest.
Can I get involved?
If you are a current CSU student, feel free to contact us. If you are not a current CSU student, you should apply to the university. Prior experience with or a desire to learn computer graphics and/or machine learning is recommended. Strong software engineering skills are a plus. There is no guarantee that we are accepting new students but exploratory contacts are welcome.