Computer Intelligence Project

The Computer Intelligence Project builds on the BigCode Project, developing open models, datasets, and evaluation tools for computer-level code intelligence.

The ultimate goal of code intelligence is to automate computer-level tasks via executable programs. These computer-level tasks are typically based on graphical user interfaces (GUIs) rendered by software applications. This means successful automation requires models to understand both the textual information and the visual contexts perceived by human users -- including layout, UI elements, and their functional relationships. However, current multimodal models still struggle to effectively design and interact with these visual contexts. Additionally, there is limited data available to train such models.

In this project, we aim to develop a new generation of code intelligence -- Computer Intelligence, that can fully utilize the computer-level visual information to perform complex and practical tasks. Particularly, we focus on three key application domains: Web Interface Automation, Data Science Workflows, and Full Stack Development. The expected outcome will be open models, datasets, and evaluation tools.