I've been coding since I was 14 years old. As a result, I've accumulated several projects. Here are my favorites:
- CoMo: a Whiteboard that Converses about Code was my Sc.D. dissertation at MIT CSAIL: a smart whiteboard that held structured conversations with engineers, synthesized working C code from the examples they provided, and animated the result for verification.
- The Autonomous Forklift Project was my Master's work at MIT CSAIL: building the human operator interface for a self-driving forklift capable of navigating unstructured environments alongside people.
- GOPF: GNU Online Player Framework is a self-hosted media server: stream your personal media files to any device via a browser.
- Deskcorder is a screen and audio recorder aimed at lecture capture, coauthored with Ali Mohammad.
CoMo: a Whiteboard that Converses about Code
CoMo held a structured, mixed-initiative conversation with an engineer about a data structure manipulation. It asked questions by speech and sketch, collected before/after state examples, synthesized working C code using the Storyboard Programming Tool that Rishabh Singh developed, and animated the result on the user's drawn structures. If the animation revealed a bug, the user corrected an example and CoMo re-synthesized with the updated input.
The core intellectual contribution was the Mixed-Initiative Code-Generation Framework (MICGF): an orchestration layer that decided what to ask, when to ask it, selected the appropriate control flow graph, invoked the synthesis backend, handled timeouts by augmenting the search space, and closed the verification loop. Before "AI orchestration" was a term, the MICGF was already the same architectural pattern—a structured harness between a user and an AI backend, managing state and conversation flow, knowing which examples the backend needed and systematically eliciting them.
See the Media page for videos of CoMo in action.
The Autonomous Forklift Project
We built a forklift capable of safely driving itself with the guidance of a human operator. My primary task was designing and building the supervisor's hand-held interface and integrating it with the rest of the system, a highly collaborative effort involving researchers across several labs.
The forklift used LIDAR, cameras, and microphones to scan and model the environment around it. That model was sent to a Nokia N810 internet tablet, which served as the operator's primary control interface. The tablet accepted speech and sketch as input and translated them into actionable forklift commands—my adviser and I called this "Drawing on the World," the idea being that you draw on the world itself rather than a canvas or whiteboard.
Echoes of the interface ideas from this project show up in autonomous vehicle interfaces today at companies like Waymo and Tesla.
GOPF: GNU Online Player Framework
GOPF is the GNU Online Player Framework. Using HTML5, it lets you host your media files on a personal machine and serve them as a webpage to whichever device you're on. Licensed under the Affero GPLv3; source is on GitHub.
Deskcorder
Deskcorder records screen activity and audio and exports to PDF, PNG, or HTML5. I started it to support my sketch recognition research; Ali Mohammad used it to record and distribute lectures to his MIT 6.01 students. Written in Python and released under the GPLv3.