We investigate methods for computer vision architectures to self-improve in unlabelled data, by exploiting rich regularities of the natural world. As a starting point, we embrace the fact that the world is 3D, and design neural architectures that map RGB-D observations into 3D feature maps. This representation allows us to generate self-supervision objectives using other regularities: we know that two objects cannot be in the same location at once, and that multiple views can be related with geometry. We use these facts to train viewpoint-invariant 3D features (unsupervised), and yield improvements in object detection and tracking. We then discuss entity-centric architectures where entities are informed from associative retrieval or through reconstruction feedback, and show their superior generalization over models without memory or without reconstruction feedback. We then shift focus to extracting information from dynamic scenes. We propose a way to improve motion estimation itself, by revisiting the classic concept of “particle videos”. Using learned temporal priors and within-inference optimization, we can track points across occlusions, and outperform flow-based and feature-matching methods on fine-grained multi-frame correspondence tasks.
Katerina Fragkiadaki is an Assistant Professor in the Machine Learning Department in Carnegie Mellon University. She received her Ph.D. from University of Pennsylvania and was a postdoctoral fellow in UC Berkeley and Google research after that. Her work is on learning visual representations with little supervision and on combining spatial reasoning in deep visual learning. Her group develops algorithms for mobile computer vision, learning of physics and common sense for agents that move around and interact with the world. Her work has been awarded with a best Ph.D. thesis award, an NSF CAREER award, AFOSR Young Investigator award, a DARPA Young Investigator award, Google, TRI, Amazon, UPMC and Sony faculty research awards.