Artificial Intelligence seeks agents that can ‘perceive’ the world and ‘act’ accordingly. Despite remarkable progress toward this goal, computer vision solutions face a challenge in scaling to the complexity of the real world -- which often leads to reducing the operation domain of active agents to perceptually simplified ones, such as controlled spaces or video games. Additionally, the predominant approaches to developing vision solutions are in disconnect from action. I will present some of our efforts toward addressing these two core problems. I will show a mid-level vision module can notably improve the sample efficiency and generalization of active robotic agents. I will discuss how we learn such a mid-level vision, how we make it robust, and how we make it consistent.