In this blog, we shed light on learning to drive and testing for human drivers within the context of current EU regulations. We hypothesize that these training lessons and driving tests are a possible blueprint for a test of generalization for AVs.

We posit that with large, autoregressive models robotics will benefit from e2e models. In this blog, we take the idea of next word prediction given a partial sentence to next action prediction given the context (e.g. image, sensor data). We found that a simple formulation doesn't work, and requires modality-specific changes to GPTs for them to learn effective, cross-modal attention and embeddings.