Super Mario Bros, the iconic video game, has found a new purpose as a training ground for artificial intelligence (AI). Researchers from Hao AI Lab, a research org at the University of California, San Diego, have utilized this classic game to test various AI models, revealing both the potential and limitations of these systems in real-time decision-making.
The experiment involved an AI-controlled Mario, powered by an emulated version of the game integrated into GamingAgent, a custom-built framework. The AI received screenshots and basic instructions, such as "If an obstacle approaches, jump to the left," and generated Python code to control Mario. The goal was to assess how well these models could adapt and develop gaming strategies.
Surprisingly, advanced models like OpenAI's GPT-4o struggled with the task. Their main issue? Overthinking. In a fast-paced game like Super Mario Bros, hesitation leads to failure. Conversely, less sophisticated but more responsive models, such as Anthropic's Claude 3.7, performed better, highlighting the importance of quick decision-making in dynamic environments.
While using video games to evaluate AI isn't new, some researchers question their relevance. Super Mario Bros challenges AI to anticipate and react quickly, but it remains a game with fixed rules and a limited environment. Andrej Karpathy, a researcher at OpenAI, even describes an "evaluation crisis," where it's unclear which tests truly reflect the capabilities of modern AI models.
Ultimately, while Super Mario Bros exposes the limitations of certain AI models in real-time scenarios, it also underscores the need for more diverse and complex evaluation methods to fully understand AI's potential.