Autonomous decision-making under uncertainty is a fundamental problem in AI and robotics. Despite recent progress, existing approaches remain insufficient for reliable real-world deployment. Methods that rely on models trained with limited or inaccurate data often produce unsafe behavior or suboptimal performance. Robust decision-making methods address this by accounting for model uncertainty, but existing robust algorithms are typically limited to offline settings and scale poorly. Conversely, online planning methods scale to large and continuous domains. In this project develop we develop theoretical and algorithmic foundations for online robust planning in MDPs and POMDPs, explicitly accounting for model uncertainty and providing formal performance guarantees.