Planning in continuous domains is essential for robotics, autonomous driving, and other physical and real-world systems. While gradient optimization is the backbone of modern AI, integrating it into traditional online planning algorithms has been a longstanding challenge, especially in online MDP and POMDP settings. Our work bridges this gap, unlocking gradient-based optimization within online planners and paving the way for scalable, efficient decision-making in continuous environments.