ESE 651 · Physical Intelligence · Penn
Drone Racing
Learning a faster line.
How do you teach a drone to race? We started by telling it where to fly, then learned to give it room to find its own way. A project in reinforcement learning, from the first power loops in simulation to a real-world flight.
- The challenge
- 3 laps, fastest time
- The learner
- PPO
- Simulation iterations
- ~21 s ~19 s
- The destination
- Real flight
The setup
A small drone. A demanding course.
Build a world to practice in.
The goal was simple: fly through the gates in order and finish three laps in the shortest time. Getting there meant learning when to accelerate, how to turn, and how to arrive at each gate ready for the next one.
We used NVIDIA Isaac Lab, built on Isaac Sim, to train a simulated Crazyflie quadcopter. Thousands of copies of the course could run in parallel, letting the policy collect flight experience much faster than a single physical drone could.
A little more about the environment
The setup uses Isaac Sim 4.5, the course’s Isaac Lab fork, a Python 3.10 environment, CUDA-enabled PyTorch, and a local copy of rsl_rl. Physics and the low-level controller run at 500 Hz; the policy sends a new command at 50 Hz. Its observations include body velocity, orientation, and the current and next gate geometry, giving it a look ahead instead of only a single target.
Teaching the policy
Reinforcement learning
The reward is the instruction.
We used Proximal Policy Optimization (PPO). The policy tries a flight, receives a score, and updates its behavior. PPO keeps each update relatively small, so a promising controller can improve without changing too abruptly.
The interesting part was deciding what “better” meant. A reward function turns our intent into feedback: encourage useful behavior, charge for mistakes, and balance those signals until the drone learns a complete race.
Make progress
Reward the decrease in distance to the next gate at every step. This gives the drone useful feedback long before it finishes a lap.
Pass the right gate
Add a bonus only when the drone crosses the gate opening in the correct direction. A valid crossing advances the target, so hovering near a gate is not enough.
Move with purpose
Reward speed directed toward the next target. We adjusted this term to encourage a faster racing line, rather than simply moving quickly in any direction.
Make time matter
Charge a small cost for every step until three laps are complete. Waiting and taking a long detour now have a price, encouraging the drone to finish sooner.
Keep the motion controllable
Penalize aggressive roll, pitch, and yaw commands. We tuned their influence so the policy could turn quickly without becoming erratic.
Make mistakes expensive
Penalize collisions and invalid gate passes. These costs keep the search for speed tied to actually completing the course.
Arrive at the right height
In later flight refinements, add a height-error penalty near the gate. It encourages the drone to line up vertically before reaching the opening.
What we changed in the code
In quadcopter_strategies.py, the racing logic scores progress, checks gate
crossings, advances targets, and combines the reward terms. We added look-ahead gate
observations and resets from previously successful flight states, so training could
practice beyond the starting line. Reward weights are set in train_race.py.
In the local ppo.py, we implemented the policy update: compare the new action
probabilities with the old ones, clip large changes, and train a value function to
estimate how promising each state is. The learner stays the same while we change the
feedback and the world it practices in.
Finding speed
Two versions, one useful lesson
From a prescribed path to a learned line.
First, make it go over.
~21s / 3 laps
Our first approach guided the shape of the maneuver. We added an elevated virtual target to encourage the drone to climb above the gate and come back around. This made the intended loop easier to learn, but also asked the drone to follow a path we had chosen.
Version 1 · Guided flight
Then, let it find the speed.
~19s / 3 laps
We then asked a different question: if we reward moving quickly toward the next gate, can the policy discover a better route itself? We moved away from the forced high waypoint and tuned the balance between progress, useful velocity, and control penalties. The gate sequence still defined the race; the policy had more freedom to choose the line.
Version 2 · Learned racing line
About two seconds faster across three laps, with more of the racing line discovered by the policy.
Approximate simulation run times. The clips show the two versions from different views; video duration is not the race timer.
Sim to real
Prepare for an imperfect world
Train for more than one drone.
A fast simulation policy still has to work on hardware. The real drone will not respond exactly like the model: thrust, drag, and controller response can all differ. We used domain randomization to vary those properties during training, then fine-tuned the racing policy across the variations.
Think of it as practicing the same course with a slightly different drone each time. The policy has to learn behavior that works across a range of conditions, instead of depending on one perfect set of physics parameters.
Thrust strength
Give each simulated drone a little more or less lifting power, so the policy can handle a different thrust response.
Aerodynamic drag
Vary how strongly the air slows the drone. The same command should still work when the real vehicle carries speed differently.
Controller response
Vary the low-level PID gains, changing how strongly the drone responds and settles after a command.
How the randomization was added
At reset, quadcopter_strategies.py samples thrust-to-weight ratio, aerodynamic
drag, and PID gains for each environment. We widened these ranges for robustness training: P
and I gains went from ±15% to ±20%, and D from ±30% to ±35%. A saved racing checkpoint provides
the starting point for fine-tuning.
We also added configurable observation-noise support. Its noise levels are zero in the saved configuration, so the active randomization here concerns flight dynamics and controller gains.
The flight
Out of the simulator
Bring the policy into the room.
The final step was testing the trained policy on a physical course. After all the reward tuning and simulated practice, this is the part where the project becomes tangible: a real drone, real gates, and the learned controller in flight.
Real-world flight
We started by teaching a maneuver. We ended by learning how to write a better objective—and how to give the policy enough room to solve it.