# PARTIALLY-CONTROLLED MARKOV DECISION PROCESSES FOR COLLISION AVOIDANCE SYSTEMS

### Mykel J. Kochenderfer, James P. Chryssanthacopoulos

#### Abstract

Deciding when and how to avoid collision in stochastic environments requires accounting for the likelihood and relative costs of future sequences of outcomes in response to different sequences of actions. Prior work has investigated formulating the problem as a Markov decision process, discretizing the state space, and solving for the optimal strategy using dynamic programming. Experiments have shown that such an approach can be very effective, but scaling to higher-dimensional problems can be challenging due to the exponential growth of the discrete state space. This paper presents an approach that can greatly reduce the complexity of computing the optimal strategy in problems where only some of the dimensions of the problem are controllable. The approach is demonstrated on an airborne collision avoidance problem where the system must recommend maneuvers to an imperfect pilot.

#### References

- Bellman, R. E. (1961). Adaptive control processes: A guided tour. Princeton University Press.
- Bertsekas, D. P. (2005). Dynamic Programming and Optimal Control, volume 1. Athena Scientific, Belmont, Mass., 3rd edition.
- Bilimoria, K. D. (2000). A geometric optimization approach to aircraft conflict resolution. In AIAA Guidance, Navigation, and Control Conference and Exhibit, Denver, Colo.
- Carpenter, B. D. and Kuchar, J. K. (1997). Probabilitybased collision alerting logic for closely-spaced parallel approach. In AIAA 35th Aerospace Sciences Meeting, Reno, NV.
- Chamlou, R. (2009). Future airborne collision avoidancedesign principles, analysis plan and algorithm development. In Digital Avionics Systems Conference.
- Chryssanthacopoulos, J. P., Kochenderfer, M. J., and Williams, R. E. (2010). Improved Monte Carlo sampling for conflict probability estimation. In AIAA Non-Deterministic Approaches Conference, Orlando, Florida.
- Davies, S. (1997). Multidimensional triangulation and interpolation for reinforcement learning. In Mozer, M. C., Jordan, M. I., and Petsche, T., editors, Advances in Neural Information Processing Systems, volume 9, pages 1005-1011. MIT Press, Cambridge, Mass.
- Dowek, G., Geser, A., and Mun˜ oz, C. (2001). Tactical conflict detection and resolution in a 3-D airspace. In 4th USA/Europe Air Traffic Management R&D Seminar, Santa Fe, New Mexico.
- Duong, V. N. and Zeghal, K. (1997). Conflict resolution advisory for autonomous airborne separation in lowdensity airspace. In IEEE Conference on Decision and Control, volume 3, pages 2429-2434.
- Eby, M. S. and Kelly, W. E. (1999). Free flight separation assurance using distributed algorithms. In IEEE Aerospace Conference, volume 2, pages 429-441.
- Khatib, O. and Maitre, J.-F. L. (1978). Dynamic control of manipulators operating in a complex environment. In Symposium on Theory and Practice of Robots and Manipulators, pages 267-282, Udine, Italy. Elsevier.
- Kochenderfer, M. J. and Chryssanthacopoulos, J. P. (2010). A decision-theoretic approach to developing robust collision avoidance logic. In IEEE International Conference on Intelligent Transportation Systems, Madeira Island, Portugal.
- Kochenderfer, M. J., Chryssanthacopoulos, J. P., Kaelbling, L. P., and Lozano-Perez, T. (2010a). Model-based optimization of airborne collision avoidance logic. Project Report ATC-360, Massachusetts Institute of Technology, Lincoln Laboratory.
- Kochenderfer, M. J., Edwards, M. W. M., Espindle, L. P., Kuchar, J. K., and Griffith, J. D. (2010b). Airspace encounter models for estimating collision risk. Journal of Guidance, Control, and Dynamics, 33(2):487-499.
- Kurniawati, H., Hsu, D., and Lee, W. (2008). SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Robotics: Science and Systems.
- Kuwata, Y., Fiore, G. A., Teo, J., Frazzoli, E., and How, J. P. (2008). Motion planning for urban driving using RRT. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1681-1686.
- LaValle, S. M. (1998). Rapidly-exploring random trees: A new tool for path planning. Technical Report 98-11, Computer Science Department, Iowa State University.
- Powell, W. B. (2007). Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley, Hoboken, NJ.
- Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley series in probability and mathematical statistics. Wiley, New York.
- RTCA (2005). Safety analysis of proposed change to TCAS RA reversal logic, DO-298. RTCA, Inc., Washington, D.C.
- RTCA (2008). Minimum operational performance standards for Traffic Alert and Collision Avoidance System II (TCAS II), DO-185b. RTCA, Inc., Washington, D.C.
- Saunders, J., Beard, R., and Byrne, J. (2009). Vision-based reactive multiple obstacle avoidance for micro air vehicles. In American Control Conference, pages 5253- 5258.
- Smith, T. and Simmons, R. G. (2005). Point-based POMDP algorithms: Improved analysis and implementation. In Uncertainty in Artificial Intelligence.
- Temizer, S., Kochenderfer, M. J., Kaelbling, L. P., LozanoPérez, T., and Kuchar, J. K. (2010). Collision avoidance for unmanned aircraft using Markov decision processes. In AIAA Guidance, Navigation, and Control Conference, Toronto, Canada.
- Yang, L. C. and Kuchar, J. K. (1997). Prototype conflict alerting system for free flight. Journal of Guidance, Control, and Dynamics, 20(4):768-773.

#### Paper Citation

#### in Harvard Style

J. Kochenderfer M. and P. Chryssanthacopoulos J. (2011). **PARTIALLY-CONTROLLED MARKOV DECISION PROCESSES FOR COLLISION AVOIDANCE SYSTEMS** . In *Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,* ISBN 978-989-8425-40-9, pages 61-70. DOI: 10.5220/0003135800610070

#### in Bibtex Style

@conference{icaart11,

author={Mykel J. Kochenderfer and James P. Chryssanthacopoulos},

title={PARTIALLY-CONTROLLED MARKOV DECISION PROCESSES FOR COLLISION AVOIDANCE SYSTEMS},

booktitle={Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,},

year={2011},

pages={61-70},

publisher={SciTePress},

organization={INSTICC},

doi={10.5220/0003135800610070},

isbn={978-989-8425-40-9},

}

#### in EndNote Style

TY - CONF

JO - Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,

TI - PARTIALLY-CONTROLLED MARKOV DECISION PROCESSES FOR COLLISION AVOIDANCE SYSTEMS

SN - 978-989-8425-40-9

AU - J. Kochenderfer M.

AU - P. Chryssanthacopoulos J.

PY - 2011

SP - 61

EP - 70

DO - 10.5220/0003135800610070