High order functions are functions that either take other functions as parameters or return functions. They allow functions to be passed into other functions as arguments and support abstraction. Some key advantages are that they can reduce program size, accelerate development for large programs, and have zero runtime cost in many cases by supporting abstraction and code reuse through passing functions as arguments.