Should I be angry or am I being unfair?

Yes you have the right to be angry but it might be best that you get the bus from now on if he isnt reliable. Also you can also refuse to pay him on the next week he takes you.