As they say , let sleeping dogs lie. The only thing I would recommend is to try not to sneak up on him. If you find you are a light walker, try tapping your foot on the floor next to him before waking him, so he knows you are there. Instead of reaching straight for his head, pet his flank until you see him wake up.
Do you find any other circumstances where he is grouchy?
Do you find any other circumstances where he is grouchy?