I wouldn't say it was aggression either, just typical kitty stuff.
Sometimes cats can get over stimulated when you are patting them and neutering will not change that. Normally you will see a change in facial expression or the tail will start flickering just before they bite. Watch for these signs and stop the patting.
As for play time, a firm no when he bites your hand and then redirect his attention to the toy. Do you allow him to get the toy and is it something he can get his teeth into?
Cat maid to:
Jasper, male Ragdoll ?? (approx 10 yrs)
Sweet Pea, female silver tabby DSH (approx 8 yrs)
Rose semi feral, a cpietra rescue, female tabby (approx 7 yrs)
Puddles RIP (1996-2014
Snowball RIP (1991-2005)
In a cat's eye, all things belong to cats.-English Proverb
“While we are free to choose our actions, we are not free to choose the consequences of our actions.” Stephen R. Covey