UPDATE
I didn't take him to the vet, cause I called and they said they didn't think it was possible for ear mites to live under a ball of wax, so I just kept cleaning it with Q-tips and giving him some general ear drops and it's much better now. I still clean it about once a week and it only takes about 2 Q-tips instead of 12 like before.
