IMO the kitties should stay with you. Could you offer to keep them and let him visit or would you not feel comfortable with that? You have been their major caregiver for quite some time. Obviously Punky feels the same way since he comes back to you.
He may care about them but he isn't very responsible.
