I'm with Frenchy on this one. He might seem like a nice guy but I wold NEVER leave town if my cats (who are indoor kitties to start with so it's moot) outside. I'd find them or stay home. It's like leaving your kids alone because they were outside playing while you had to leave. Makes no sense!!
I have to admit I;d have been sorely tempted to have him neutered (if a vet would do it at a good price for you knowing you are in rescue) before the owner arrived and just explain you thought he was a stray. Literally for that short period, he WAS!!!
I really hope you have good luck with education about neutering!
"There are two means of refuge from the miseries of life: music and cats" Albert Schweitzer