Coco84 - these people are only telling you this because it is them (and myself) that are picking up abused, neglected and KILLED cats that are left outside. We have cut them out of the engines of cars, removed their carcasses from dogs, and bring them to vets to be euthanized because of feline aids or leukemia.
Right now, there are two men that are clubbing outdoor cats and doing who knows what with them.
Cats that live outside have a shorter life expentancy. I have 3 out of 4 cats that were abandoned outside - one was litterally strangled by a man that did not want a cat in 'his' yard. She was only 6 months old. Luckily a rescue group took over and I fostered her back to health and ended up adopting her myself - she never fully recovered. And I have to tell you - not one of them (especially the strangled one) even bothers to ask for the door.
It is absolutely your business as to what you want to do with your pet - but the people here truly do know what is happening to outdoor cats because they have hands on.
Believe it or not, everyone wants nothing but the best for your cat. We pray that he is alive and not euthanized already by animal control, suffering somewhere alone, or abused or killed by some psychopath.
If you really were aware of all the dangers, then you would not let him out again.