Sounds like this sweet kitten really lacks confidence, which may partly be the fault of the breeder not sufficiently socializing him to a wide variety of situations, but like Marko said, could also just be the personality of this particular cat. There is a great chance that with the right care in your loving home, he will come out of his shell. Any chance you could adopt another cat his age, or perhaps buy one of his litter-mates? I think he'd probably benefit from some feline companionship to help him feel more comfortable.
Something else I would recommend is picking up a couple of Feliway
diffusers. They simulate the facial pheromones that cats rub on stuff with their cheeks and can help imbue the atmosphere with a sense of calm.
Cats also pick up on our emotions so he may be sensing that you're stressed, and particularly that your husband is getting perturbed. If you can both manage to exude only happy calmness around him, he might start to reflect that back.
Good luck with him! Please hang in there. Often the cats that are the most challenging to "win over" end up being the most loving and loyal kitties you can imagine.