So how does he get along with Finn? Did he do this before Skylar came along or is it since the addition of a larger, more dominant animal?
Is there a room that could be just his? As in, no other animals come in? Maybe you have a small room or a large closet that could be set up for his use only. When you are home and can keep an eye on him then he is out with you but when you are gone or outside working then he can go into his room. I'm just throwing out ideas here.
If you are interested in an outside enclosure I can send you a link for a picture of the one hubby made for KMK. H doesn't want to come out of his. I open the door so he can go explore with the rest of the kitties and he runs to the back of the enclosure. It is 6d x 8w x 6h with a section at one end closed off for his food, litter and bed. Not saying that Angus will want to be outside 24/7 but just in case. You could add shelving, hang ropes, a hammock, lots of things for him to play with and do. Maybe something like that so he can be outside would be enough to make him happy. You would still bring him in at night or let him stay out if he is happier. At least you would know he is safe. If you have no shade trees you could use a large tarp for the top.