Your sweetie just learned that if he growls, you back off and he wins which means he's the top dog.
Dogs catch on to this so quickly it's amazing and it's how they behave with each other. The weaker dog will back down as you did.
You can try taking something irresistable out with you - bacon or hot dogs or anything else your dog goes nuts for. Make sure it's something really special and something he gets ONLY when he drops what you don't want him to have.
When he picks up something, hold the treat in front of his nose, and say "Drop it". He will likely let go of what he's holding to get the new treat. Give it to him immediately and move on. Let the amount of time between when he drops the unwanted item and you give him the treat increase and eventually phase out the treat altogether.
You can practice this "trading" of items at home.
Don't get into tugging contests with him and do not show fear and back down.