Babies are gone now; Do I remove Nest?

[Smiley14]

I had some house finches move in while I was on vacation for two weeks. I came back to a nest and when I peeked inside, there were four little light blue eggs. They hatched a week later and then a few short weeks later, they suddenly disappeared one day and I haven't seen or heard from them since. The nest appears to have been abandoned now for the past week. Should I leave it there just in case or is it safe to take it down now? It's full of bird poop and making my deck a little stinky, so I'd like to take it down and clean up my light now, LOL, but I won't if they come back to their nests. I only got a few blurry pics of them while they were there unfortunately, but here's a few of the cuties.

Mom:


Dad feeding the babies.