How deep are your boxes? Scarlets will do less well in shallow soil, ideally it should be a foot or more deep. When I yank the vines at the end of the season, the root systems are quite impressive. The average window-box depth (5-7 inches) will not be sufficient.
Hummingbirds are attracted by the colour red. You may want to add a hummingbird feeder to make your courtyard even more attractive.