Depends how big/heavy the basket is too. But I've always put mine into the brick, or wherever possible - somewhere that doesn't require drilling into my house. If the mortar's quite old, they can gradually work their way out and before you know it, plants all over the path!
We've got a nice pergola and they hang quite nicely on the wooden frame.