Large hole should be filled with mortar. A mortar gun helps. For any other exterior sealing, I use hybrid polymer sealant. These are high modulus, meaning minimum shrinking. Anything that shrinks will rip itself apart.
Silicone fail. End of.If you want to get into further detail they also transfer less stress and using an acetoxy silicone can cause uPVC surface changes and adhesion failures.
Silicone fail. End of.