A house 5 mins down the road from me (East Yorkshire) has just been built on a small equine/ small holding site.
The owners have lived in a static caravan for around 20 years and have been playing tennis with the local planning authorities for many years now to get the desired permision.
But finally this year they have built a brick house, and a very substantial brick house at that!
So maybe explore buying a plot of land, getting services to it, and then putting a static or holiday lodge on it. My F.I.L works in the hoilday park industry and the lodges he deals with are very nice and can be lived in rather comfortably.
This way you can take your time planning the house of your dreams as you will know the site better. And you may find out that looking after live stock is not all its cracked up to be and choose a different hobby - land no longer required.
"EDIT" i share your dream too by the way but managed to buy the land with the house already built. Don't give up.