Knowing history of the real world helps so much with your world building. Looking at pictures of actual place gives you ideas for your settings.
I love looking up haunted mansions and reading about the ghosts. They've inspired the building in my world. Some of the stories behind these haunted places are more interesting than some novels I've read. Also, let's say, you have some supernatural element in your story. Reading about "real" hauntings help you create a chilling atmosphere. You can't just rely on movies.
One thing I ran into while making my world map was architectural style. It never occurred to me until then, but mansions and castles are defined by a stylistic period i.e. buildings from the same style period will have similar characteristics. Gothic buildings have high pointed arches and flying buttress.
For my village map, I had to keep in mind that the buildings needed to have similar characteristics or explain, in the story, why they didn't.
Last week, i09 put together this amazing list of haunted mansions and castles complete with pictures and the story behind the hauntings. This is the largest list of castles and mansions I've come across so far. The pictures are awesome. This article is a good starting point for more in depth research. World building aside, I just like a good ghost story and looking at historic buildings.
"Raynham Hall, built in the 1630s, Norfolk, Great Britain
The sister of Robert Walpole, the first Prime Minister of Great Britain was the second wife of a British Whig statesman named Charles Townshend, who was famous for his violent temper. When Townshend discovered that his wife had cheated him with Lord Wharton he locked the Lady to her rooms. She remained there until her death in 1726. Her ghost was first observed in 1835, and the famous photograph was taken by Captain Hubert C. Provand in 1936."One day, I'm going to visit all those places.