Opposite the church there was once a complex of barns, which probably included the Tithe Barn, and behind these was an old house, which may have been the original farmhouse. By the time Lovelace bought the estate, it had been divided into three cottages for farm workers. This would have been the farm belonging to Bishops Manor which finally became Horsley Towers.
The original Bishops Manor House stood behind the church and until the Reformation it belonged to the Bishops of Exeter. It was rebuilt by William Currie in 1820 a little further to the east, and became the central part of Lord Lovelaces creation, later called Horsley Towers.
Lovelace went on to build his new Home Farm to the north of Horsley Towers. Then on the site of the original Manor Farm he built The Manor House for the express purpose of renting it out to persons of the more genteel classes. The Manor House appears to be of several builds. The front portion is in red brick with moulded corbelling and terracotta friezes, while the part at the back and sides may have been built a little later and is in the flint and brick he adopted for his East Horsley style.