From Thomas Jefferson Encyclopedia
The mountain that Thomas Jefferson named Montalto is a part of the Carter's Mountain range and is today sometimes known as Brown's Mountain. In 1777, Jefferson purchased 483 acres of land on the mountain from Edward Carter, the second son of John Carter, who had received the land from King George II as his Colonial Secretary. Jefferson never built anything on Montalto, although he did draw plans for an observation tower there. Following Jefferson's death, the land was inherited by his daughter Martha Jefferson Randolph. In 1832, Montalto was sold to Benjamin Sneed, and subsequently passed through numerous owners.
In 1905, the land was purchased by James Addison Patterson, who began construction of the house and barns, designed by architect Charles Barton Keen. The house was named “Repose”, and upon Patterson's death in 1931, was bequeathed to the Martha Jefferson Hospital, who rented portions of it out.
In 1950, the property was purchased by Lois and Nelson Brown, for whom it is now named. The Brown family renovated the house and made minor additions. In the 1950s, the Brown's operated a gift shop out of the barn, until it burned in 1959. After that time, the remaining barns were rented as apartments.