In the mid 19th century Earlston was experiencing an industrial boom. Buildings such as Rhymer's Mill, and new houses, were being built in the vicinity of the tower ruins. Locals tell that many of the stones used in these constructions were indeed taken from the tower. Two thatched cottages, which stood within the ruins of the tower until the early 20th century, are said to have been built with stones taken from the tower. The tower was slowly disappearing. In 1894 the tower, and adjoining land, was bought by The Edinburgh Borders Counties Association, to safeguard what we see today. All that remains of this once grand tower is two walls. It was granted a Category B listed building status by Historic Scotland on June 9th 1971.
There is a plaque on one of the remaining walls bearing the words written by Sir Walter Scott.
"Farewell my Fathers ancient tower!
A long farewell" said he;
"The scene of pleasure, pomp or power, thou never more shalt be.
Adieu !Adieu!" again he cried,
all as he turned him roun'
"Farewell to Leaders silver tide!
Farewell to. Ercildoune".
This plaque was unveiled in August 1894 by Mr Wallace Bruce, United States Consul in Edinburgh, to celebrate the new ownership of the Tower by the Edinburgh Border Counties Association.
On March 30th 1966, the 'Rhymer's Tower Trustees' officially took over the responsibility of the Tower, from the above mentioned Association.