The current name of the town, Vysoký Chlumec, dates to mid-19th century, since by that point it became necessary to differentiate several settlements in Bohemia named Chlumec. The town's most significant landmark is its castle, originally Gothic and later rebuilt in the Renaissance style, founded sometime before 1382. The development of that noble residence and the manor as a whole was significantly influenced by the arrival of the Lobkowicz family. The manor prospered in large part due to the efforts of Zdeněk Vojtěch of Lobkowicz, who married Polyxena of Pernštejn in 1603, thereby significantly expanding the domain of his noble house and establishing the Chlumec manor as one of the largest in Bohemia. The Lobkowicz property was confiscated during the period of the Nazi protectorate and then expropriated again under the communist regime. It was only returned to its rightful owners in 1992, and then later purchased in the 1990s from them by Riprand Graf von und zu Arco-Zinneberg.