Perched on a hill above the village and the Escalote River, the Castle is an impressive fortified complex that illustrates the evolution of Spanish military architecture from the Middle Ages to the Renaissance. The site first held a Muslim fortress in the 10th–11th centuries, later expanded in the 12th century with the outer defensive wall that still encircles the hill at its base. Between 1460 and 1480, Luis de Tovar and María de Guzmán ordered the construction of the late-medieval seigneurial castle, designed both as a defensive stronghold and as a noble residence overlooking the old walled town. This inner castle has a rectangular ground plan with a circular tower at one corner, a prominent keep at the opposite side, and two interior courtyards, including a porticoed palatial patio with Gothic columns and a central cistern. In 1512 a new artillery fortress was begun, wrapping around the medieval core with massive stone walls up to five metres thick, angular bastions and casemates adapted to the hill’s steep topography and to the needs of gunpowder warfare. The wider monumental ensemble includes the ruined Renaissance palace of the Dukes of Frías and the 12th-century curtain wall, partially restored in recent conservation campaigns.
It is located in the old slaughterhouse and features pieces belonging to different churches in the town, as well as a series of explanatory panels, models and an audiovisual presentation to show the splendor lost in the Hermitage of San Baudelio due to the plundering of the paintings through a virtual reconstruction of what the hermitage would have looked like in its origins.