The castle was started in the sixth century to protect against the Goths. The first news we have of it was when Frederick II of Swabia, who stayed here, fortified it. The present structure is large and is called Angevin-Aragonese castle because it consists of two adjoining buildings constructed at two different historical moments. The lower one, called "Angevin" was realized during the French domination of Anjou, and the upper one called "Aragon" was built by Emperor Charles V. The Angevin wing was home to the military prison of Gaeta and is currently closed. The Aragonese wing was home to a battalion and today is home to the Nautical School of the Guardia di Finanza. In the dome of the highest tower of the castle there is the Royal Chapel, commissioned by King Ferdinand II of Bourbon in 1849.