The original settlement, tied to the complex of the Villa at Castello, known also as Villa dell’Olmo or Villa Reale, took its origin from the presence the Roman aqueduct of Valdimarina, situated between Sesto Fiorentino and Florence. The very name of the place derives from castellum, the meaning of which was that of water tank or cistern. In 1477, Lorenzo and Giovanni di Pier Francesco de’ Medici purchased a rural construction built around a thirteenth-century tower, which they enlarged and embellished with works of art, including the Birth of Venus and the Primavera, commissioned by Lorenzo from Botticelli, and transferred to the Uffizi Gallery in the 19th century.

Following the expulsion of the Medicis from Florence and the serious damage it suffered, the complex was targeted with major renovation by Cosimo I who, around 1538, entrusted Niccolò Pericoli, known as Tribolo, with a project for the garden, realised on the basis of a complex allegorical programme, attributed to Benedetto Varchi and to Tribolo. The programme provided for a system of allegories which were supposed to exalt Medici power by means of several sculptural groups portraying mountains and cities of the Tuscan State, all connected by a waterway, representing the rivers that crossed them. Personifications and allegories of the virtues, in support of Medici power, were incarnated in statues placed in niches along the side-walls, including the Four Seasons and the portrayals of Generosity, Wisdom, Nobility, Value, Compassion, Justice, the Arts, Languages, Sciences, Arms, Peace, Laws. The garden is a stupendous example of Italian parterre. The exotic plants in the garden were described by French botanist and zoologist Pierre Belon who visited the villa during his journey between 1546 and 1549. The particularity of the plants and lanes of the garden and woods, was described by Montaigne who sojourned in the villa during the second half of the 16th century.

Particularly fascinating is the Animal Grotto, composed of two chambers with two fountains in white marble and one in peach-coloured marble from Seravezza, surmounted by sculptural groups portraying a suggestive array of animals from whence the grotto takes its name. Spectacular giochi d’acqua are created by a series of spouts on the floor of the grotto, supplied by a complex hydraulic system. On the sides of this space are two secret gardens which recall the medieval garden: in the one towards the east, called "Ortaccio", a pavilion was built on a large tree where a marble table with a musical fountain were placed; the second one, planted in the 17th century, was called "Stufa dei mugherini", and contained the collection of jasmine begun by Cosimo III. The other secret garden, connected to the kitchen and called "dell’imbrecciato" because of its flooring in polychrome gravel, is thought to have contained a statue of Asclepius, god of medicine and of herbs. To supply the complex water system of the numerous fountains, Piero da San Casciano built the Castellina aqueduct which conveyed water from a nearby spring.

In the Napoleonic era, an ice house was built at the centre of the garden and the statue of Florence was replaced with that of Hercules and Antaeus by Bartolomeo Ammannati. With the return of the Lorraines, Bohemian architect Joseph Frietsch created the spacious English-style woods which were to connect the villas of Castello and Petraia. Today, the Villa of Castello is the headquarters of the Accademia della Crusca.