The palazzo with adjoining garden was built in the late 15th century on property belonging to Nannina de’ Medici, sister of Lorenzo the Magnificent, and her husband Bernardo Rucellai. The complex was a very important cultural centre as it hosted, after the Medici’s expulsion, the meetings of the Platonic Academy formerly held at the Villa of Careggi where Rucellai too, had also participated. The name of the garden (also known as selva) comes from that of the Rucellai family, an ancestor of which introduced ruscus to Florence, a grass used to dye wool violet and called Oricella or Lichen Roccella in Linnaeus’ classification.

Purchased by Bianca Cappello in 1573, the Garden was restored its original beauty, becoming a place of delights and leisure, abounding in entertainment and games organised by the grand duke and duchess for their guests. In the mid 17th century, cardinal Giovan Carlo de’ Medici became the promoter of the first major series of works, creating an Italian-style garden. Inspired by the work of Bernardo Buontalenti at Pratolino, it was planned to convey water from Boboli to the garden, exploiting the channelling system along the via Maggio and Ponte Santa Trinita, in order to supply the fountains designed by sculptor Antonio Novelli. The most grandiose of the fountains is the square-plan fountain portraying Polyphemus drinking from a water-bag, made of plastered masonry with an iron framework, the same technique used by Giambologna for the statue of the Apennines at Pratolino. Novelli also did the Garden Grotto: dug into a manmade hill, it presents the likeness of a cave decorated with sponges and embellished by a series of statues representing the winds.

Giuseppe Stiozzi Ridolfi, the new owner in the mid 19th century, transformed the garden, entrusting the project to Luigi de Cambray Digny who evocatively interpreted the theoretical dictates of the romantic garden. Characterised by a central axis culminating at the temple of Flora, and by the presence of winding footpaths, hillocks, ponds, statues, and artificial ruins, the new arrangement adhered to a precise programme that, in the use of symbolic elements, underground tracts and inscriptions, aimed to create an initiatory route that led to the Pantheon, intended to preserve the memory of illustrious exempla virtutis. In 1832, Emilio Burci published a series of engravings depicting the Oricellari Garden, with views of the Abbey of Sant’Anna, the temple of Venus, the remains of an ancient temple and the circus, the colossus of Polyphemus and the grotto, the "flower garden", fortress, "turret, and the Pantheon.

In 1861, instructed by the new owners to give the garden a different organisation, Giuseppe Poggi proposed a solution in the classical style which provided for restoring the basin with the statue of Polyphemus. In the course of works to make Florence capital of Italy, the garden was divided into two parts by the new via Rucellai. Other divisions and alterations followed in time, but the physiognomy authored by Poggi has been maintained. In particular, it is still possible to admire the Polyphemus, the Garden Grotto (the part that is property of the Cassa di Risparmio di Pisa) and the fortress with the turret.