Built in the 15th century, the palazzo originally had a small flower garden and a large vegetable garden. According to Giorgio Vasari, the palazzo was remodelled on a project by Raffaello Sanzio around 1514 for bishop Giannozzo Pandolfini. The garden was embellished by statues, fountains, waterworks and numerous flower and plant varieties. This was an Italian garden, divided into two sectors. The largest, divided into four beds, had a small man-made hill, surrounded by flowers, citrus trees and a trammel-net of holm-oaks and laurel. The smaller sector facing the via San Gallo, instead, had a fountain in the middle, surrounded by geometric flower-beds and then by grapevines and fruit trees.

During the first half of the 19th century, Eleonora Pandolfini gave the garden an English layout and had a greenhouse built to shelter ornamental plants in winter. From 1870 to 1885, Alessio Pandolfini remodelled the palazzo, while his wife Sofronia Stibbert restored the original splendour to the garden, which became famous for its fine collections of camellias and cinerarias. In the late 19th century, several botanical rarities cultivated by Sofronia Stibbert were awarded prizes by the Società Botanica dell'orticultura [Botanical Society of Horticulture].

Further restoration was conducted in the course of the 20th century; the last modification was the construction of a greenhouse for the orchids.