It was built in 1538 by Bartolomeo Panciatichi who incorporated at least three old buildings, including the house of Rinaldo degli Albizi, rival of Cosimo the Elder. Towards the mid 16th century, senator Baccio Valori whose family had entered into possession of the Albizi building, entrusted Giovanni Caccini with enlarging the building and renovating the facade. Valori laid out the iconographic programme of the facade’s decoration, which he wanted adorned with fifteen herms of illustrious figures of Florentine culture. The personages portrayed include Marsilius Ficinus, Amerigo Vespucci, Leon Battista Alberti, Francesco Guicciardini, Dante, Petrarch and Boccaccio. These portraits were called "Visacci", that is to say ugly faces, hence the name of the building. The Visacci were described in 1604 by Baccio’s son. Valori, librarian of the Biblioteca Medicea Laurenziana and president of the Academy of Drawing Arts, made Palazzo Altoviti one of the propulsive centres of cultural life in Florence.