In 1868, Judge Edwin B. Crocker purchased the property and existing buildings on the corner of Third and O Streets. He then commissioned Seth Babson (1830-1908), a talented local architect, to redesign and renovate the home into a grander, Italianate mansion. In addition, Crocker asked Babson to design an elaborate gallery building that would sit adjacent to the mansion and display the family’s growing art collection.

Babson saw the home and gallery as an integrated complex, unique in design and demanding the finest materials. The gallery building included a bowling alley, skating rink and billiards room on the ground floor; a natural history museum and a library on the first floor; and gallery space on the second floor. Completed in 1872, the Crocker family mansion and art gallery are considered the masterpieces of Babson's career.

Today, the art gallery building retains its original Victorian Italianate design and serves as the main entrance to the Museum. The family mansion went through several uses and reconstructions until a 1989 renovation restored the historic façade and created a modern gallery interior. The original buildings, now connected, as well as the Herold Wing addition of 1969, were renamed the Crocker Art Museum in 1978.