The damping properties of concrete members are closely related to concrete aggregate characteristics. Replacing natural sand and crushed stone with different recycled aggregates (RA), the first damping ratio of suspension beam of recycled aggregate concrete (RAC) in elastic stage was tested by using the free vibration attenuation method, and the relationship between damping ratio and concrete strength/elastic modulus was analyzed. Moreover, the RAC pore structure and interfacial transition zone (ITZ) characteristics were investigated by using the nitrogen adsorption method and scanning electron microscopy (SEM), and the damping mechanism was discussed. The results show that the damping ratio of RAC beams increases significantly, when compared with that of natural aggregate concrete (NAC) beams, and the damping ratio increases with the decrease of concrete strength and elastic modulus. The effect of aggregate property, especially the aggregate surface modification, on the concrete damping property is noteworthy. The total pore volume and average pore diameter of RAC are larger than those of NAC, and the ITZs in RAC are weaker. Therefore, the sliding and friction in ITZs and the flexible cushion action of pores result in more energy dissipation in RAC beams.