Understanding the effects of neutron radiation on typical structures of a nuclear power plant is crucial to supporting the second license renewal of the US fleet of light-water reactors. The biological shields surrounding the reactor pressure vessels are constituted by concrete, which is about 70% aggregate in volume. Aggregates are sourced locally and are unique to each concrete formulation. The minerals present in aggregates sustain radiation-induced volumetric expansion. The swelling amplitude depends on the minerals’ composition and structure and is larger for framework silicates, followed by single chain silicates and carbonates. Differential expansion on different minerals that conform an aggregate is expected to result in stresses, pores, and cracks. The evolution of porosity from the nano- to the microscale was investigated via ultrasmall-angle x-ray scattering in six different aggregates—one altered tuff, four sandstones, and one limestone—from the pristine state to the irradiated state with neutron doses of 1.28 × 1019, 4.12 × 1019, and 8.25 × 1019 n/cm2. An increase in the total pore volume fraction at the highest neutron dose was observed for the altered tuff and the sandstones, whereas the limestone showed minimal differences in total pore volume fraction across the explored doses. For pore sizes ≤100 nm and ≤1 μm, the cumulative porosity change was linearly correlated with the aggregate experimental and estimated expansions.