Lubed wads do swab the bore with each shot to prevent the barrel from getting fouled with powder residue. But each powder is different by producing a different amount and type of residue. I would never be able to fit 40-45 grains of APP fffg powder into the 1858 chamber if it wasn't first rammed and compressed before the ball was loaded. A hard wad does provide a flat, firm surface for when compressing it with the greater force of a loading press.
More powder compression can result in more velocity and compressing it separately from the ball in 2 stages reduces the amount of stress on the hand and on the gun's loading lever components too.
The variation of thickness and hardness of the different wad materials each offer some distinct benefits.