Yes, if a Live USB is persistent then it will store all changes to programs and files. However, you should not use persistence as a replacement for a regular installation, only a temporary measure. Ubuntu's persistence slows down dramatically over time because of the architecture of the Live USB.
A brief primer on persistence and Live CD/DVD/USBs
Linux's Live CD feature is very efficient. Using a read-only filesystem called SquashFS, you can compress a ~4GB filesystem image down to around 700MB, which is small enough to fit on a CD. Recently, Canonical decided that, because of the rapidly declining usage of CDs, it wasn't worth it to go to the effort of making their CD (now DVD/USB) images fit on a CD, so they decided to target their images toward DVDs instead. This means that you can no longer fit a stock Ubuntu image onto a CD. However, that is no excuse for letting the (compressed) image balloon up, so Ubuntu (along with most other Linux distros) still uses SquashFS for compression. However, there is a drawback to getting ridiculously high compression rates: as mentioned above, SquashFS is read-only.
So what about persistence? How is persistence possible is Live USBs use a read-only filesystem? Well, what happens is that, when you exit a persistent session, Ubuntu saves all file changes to a persistence file. Then, the next time that the Live USB is booted up, Ubuntu first loads the "vanilla" Ubuntu installation. After that has completed, it then gets the persistence file and uses the data in it to re-modify the files back to where they were when you last left off.
This is a pretty clever mechanism, but is has a few drawbacks, one of them being that, as you use the system more and update some of the packages, that persistence file becomes larger and larger, which means longer and longer boot times, not to mention other nasty side effects.
So, the lesson is that, while persistence is a very handy feature for a Live USB to have, you should only use it when necessary, e.g., your main computer is down.