Intel 80386 Reference Programmer's Manual is a good reference. (especially Part IV)
Here's a decent place to get started. I think it's GAS flavored rather than MASM, but they're similar enough that you shouldn't have any trouble.
Once you have enough of an understanding to recognize what's happening when you open the disassembler view in VS debugger, definitely use that as a learning aide. Write simple programs and watch how the compiler implements them. When you compile for debug (with optimization disabled) you should see easy-to-understand assembly. If you enable optimization you'll often see something very different.
It's not bad to learn assembly. In fact, it give you a lot of insight into how computers work, and that may influence the way you program in general. It's also very fun, for a certain type of people. Long story short, the benefits you get from learning assembly won't come so much from writing assembly, but more from understanding how computers work in general. Modern compilers are ridiculously smart, and they do the 'right thing', even in some really obscure situations. Moreover, optimization usually comes from changing the structure of a program rather than from changing the speed with which it executes some specific routine.
Just be aware of that and you should be fine.