How to use your amp's effects loop, and pedals vs. rackmount:
The short and simplified version is that pedals are meant to go between your instrument and the main input of your amp, while rackmount signal processors are meant to go in an effects loop, or between a preamp and a power amp. This is because they operate at different ranges of signal level (amplitude, strength), and have different impedances.
The higher amplitude range is called "line level", and the lower range is called "instrument level". At even lower amplitude there's "mic level", which is the expected output of a typical microphone or a DI box, both of which are intended to plug into a mic preamp with a large amount of gain. Ideally you want the devices you are connecting to match up in their ideal range of input and output levels, otherwise you may get distortion, noise, or a weak signal. Effects loops often (but not always) run at line level.
Impedance ("z") describes the efficiency of signal transfer between any one device's output connected to any other device's input. An inefficient relationship will mean loss of signal strength and loss of tone. You want the input impedance to be much higher (at least 10x, but more is better) than the output impedance of whatever is plugging into it. Effects loops typically have low output impedance (from the send jack), but whether their return jack has low or high input impedance will depend on the specific amp--there is no standard. With pedals, too, there is no standard. It's usually not important to know the in/out impedance of your pedals, but if you run into a situation where a pedal sounds bad in one connection and not another, very often some impedance problem is the reason.
Pedals generally operate best with an input signal directly from your guitar/bass or other pedals, and their output is meant to be fed into the main instrument input of an amp head, combo amp, or rack preamp. Those types of preamp stage boost the instrument's signal strength and lower the overall output impedance, which is what is required for driving a power amp, line-level processor, or many effects loops. Without that, most pedals don't have strong enough output to do the job. Of course there are many exceptions, such as certain preamps in pedal format, and you have to examine those on a case-by-case basis.
Rackmount signal processors are almost always designed to run at line level, so they are suited for use with effects loops. Their in/out impedance is also usually a good match. The loop is a direct connection between the preamp and power section of your amp, so you can think of the loop return jack as a power amp input; see that "preamp" article linked in the first paragraph for technical details about driving power amps. The signal from most guitars/basses is too low, and the impedance is too high, to go directly into most rack processors. Technically you can plug your instrument straight in, but it will not work nearly as well as if you put a preamp between them, and it may not work at all depending on the particular gear.
There are exceptions: Some rack units have a "Hi Z" input, which is designed for you to be able to plug your instrument straight in; and some rack processors (especially older guitar-oriented units) are designed to operate at instrument level, or at least have enough input gain available to be used that way. There are also some effects loops which are able to operate fine with pedals in the loop- you'll need to check the manual for your specific amp, or just experiment. And finally, some pedals can handle a line-level signal. One fairly common example would be a loop that operates at -10dB, paired with an fx pedal designed to work well with high-output basses; that combination can often work out fine.
Really the best way to see what works is to experiment. You won't hurt any of your gear, don't worry. When experimenting, listen carefully and ask yourself these questions: Do you hear any distortion? Does the signal seem weak? Does a dynamic effect seem to react too strongly, or not enough? How bad is the hiss, and does the hiss change in level or "quality" when you switch positions? With a compressor, does it have meters to indicate signal levels or compression amount, and do those meters seem to read the same or differently in each position? If you hear distortion, noise, weak signal, or "tone suck", those are your signs to adjust the levels or change the placement of the various pieces of gear. Again, you won't damage either the amp or the effects in this process.
Another set of factors to consider:
-Pedals almost always have 1/4" unbalanced (AKA regular instrument cable) inputs and outputs;
-Rack processors may have either unbalanced or balanced (AKA XLR, mic cable, 1/4" TRS/stereo) in/outputs;
-The jacks for connecting a preamp to a power amp may be balanced or unbalanced;
-And an amp head's effects loop is typically unbalanced.
Note that 1/4" balanced and unbalanced jacks look the same from the outside. So you'll need to read the manuals of your specific gear to identify which types of plugs and cables you need. Again, experimentation is fine; the only thing you cannot do is place your compressor or other fx after the output of a power amp, as that would kill your gear.
Effects loops can be "series" or "parallel". Series means one device leads straight into the next, with no signal splitting. Generally speaking, a serial connection is best for a compressor (although there are esoteric exceptions). Parallel means the signal from your preamp is split into a "clean" channel and an "effected" channel, and then those two are blended back together. Parallel loops typically have a "blend" knob (wet/dry). If your fx loop is parallel, you may occasionally find that the wet and dry signals interfere with each other, causing spikes or dropouts of signal level at different frequencies. A loss of lows is common. If that happens, you can solve it by either setting the blend knob to 100% wet, or using a specialized device to adjust the phase of one of the signals. Bear in mind though that a few parallel fx loops cannot be set to 100% wet! There are some where the maximum setting of the blend knob is only a 50/50 blend.
Guitarists will frequently insist that certain effects (e.g. chorus, flanger, wah) belong in front of the amp, while certain others (e.g. delay or reverb) belong in the fx loop. This is because they are accustomed to using their amp as a distortion effect, and they like the distortion at a particular place in the chain. If you are not using your amp as a distortion effect, you can ignore those claims. Plus that approach does not address the question of instrument-level versus line-level.
Rackmount gear is not necessarily better quality than pedals! There are good and bad units in either format. That said, it's true that more of the "high end, high quality" market is focused on rack units, and most pedals are designed for low cost rather than high quality. So statistically rack gear tends to be better, but just don't get caught up in assuming it's always that way. I can't stand it when people post "pedals suck, they are toys compared to the Real Thing" because in the last few years the pedal market has seen some incredible technological and design developments, and at the same time the rack gear market has been aiming more and more for the budget consumer. Things change.