Well, dairy isn't strict, orthodox Paleo, but it is in a Primal "grey area." Generally, the advice is to do 30 days without it, see how you feel, and treat dairy accordingly.
There are a couple issues with dairy: lactose intolerance, casein intolerance, insulin response, and toxins.
If you are lactose or casein intolerant, no you shouldn't drink milk. However, the higher fat dairy products (cream, creme fraiche, butter, etc) are very low in these compounds and may be well tolerated even if milk isn't. This is made even more complicated by the fact that in many people, lactose intolerance clears up once gluten is removed from the diet.
Regarding the insulin, keep in mind that while some people do Primal as a VLC diet, others have higher carb needs, especially if they are very active. In particular, milk's insulin spiking properties can be very useful if you're working on adding muscle.
As for toxins, that is why it is so critical to buy organic dairy, especially from pastured cows.
On the other hand, dairy (especially raw butter, cream, or ghee from grass-fed cows) has some unique nutritional benefits, like vitamin K and CLA. And cultured dairy products like traditionally made yogurt and kefir are valuable sources of probiotic organisms.
Personally, I eat butter and ghee daily, a couple ounces of hard cheese a week, and drink kefir or yogurt as needed to keep my digestion functioning properly - some weeks that means every day, other weeks it means that I don't have any.