To simplify the Tools palette in QuarkXPress 8 and above, Quark rethought how the tools work together. So, for the “box shapes” they simplified your choices to a rectangle, circle and starburst. To make a rounded-corner rectangle, create a rectangle and then change its corner style in the Measurements palette.
Try this: create a rectangle box, then look in the Classic tab of the Measurements palette. There, you’ll see a “Box Corner Radius” control. This gives you all the corner shapes you had before: rounded corner, concave, and beveled.
In QuarkXPress 9, you can also use the new ShapeMaker feature (Utilities> ShapeMaker) to create a box where each corner is shaped differently: