A private API or undocumented method is any object or method that is not part of the official documentation. In Objective C, and some other languages, it is relatively easy to find the list of methods (messages) that an object supports as well as the objects underlying the framework. (For example, just go into the debugger and look at the view hierarchy. You will likely see several view objects that don't exist in the documentation.) Sometimes you will even see instructions on how to use these methods and objects on the web.
One example that leaps to mind is
-UIWindow _autolayoutTrace, I use it all the time when debugging autolayout, but it isn't documented and the leading underscore is a hint that you shouldn't be using it. That's fine for debugging, but if I shipped code that used that method it would be certain to be rejected.
But Apple specifically scans for these undocumented methods as part of the App Store review process and rejects apps that use them. This is because Apple might change how these undocumented methods work at any time. If your app was dependent on one of these hidden classes or methods your app might break when Apple released a new version of the SDK that changed this behavior.