For static fields (those that don't change in time), one could perhaps make a case that all the energy due to a field could be classified as "potential energy". However, when one deals with general fields, it's surely not the case that all the energy due to a field is "potential energy". Some of it is also "kinetic energy" – a term that makes sense both for particles as well as fields. For example, electromagnetic waves carry energy but they're not just "potential energy". The waves may exist independently of the charged sources.
In simplified and approximate models, potential energy may exist even in the absence of a field. That's how it worked in Newton's physics. However, such a potential energy implies that the action at a distance exists. Albert Einstein discovered special relativity in 1905 and it implies that signals and information can't propagate faster than the speed of light. It means that the forces sparked by changes of the potential energy can't act immediately. They must act with a delay and a field is always needed for such a delay. So once special relativity is taken into account, the answer is Yes, potential energy always has to be due to some fields.
Quite generally, I think that you should appreciate that there may exist many forms of energy. The energy is the most general formula for a conserved quantity that typically contains many terms. These terms may depend on the positions of particles, velocities of particles, values of the fields, and the rate of change of fields as functions of spatial directions as well as time. The term "potential energy" is usually reserved for terms in energy that don't depend on derivatives. However, the potential energy between two electric charges, for example, boils down to the $E^2/2$ term in the energy density of the electric field and here, $E$ contains $\partial A/\partial t$, so it's a "kinetic term for the field $\vec A$", anyway. In general, one must be careful about the classification of terms in the energy to potential and others (e.g. kinetic). It's always more accurate to mathematically express what particular term or what class of terms we are talking about.