The ensemble of real orthogonal matrices (uniformly distributed with respect to the Haar measure) is the socalled Circular Real Ensemble (CRE) of random-matrix theory. (Not the Circular Orthogonal Ensemble, COE, which confusingly enough contains symmetric complex unitary matrices.)
The probability distribution of the eigenvalues in the CRE is known, and you could use that to test whether your set of matrices is indeed drawn from that ensemble. (For example, by comparing moments of the eigenvalues or by comparing the spacing distribution.)
You can find the eigenvalue probability distribution in Section 2.9.2 of P.J. Forrester's book "Log-gases and Random matrices". You will have to distinguish the cases of determinant equal to +1 or -1, and even or odd N. The formulas are a bit lengthy, but if the book is not available to you, let me know and I will try to record them here.
Alternatively, you could just generate matrices from the CRE and compare the eigenvalue statistics of those matrices with your M matrices from an unknown ensemble. Generating matrices from the CRE is easy: They contain the eigenvectors of random real symmetric matrices with a Gaussian distribution of the matrix elements (the Gaussian Orthogonal Ensemble --- yes, here the word orthogonal is appropriate).