I am conducting a CFA of a particular questionnaire in a sample of offenders. There is theoretical and empirical evidence (most studies have found this factor structure both via EFA and CFA) for a four-factor structure. However, one study did obtain a two-factor structure as best fit (which corresponds to another theoretical stream). Since CFA is meant as a confirmatory procedure, and experts tend to advice against using it unless one has sufficient evidence for the proposed model, I was going to specify four factors and test that model. However, others now suggest I should test both models, and present the one with best fit, using the same sample for both tests. Is this acceptable usage of CFA? If not, whY?
The p-values are not right if you do a series of tests where the second test depends on how the first test came out, etc. But since you have two tests in mind beforehand, this will be fine. I would also compare the models using BIC.