I am trying to assess the performance of my Bayesian classifier. One measure that I calculate is the false discovery rate (FDR):
FP / (FP + TP), where FP = False Positive and TP = True Positive.
The ratio (and sizes) of my positive and negative training sets do not accurately reflect the estimated 'real' ratios (and sizes). To clarify, I am working with sets of genes that have a certain property. It is estimated that there are approximately 18000 negative (not having that property) and 400 positive (having that property) genes (ratio 45:1). However, I only have 6000 negative and 200 positive examples (ratio 30:1) for which this property has been confirmed (my training set).
I've read that if the sizes of the training sets do not reflect the prior odds, the FP an TP rate must be scaled in order to avoid over- or underestimating the false discovery rate.
My questions are:
- Do I have to scale/correct the FP and TP? If so, is this for the reason stated above (underestimation of the FDR) or some other reason?
- If scaling is necessary, how can I achieve it?
- Other performance measures, such as specificity and sensitivity, also use FP and TP (and TN&FN). Should I use corrected FP TP TN FN for these too?
A more general question:
I believe that what I refer to above as "estimated 'real' ratios (and sizes)" of the positive and negative sets are the prior odds. Is that correct?
I am not a statistician and I have a hard time trying to find good resources on the subject. This is mainly because false discovery rate seems to be primarily associated with Benjamini-Hochberg multiple testing correction. I'm not sure if the two applications of false discovery rate are in fact the same..
Any help would be greatly appreciated.