I have a question regarding PU-Learning, which deals with learning from positive-labeled (no labeled negative examples) and positive/negative-unlabeled data.
Particularly, my question is about the paper Learning classifiers from only Positive and Unlabeled data. This paper converts a non-traditional classifier $g(x)$ which is learned from labeled/unlabeled datasets and outputs the probability of an example of being labeled, into a traditional classifier $f(x)$ which outputs the probability of an example to be positive. I have implemented the first proposed approach (section 2, "learning a traditional classifier from nontraditional input"). However, the probability $f(x)=g(x)/p(s=1|y=1)$ becomes greater than 1 for several examples (this should not be greater than 1, since it is a probability)
For learning the non-traditional classifier $g(x)$, I am using a non-traditional dataset composed of
- 100,000 labeled examples randomly chosen from the whole labeled data, and
- 100,000 unlabeled examples randomly chosen from the whole unlabeled data.
And for estimating $p(s=1|y=1)$--with the first proposed estimator--I am using a validation dataset composed of 30,000 labeled instances randomly chosen from a separate labeled dataset.
According to the paper, the probability $f(x)$ is guaranteed to be well-formed if (1) and (2) overlap in the example space. In my case, (1) and (2) have a 20% of overlapped examples, but I am still getting non well-formed probabilities (>1).
How could I achieve a well-defined probability for $f(x)$?
According to the paper, $g(x)$ must be a classifier that produces correct probabilities as its output, like Logistic Regression, or a calibrated classifier such as Naive Bayes/SVM, in order to get the approach to work. I am simply using Maximum Entropy classifier (from the NLTK package), also known as Logistic Regression. Therefore I guess there should not be any problem with this.
Although the paper states that the labeled (1) and unlabeled (2) datasets for training $g(x)$ are "samples from overlapping regions in feature space", such datasets seem to be disjoint (see experiment from Section 5, P and U are disjoint). I have tried with disjoint labeled and unlabeled datasets as well, however I am still getting probabilities over one for $f(x)$.
Balancing datasets does not make any difference, neither.
I have tried the second estimator proposed(e2). Still getting probabilities over 1 for $f(x)$.
In section 5, which describes an example with real-world data, we have a set P of labeled+positive examples from the database TCDB, and a set U of unlabeled examples randomly sampled from the database SwissProt, being P and U disjoint. Then they use P,U to learn a non traditional classifier. I think that this approach can be applied to my problem, since I also have a set of positive examples, and a set of unlabeled examples randomly sampled. What do you think ?
I have tried the third estimator proposed(e3). Since it takes the maximum probability of the examples from the positive-labeled set (in my case around 0.98), I am no longer getting probabilities over one. However, since this estimate is only based in one example and not the average between all examples, therefore this does not look like a good estimator to me. Any thoughts regarding the validity of this estimator?