I am experiencing a strange problem with varying the test set size.
This is mildly confusing to explain, but I'll do my best. I'm using octave to train an SVM on timeseries-like data and it's generally yielding good results.
However, occasionally when we've introduced "bad" features, the SVM will "give up" almost instantly during training and produce a model that effectively pins predictions on a single feature. In these cases, the train completes in ~1 second vs the normal 4-5 minutes. I've never really known what causes it, but I just presumed it couldn't find a useful model, which seemed reasonable.
The problem I'm facing now is that it appears something as simple as normalization can be the difference between a very effective model and a "give up instantly" result.
The specific case is this:
I have a dataset of say 5000 training examples. They are split into a 90/10 split for train/test and work well.
Similarly, if I split the data set 95/5, that also works fine (better, in fact for test performance, simply because it's smaller).
However, weirdly, if I split the dataset 90/5/5 and throw out the last 5%, the machine refuses to train. So all I'm doing is removing some of the test set, which is causing the entire process to fail.
To make things even more confusing, I found that normalization is the cause: If I normalize (zero-mean + scaling) then slice, everything works fine, however, if I slice, then normalize, the machine fails.
Basically, my conclusion is that the exclusion of certain training examples in the test set is causing normalization to (naturally) come up with differently values scaled that are for some reason problematic.
I have eyeballed the min/max/mean of the scaled values and none of them vary by more than a few percent, so I'm somewhat at a loss of what to pursue next.
Any guidance of how to go about diagnosing this or what may be happening would be most appreciated.
After some further work on this, I've realised that the machine "gives up" instantly if there is a feature with all values being zero (this can happen either on certain subsets of training data or just bad luck).
Although I admit a feature of all zeroes is of little value, is there a specific reason why this would cause the algorithm to behave as described?
Incidentally, we're using a Gaussian kernel if it makes a difference.