There's been a lot of recent work done in unsupervised feature learning for classification and there are a ton of older methods that also work well. The purpose of this competition is to find out which of these methods work best on relatively large-scale high dimensional learning tasks.
There's recently been a lot of work done in unsupervised feature learning for classification, with great advances made by approaches such as deep belief nets, graphical models, and transfer learning. Meanwhile, there are a ton of older methods that also work well, including matrix factorization, random projections, and clustering methods. The purpose of this competition is to find out which of these methods work the best, on relatively large-scale high dimensional learning tasks.
The Short Version
In this task, you'll do the following:
- Learn a feature representation of at most 100 features, using a small amount of labeled data and a large amount of unlabeled data. The orginial data is very sparse.
- Transform training and test data using your learned feature representation.
- Train a standard linear classifier on the transformed training data.
- Measure the AUC of the classifier on transformed test data.
We have public data to be used for the leaderboard evaluations, and a separate private data set used for final evaluation through a special submission process. We have scripts to simplify the training and evaluation; how you do the feature transformation is up to you.
The Long Version
The task that we're evaluating on is a binary-class classification problem, drawn from web classification. (The data has been cleaned heavily and anonymized.) The data itself is sparse, high dimensional data with about a million features. A few features have non-zero values in many or even all examples; most features have non-zero values in very few examples.
Your task is to transform the data from a high dimensional space to a lower dimensional space of at most 100 features. The goal is to make this new feature space so rich and informative that it allows a new classifier to be trained with the best possible predictive performance. Any method of producing a condensed representation is fair game: deep learning graphical models, transfer learning, supervised learning, semi-supervised learning, matrix factorization, random projection, clustering, feature selection, or anything else you can invent.
In addition to a small amount of labeled data, you will also be given a large amount of unlabeled data. Both the labeled and the unlabeled data can be used to learn good ways to transform the feature space.
The final evaluation will be done by using your method to transform training and test sets whose labels are not known. These transformed data sets will be sent to tne organizers, who will apply the hidden labels and use these new data sets to train and test a standard supervised classifier. The data set that produces the best classification performance on test data (using AUC as the evaluation measure) will be declared the winner. We also provide versions of our evaluation scripts to be used on public versions of our private evaluation data sets, and these results can be used to update the leaderboard. Please see the "Evaluation" page for full details.
Although there is a modest cash prize, the main goal of this competition is to encourage research and share ideas. The results of this competition will be included in a paper submitted to the 2011 NIPS workshop on deep learning and unsupervised feature learning. Contestants will be acknowledged by name in this paper for noteworthy performance, including results that do especially well or which are especially interesting.
Started: 4:01 am, Saturday 24 September 2011 UTC
Ended: 11:59 pm, Monday 17 October 2011 UTC (23 total days)
Points: this competition awarded standard ranking points
Tiers: this competition counted towards tiers