The Research and Advertising Sciences teams in Yahoo! Labs are working on a new generation of contextual advertising technology. Filling a critical Yahoo! need, the teams developed ad selection technology for direct and online marketers to place non-guaranteed advertisements on publisher sites. First launched in the US in April 2008, the resulting system, called "Keystone," was jointly developed with Yahoo! Engineering and Product Management.
In contextual advertising, the goal is to maximize a total utility function that captures the expected revenue, user experience and advertiser return on investment. The challenge is to select an optimized slate of eligible ads for an ad unit on each page view by a user.
The basic approach to contextual advertising has similarities to that of search advertising. However, the world of contextual advertising presents some unique challenges. User intent is unclear when the user is viewing a page, and semantic understanding of page content is a tough problem. Clicks are much sparser than in search advertising, and the position and size of ad slates introduce variable click bias. In addition, how do you design an ad search engine when a query has hundreds of attributes? How do you handle the huge volume of page view traffic with extremely varying quality?
"The biggest scientific challenge in contextual advertising is that compared to sponsored search or Web search, user intent is not very clear," says Dr. Jianchang (JC) Mao, Head of the Contextual & Display Advertising Sciences team, who leads the applied sciences Keystone effort. "We have to infer user interest from the page content and other context of the user."
The Keystone system works by first extracting "essence" from opportunity - understanding what the content is about and who is viewing it. Next the system searches for the ads with the highest relevance or click probability, making use of reverse indexing, response prediction, and information retrieval models. Keystone then filters the ads for eligibility (taking advertiser and publisher constraints into account), and finally selects the slate of ads to serve, optimizing among the eligible ads for maximum return.
A key difference between Keystone and other contextual advertising systems is that Keystone tries to predict and model user response based on all user context, including page content, user attributes like behavioral and geographical data, referrals to the page (how the user got there), and information about the publisher page.
Keystone was designed to provide global optimization via a top-down design: start with an objective, build a model that optimizes the objective, design an index structure that facilitates model evaluation, and design an index traversal algorithm that does efficient scoring. Keystone also uses automatic rule-induction that uncovers links between request attributes and offers attributes that may not match syntactically during training. And notably, Keystone scientists and developers leveraged a computing grid significantly for indexing and modeling, leading to fast turn-around of experiments.
In 2008, the Keystone team focused on getting the major infrastructure in place. In 2009, the focus has shifted to improving performance against key metrics. While Keystone has the potential to be a platform for other ad selection systems, such as search ads, the team sees plenty of challenges left in contextual advertising.
"We see a future opportunity for serving contextual ads to publisher pages," says Mao. "There are lots of interesting challenges and research problems because of the diversified user context including publisher pages."
"We want to be the model and incorporate all the contextual information and make intelligent ad selection decisions," adds Mao. "The user wants relevance, the publisher wants to maximize revenue and improve user experience, and the advertiser wants to maximize return on investment. Our top-down objective is to help all the major participants optimize their utilities in the ecosystem."