Full Results Download
Interested in the full results for all extracted factors including p-values and other evaluation metrics as described in the methodology?
File size 78KB
Raw Data Download
Want to do your own analysis of our data? Feel free to download.
Warning: The file is relatively large and needs special software to open. The file provided below is a 1% sample of the full dataset. After analyzing the 1% file if you would like the full set, please email email@example.com.
File size 815KB
As SEOs, we are interested in the most influential factors in commercial web search results. Accordingly, we have been conducting ongoing studies looking at the relationship between web search results and link metrics/anchor text from Mozscape, social media signals from Facebook and Twitter, and on-page/URL/domain keyword factors.
This document explains our methods, including the construction of the data set and statistical analysis. It is structured as follows: the following section includes some details on the dataset itself, from the choice of the keyword list to data sources and feature extraction.The last section describes our statistical analysis methods.
Before diving into the details, we want to mention several important things about the analysis. We only considered English language, U.S. search results from Google’s search engine, although we plan to extend the analysis in the future. All data collection was conducted during March 2011, after Google’s Farmer/Panda update.
The first step in building the dataset is selecting a list of keyword queries. Since this list of queries determines the composition of the dataset, it is important to ensure that a wide variety of subjects and query types are included. To this end, we used the suggested queries from all 15 top-level categories in the Google Adwords tool (Table 1 gives a list of the keyword categories). The tool provides 800 queries for each category, for a total of 12,000 queries. After removing duplicates from the list (some of the queries are included in more then one category) we obtained the final list of 10,980 queries.
The final list of keywords contains samples from the head, middle and tail queries, as measured by search volume. Table 2 lists counts of the number of queries in different local search volume buckets. All search volumes are well represented, from infrequent (less then 1000 searches / month) to frequent (more then 20,000 searches / month, and the data contains some keywords with more then one million searches / month).
We pulled the top 30 search results for each of the 10,980 queries on the query list from Google’s U.S. search engine, in a location and personalization agnostic manner. We removed all non-web results (images, video, news, etc) from the response, as well as results that were promoted to higher positions as a result of retrieving 30 results instead of just the first page of results. Finally, we excluded all queries that returned less then 15 results to ensure that each SERP had sufficient data points for analysis. This left a total of 10,271 queries/SERPs, 279,480 unique query – URL combinations and 223,737 unique URLs.
With the query – URL list from the second step, the final step in building the dataset is to calculate ranking factors. We collected factors from a variety of sources, as follows.
i. Linkscape URL metrics. All of the link related factors were sourced from Mozscape, using the url-metrics API call.
ii. Mozscape anchor text. For each URL, we pulled the top 1000 anchor text terms and phrases using the Mozscape anchor text API call. Then, for each query-URL combination, we determined whether there was a partial and/or exact match to the query. Here, “exact match” means that the entire query exactly matches the anchor text, while “partial match” means that at least one word from the query matches the anchor text.
iii. Social media signals. For each URL, we obtained a variety of social media signals from Facebook, Google Buzz and Topsy (for Twitter) APIs.
iv. On page factors. We retrieved the original HTML/XML content for each URL, and, for each query-URL combination, computed various factors of interest such as the number of keyword matches in different page elements, the length of the document, etc.
v. Domain/URL factors. We also extracted a variety of factors related to the URL and domain such as whether the query matched the domain name, whether the domain contained any hyphens, etc.
The complete list of factors and a description of each can be found in the accompanying dataset with all results.
The main goal of our analysis is to order the factors from most influential to least influential, while providing some estimate of the relative influences across factor type (link vs. on-page vs. social, etc). To this end, we have computed a number of different evaluation metrics between search position and the ranking factors, as well as a measure of overall prominence of a factor in the SERPs.
Mean Spearman Correlations
This is our preferred metric, and the one illustrated elsewhere in this report. Since we have a wide variety of factors and factor distributions (many of which are not Gaussian), the Spearman correlation is preferred to the more familiar Pearson correlation (as Pearson correlation assumes the variables are Gaussian). In our analysis, we treated each query as independent and computed the Spearman correlation for each query, then averaged over all queries and reported the result.
In addition to the mean across all queries, we also computed the standard deviation across queries, the standard error of the mean and a measure of the statistical significance of the result. The standard error is computed using the familiar result N/σ0.5 where σ is the standard deviation and N the sample size1. To compute a measure of the statistical significance, we reasoned as follows. First, we computed the significance of the Spearman correlation for each query using a t-test. Then we ran a binomial test on the total number of significant results across all queries, and reported its p-value.
Precision and NDCG
In addition to mean Spearman correlations, we also computed a number of metrics used in the Information Retrieval community to evaluate ranked retrieval results. We computed the precision at 1, 5, and 10 and the mean average precision (MAP). We also computed the normalized discounted cumulative gain (NDCG) also at 1, 5, and 10.
Both the various precision and NDCG metrics are well correlated with the mean Spearman correlation (Table 3, correlations ≥ 0.90, p < 0.0001), so there is significant agreement among the relative ranks of the different factors.
In addition to Spearman correlations, precision and NDCG, we also calculated the overall prominence of each factor in the results. We chose to measure prominence as the percentage of results in the entire dataset that contain the factor, where “contain” generally means the factor is not zero (the exact definition depends on the factor). For example, the prominence of keyword matches in the title is calculated as the percentage of results that include at least one keyword match in the title, the prominence of Facebook shares is the percentage of results that have at least one Facebook share, the prominence of number of linking pages is the percentage of results with at least one link, etc.