 several large-scale observational databases and AEs.
In particular, we show results from two medical claims databases and 
acute liver injury, acute renal failure, bleeding and upper gestrointestinal tract ucler hospitalization events
in order to provide an examplar range of dataset sizes.
\myedit{Disclaimer}{
Our results explore the effects of optimization and parallelization and are not meant here to identify medical products associated with these events.
}
The MarketScan\texttrademark\ Commercial Claims and Encounters (CCAE) Research Database from Thomson Reuters is a large administrative claims database containing 59 million privately insured lives and provides patient-level deidentified data from inpatient and outpatient visits and pharmacy claims of multiple large employers.  
The MarketScan Lab Database (MSLR) contains 1.5 million persons representing a largely privately-insured population, with administrative claims from inpatient, outpatient, and pharmacy services supplemented by laboratory results.
These databases constitute part of the data community established within the 
the Observational Medical Outcomes Partnership (OMOP).  
OMOP is a public-private partnership between government, industry and academia to conduct methodological research to inform the appropriate use of observational healthcare data for active medical product surveillance.

\par

These example datasets span $N = 115\mbox{K}$ to $3.6\mbox{M}$ cases-only patients taking $J = 1224$ to $1428$ different drugs. The datasets provide $K = 3.8\mbox{M}$ to $75\mbox{M}$ total (unique) exposure eras per analysis. 
We perform all benchmarking on the Amazon Elastic Compute Cloud, exploiting an Intel Xeon X5570 CPU @ 2.93GHz and one NVIDIA Tesla C2050. 
This GPU device sports 448 cores @ 1.15GHz.
\myedit{Amazon}{
Performance on less expensive, commodity-grade GPUs, such as the NVIDIA GTX580, is often greater due to a slightly larger number of cores per GPU and higher memory-bandwidth.  Due to data licensing agreements, however, we are restricted to Amazon hardware.
}

\par
\begin{figure}[htb]
\centerline{\includegraphics[width=0.5\textwidth]{speed-ups}}
\caption{Maximum \textit{a posteriori} estimation for several observational databases under the Bayesian self-controlled cases series model.
We provide run-times for three implementations: dense computation on the CPU (white circles), sparse computation on the CPU (black circles) and sparse computation on the GPU (black squares).
}
\label{speed-up-figure}
\end{figure}
Figure \ref{speed-up-figure} presents the relative speed-up our algorithms enjoy when inferring MAP estimates.
These gains first compare implementing Steps \ref{exp-step} - \ref{numerator-step} as sparse operations and then porting computing to the GPU.
Sparsity generates up to a 181-fold speed-up; while the GPU multiplies this by up to another 37-fold.
To put these times on an absolute scale, MAP estimation for our largest dataset originally drained over 51 hours; sparse operations on the GPU reduce this time to 29 seconds.
Naturally, with fitting times standing in the 10s of hours, the hopes for cross-validation or bootstrap remain low, but grow very practical at 10s of seconds per replicate.



\par
\begin{figure}[htb]
\centerline{\includegraphics[width=0.75\textwidth]{bs-101}}
\caption{Exemplar uncertainty analysis of angioedema as an adverse event under the $L_1$ prior.  
Here, we plot the non-parametric bootstrap 95\% confidence intervals for the $441$ drug effects that demonstrated non-zero coefficients in at least 50\% of the bootstrap replicates. Gray-scaling reports the proportion of bootstrap replicates in which effect estimates are non-zero.
}
\label{boot-strap-figure}
\end{figure}
Cross-validation to learn the hyperparameter $\sigma^2$ across these four datasets returns optimal variances ranging from $0.05$ to $0.15$ for $L_1$ and $0.02$ to $0.13$ for $L_2$.
Importantly, these ranges are approximately an order-of-magnitude smaller than the arbitrary fixed value previously assumed in our BSCCS studies for drug surveillance.
Employing the optimal hyperparameter, Figure \ref{boot-strap-figure} reports non-parameteric bootstrap confidence intervals of drug effects for a single representative dataset under the $L_1$ prior.
This dataset explores angioedema events within the CCAE database and contains $N = 76\mbox{K}$ case-only patients, taking $J = 1162$ drugs and yielding $K = 2.1\mbox{M}$ exposure eras.
In the figure, we first rank all drugs by their MAP estimate $\hat{\beta}_j$ in decreasing order and then plot the 95\% confidence intervals for the 441 drugs for which $\hat{p}_{j} > 0.50$.
Darker interval shading reflects larger $\hat{p}_{j}$.
While a general trend holds in which larger $| \hat{\beta}_j |$ more often return 95\% confidence intervals that do not cover $0$, we identify notable exceptions.   
Namely, Drotrecogin alfa, an anti-thrombotic, anti-inflammatory agent used in the treatment of severe sepsis, returns with the fourth largest effect estimate, but its confidence interval continues to cover $0$, reflecting the high sampling variability in this estimate.


\section{Discussion}

Efficient algorithmic design and massive parallelization open the door for fitting complex GLMs to massive datasets.  
Computational statisticians regularly capitalize on the sparsity of their model and data, and this is an important design issue for the BSCCS we consider here, since the design matrix ${\bf X}$ consists purely of sparse covariates.
In particular, we identify that the sparsity of ${\bf X}$ carries all the way through to computing the subject-specific contributions to the model gradient and Hessian, resulting in over a $100$-fold speed-up compared to the most advanced CCD algorithms for GLMs of which we are aware.
Many GLMs, however, command dense covariates as well, such as baseline measurements, and other techniques become necessary.
Here fusing multiple transformations and reduction together into vectorizable kernels is the first step in off-loading the work to the GPU, and we hope our discussion in this paper raises awareness of these techniques among computational statisticians.
The end result for the BSCCS model is an approximate $30$-fold speed-up on a single GPU compared to a single CPU core.
These techniques also port directly to utilizing multi-core CPUs and multiple GPUs simultaneously, although we do not explore this avenue in this paper to simplify comparisons.

Advancing model complexity is both possible in the sampling density of the data and in the prior assumptions on the unknown model parameter.
Here we have only considered independent and identically distributed prior densities over the drug effect sizes.
More biologically plausible hierarchical distributions are conveniently available.  
For example, to borrow strength, we may favor grouping drugs \textit{a priori} into classes based on mode of action or therapeutic targets.
Similarly, we may wish to explore borrowing strength across related outcomes.
Because computation of the prior gradient and Hessian remains extremely light-weight, no modification to the GPU code is necessary and run-times should remain as quick.

\par
\myedit{RewriteSentence}{
One immediate advantage of the orders-of-magnitude reduction in run-time stands the ability to nest point-estimation within both cross-validation and bootstrap frameworks, making these Monte Carlo frameworks feasible.
Cross-validation and bootstrapping begin to allow us to estimate model hyperparameters and report measures of uncertainty around the usual point-estimates.
}
For the drug surveillance community, this represents a giant leap forward.
For example, most of the statistial methods in OMOP are implemented in the statistical packages {\sf SAS} or {\sf R} (\url{http://omop.fnih.org/MethodsLibrary}).
\myedit{NaiveTools}{
One of our own preliminary implementations of the BSCCS model in {\sf R}, using just a sparse matrix package and no further linear algebra libraries, requires around 5.3 hours to generate a single MAP estimate from a dataset with only $N = 7460$.}
With this benchmark in mind, it is no wonder why almost all computationally expensive fitting of massive datasets in the field has ignored cross-validation and bootstrapping; see, e.g., \citet{funk2011doubly}, and
a presentation at the 2011 International Congress on Pharmacoepidemiology involving a similar study redoubled this point by claiming that bootstrapping is computationally infeasible with more than $20\mbox{K}$ patients.
High performance statistical computing involving massive parallelization shows that these limitations are quickly lifting.

\par

We achieve this success by
exploiting the GPU within a serial CCD algorithm. 
CCD is a generic optimization approach and we envision extensions working for massive dataset applied to models beyond the GLM setting as well.
\myedit{OtherUses}{
Moving past CCD, \cite{zhou2010graphics} consider similar block-relaxation and majorization techniques to attack large-scale matrix factorization and multidimensional scaling using GPUs.  Here and