I have data from a load test of an automated system with several thousand data points that cover roughly 1 week of operation. I need to compare several algorithms for this system to see which is faster, if any (the metric used is the response time of the algorithm in milliseconds). Since it's technically impossible to store the response time of each operation of the algorithm, the simulator computes the mean, stdev and sample count of all the operations performed during the last minute, in order to reduce the amount of data generated.
Is it possible (or wise) to estimate the global mean/stdev based on the mean/stdev of each sample? Since each sample has a different sample count, I assume that some kind of weighting should be used for this estimation, but I'm not sure how to do this robustly, and I don't want to inadvertently alter the analysis.
A coworker suggested that computing the global quartiles (Q1, Q2, Q3) from the samples might be a more sensible approach to convey the general behavior of each algorithm. Which method should be preferred?
I'd like to ensure that the analysis is statistically robust and that I'm not seeing "things that aren't there".
I've found out that the global mean can be estimated by computing a weighted mean of the means of the samples and the global stdev can be obtained using the pooled variance estimate. Is this correct?
Nevertheless, I still don't know if it is possible to do the same with the median as I haven't found much literature regarding weighted medians. I still don't know which measure is more appropriate for my data.
I've learnt that Quartiles should be preferred to mean and standard deviation when data does not follow a normal distribution, although I don't know how to determine if my data follows it or not. I've found out some methods to compute the weighted median that I assume can be applied to any quartile but ¿does it make any sense to compute the weighted median from the collected means?