Recent content in Posts on Andrey Akinshin
In [[carling-outlier-detector]], I evaluated the probability of outlier detection for samples from the Normal distribution across different outlier detectors. I performed numerical simulations for small sample sizes, then confidently extrapolated the results to larger samples. As it turns out, this approach led to the wrong conclusion. This post corrects that error by extending the simulation to…
This post presents a fast O(n log n) algorithm for computing the Shamos estimator via Monahan-style selection. Pragmastat’s Spread implements this approach, making the estimator practical for large samples. Confidence intervals are available via SpreadBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat The Shamos scale estimator ([[shamos1976]]) is the median of all pairwise…
This post presents the Monahan algorithm for computing the Hodges–Lehmann estimator in O(n log n) time, making it practical for large samples. In Pragmastat, this estimator is Center — the recommended robust location estimator. Confidence intervals are available via CenterBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat The Hodges–Lehmann location estimator ([[hodges1963]]) is a robust…
Previous posts explored fast implementations of moving quantiles based on partitioning heaps: [[partitioning-heaps-quantile-estimator|Part 1]], [[partitioning-heaps-quantile-estimator2|Part 2]]. This post presents a cleaner implementation that is simpler to use. Compared to previous attempts, this version makes several important changes: Quantile-focused API : The approach estimates only the…
In this short case study, we explore a few more plots with Gaussian efficiency plots for the trimmed and winsorized mean estimators against the sample median.
This post explores the idea of iteratively mixing mean, median, and geometric mean, inspired by xkcd #2435. In Pragmastat, Center — the Hodges–Lehmann estimator — provides a principled robust location estimate with 95% asymptotic efficiency and a 29% breakdown point, without iteration. Confidence intervals are available via CenterBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat Let us…
This post evaluates the statistical efficiency of the tau measure of location — a weighted mean based on a robust scale estimate. In Pragmastat, Center — the Hodges–Lehmann estimator — achieves 95% asymptotic efficiency with a 29% breakdown point and is the recommended choice. Confidence intervals are available via CenterBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat For a sample…
We continue exploring various use cases of the [[lowland-multimodality-detection]]. In this post, we will consider a brief example of using weighted samples.
We continue exploring various corner cases for the [[lowland-multimodality-detection]]. In this post, we consider an example that illustrates the usefulness of THDQE .
When researchers focus on model design, they often worry whether the model is correct or not. I believe that we should accept the fact that all the models are wrong. The world is too complex to be captured by a single model: we are never able to acknowledge all the variables. Therefore, the answer to the question “Is the model correct?” is always “No”. It should not bother…
I have just published a preprint of a paper ‘Quantile-Respectful Density Estimation Based on the Harrell-Davis Quantile Estimator’. It is based on a series of my research notes . The paper preprint is available on arXiv: arXiv:2404.03835 [stat.ME] . The paper source code is available on GitHub: AndreyAkinshin/paper-qrdehd . You can cite it as follows: Andrey Akinshin (2024)…
In [[discrete-sample-jittering2]], I discussed the jittering approach that improves [[qrde]] for discrete distributions and continuous-discrete mixtures. In this post, I will show a brief example of how such an approach improves the accuracy of the [[lowland-multimodality-detection]].
I continue the topic of [[qrde]] in the context of [[multimodality-detection]]. In this post, we briefly discuss the handling of the QRDE boundary spikes in order to correctly detect the near-border modes.
In [[discrete-sample-jittering]], I proposed a jittering approach. It turned out that it does not always work well. It is not always capable of preserving the original distribution shape and avoiding gaps. In this post, I would like to propose a better strategy.
This post examines how asymmetric distributions challenge Cohen’s d and its nonparametric analogues. In Pragmastat, Disparity provides a robust effect size normalized by pooled Spread that performs well even under asymmetry. Confidence intervals are available via DisparityBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat Cohen’s d is one of the most popular measures of the…
Statistics is one of the most confusing, controversial, and depressing disciplines I know. So many different approaches, so many different opinions, so many arguments, so many person-years of wasted time, and so many flawed peer-reviewed papers. What we want from statistics is an easy-to-use tool that would nudge us toward asking the right questions and then straightforwardly guide us on how to…
If you are curious if something impacts something else, the answer is probably “yes.” Does that indicator depend on those factors? Yes, it does. If we change this thing, would it affect …? Yes, it would. If a person takes this pill, could it cause a non-exactly-zero change in the body? Yes, the presence of the pill is already a change that can always be detected with the right…
Imagine a city mayor considering a project offering to build parks in several neighborhoods. It can be a good budget investment since it can potentially increase the happiness level of the citizens. However, it is just a hypothesis: if parks do not impact happiness, it is worth considering other city renovation projects. It makes sense to perform a pilot experiment before spending the budget on…
Translation of the buisness goals to the actual parameters of the statistical procedure is a non-trivial task. The degree of non-triviality increases if we should adjust several parameters at the same time. In this post, we consider a problem of simultaneous choice of the confidence level and the practical significance threshold. We discuss possible pitfalls and how to simplify the adjusting…
Let’s say we have two data samples, and we want to check if there is a difference between them. If we are talking about any kind of difference, the answer is most probably yes. It’s highly unlikely that two random samples are identical. Even if they are, there are still chances that we observe such a situation by accident, and there is a difference in the underlying distributions.…
I continue building a weighted version of the Mann–Whitney $U$ test. While previously suggested approach feel promising, I don’t like the usage of Bootstrap to obtain the $p$-value. It is always better to have a deterministic and exact approach where it’s possible. I still don’t know how to solve it in general case, but it seems that I’ve obtained a reasonable solution for…
Mann-Whitney is one of the most popular non-parametric statistical tests. Unfortunately, most test implementations in statistical packages are far from perfect. The exact p-value calculation is time-consuming and can be impractical for large samples. Therefore, most implementations automatically switch to the asymptotic approximation, which can be quite inaccurate. Indeed, the classic normal…
In the world of mathematical statistics, there is a constant confrontation between adepts of different paradigms. This is a constant source of confusion for many researchers who struggle to pick out the proper approach to follow. For example, how to choose between the frequentist and Bayesian approaches? Since these paradigms may produce inconsistent results (e.g., see Lindley’s paradox ),…
Change point detection (CPD) in time series analysis is an essential tool for identifying significant shifts in data patterns. These shifts, or “change points,” can signal critical transitions in various contexts. While most CPD algorithms are adept at discovering historical change points, their sensitivity in detecting recent changes can be limited, often due to a key parameter: the…
[[p2-quantile-estimator]] is a streaming quantile estimator with $\mathcal{O}(1)$ memory footprint and an extremely fast update procedure. Several days ago, I learned that it was adopted for the new Paint.NET GPU-based Median Sketch effect (the description is here ). While P² meets the basic problem requirement (streaming median approximation without storing all the values), the algorithm…
This post clarifies the difference between the Hodges–Lehmann ratio estimator and Bhattacharyya’s scale ratio, correcting a misattribution found in several papers. In Pragmastat, Ratio is the recommended robust multiplicative difference estimator. Confidence intervals are available via RatioBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat Previously, I discussed an idea of a…
This post compares the finite-sample Gaussian efficiency of the Shamos estimator and the Rousseeuw–Croux $Q_n$. Pragmastat adopts the Shamos approach as Spread — balancing robustness and efficiency with a 29% breakdown point. Confidence intervals are available via SpreadBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat Previously , we compared the finite-sample Gaussian efficiency of…
When we choose a change point detection algorithm, the most important thing is to clearly understand why we want to detect the change points. The knowledge of the final business goals is essential. In this post, I show a simple example of how a business requirement can be translated into algorithm adjustments.
The usefulness and meaningfulness of the violin plots are dubious (e.g., see this video and the corresponding discussion ). While this type of plot inherits issues of density plots (e.g., the bandwidth selection problem ) and box plots, it also introduces new problems. One such problem is data inconsistency: default density plots and box plots are often incompatible with each other. In this post,…
We consider a problem of change point detection at the end of a time series. Let us say that we systematically monitor readings of an indicator, and we want to react to noticeable changes in the measured values as fast as possible. When there are no changes in the underlying distribution, any alerts about detected change points should be considered false positives. Typically, in such problems, we…
Previously, I have discussed the topic of the resistance to the low-density regions of various estimators including the Hodges-Lehmann location estimator ($\operatorname{HL}$). In general, $\operatorname{HL}$ is a great estimator with great statistical efficiency and a decent breakdown point. Unfortunately, it has low resistance to the low-density regions around $29^\textrm{th}$ and…
This post compares the efficiency of the Mean, Sample Median, and Hodges–Lehmann estimator under heavy-tailed distributions. In Pragmastat, the Hodges–Lehmann approach is Center for location and Spread for scale — both with a 29% breakdown point and high efficiency. pragmastat.dev github.com/AndreyAkinshin/pragmastat In the previous post , I shared some thoughts on how to evaluate the statistical…
This post discusses how to meaningfully compare the efficiency of estimators under heavy-tailed distributions, where variance-based efficiency breaks down. Pragmastat addresses this with Center for location and Spread for scale — robust estimators designed to perform well across distribution families. pragmastat.dev github.com/AndreyAkinshin/pragmastat Statistical efficiency is an essential…
This post compares the finite-sample Gaussian efficiency of the QAD estimator against the Rousseeuw–Croux $S_n$ and $Q_n$. Pragmastat recommends Spread — the Shamos estimator — as a robust scale estimator with a 29% breakdown point. Confidence intervals are available via SpreadBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat In this post, we discuss the finite-sample Gaussian…
Mann-Whitney U test is a good nonparametric test, which mostly targets changes in locations. However, it doesn’t properly support all types of differences between the two distributions. Specifically, it poorly handles changes in variance. In this post, I briefly discuss its behavior in reaction to scaling a distribution without introducing location changes.
The Ansari-Bradley test is a popular rank-based nonparametric test for a difference in scale/dispersion parameters. In this post, we explore its power curve in a numerical simulation.
Previously, I already discussed the Cucconi test . In this post, I continue the topic of nonparametric tests and check out the Lepage test .
This post applies the weighted Hodges–Lehmann estimator to mixture distributions, studying how well it tracks the dominant component. In Pragmastat, Center is the recommended robust location estimator. Confidence intervals are available via CenterBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat The classic non-weighted [[hodges-lehmann-estimator|Hodges-Lehmann location estimator]] of a…
Let us consider the classic problem of outlier detection in one-dimensional sample. One of the most popular approaches is Tukey’s fences, that defines the following range: $$ [Q_1 - k(Q_3 - Q_1);\; Q_3 + k(Q_3 - Q_1)], $$ where $Q_1$ and $Q_3$ are the first and the third quartiles of the given sample. All the values outside the given range are classified as outliers. The typical values of…
It is inconvenient to work with samples from a distribution of unknown form. Therefore, researchers often switch to considering the sample mean value and hope that thanks to the central limit theorem , the distribution of the sample means should be approximately normal. They say that if we consider samples of size $n \geq 30$, we can expect practically acceptable convergence to normality thanks to…
This post compares the Gaussian efficiency of the two-sample Hodges–Lehmann shift estimator against applying the one-sample estimator to each group separately. In Pragmastat, Shift is the recommended two-sample estimator. Confidence intervals are available via ShiftBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat Let us consider two samples $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ and…
In the old times of applied statistics existence, all statistical experiments used to be performed by hand. In manual investigations, an investigator is responsible not only for interpreting the research results but also for the applicability validation of the used statistical approaches. Nowadays, more and more data processing is performed automatically on enormously huge data sets. Due to the…
This post introduces a ratio estimator for multiplicative differences between two samples, built on the Hodges–Lehmann framework via a log-exp transform. In Pragmastat, this is Ratio — the recommended robust multiplicative difference estimator. Confidence intervals are available via RatioBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat For two samples $\mathbf{x} = ( x_1, x_2, \ldots,…
Previously, I suggested a weighted version of the Mann–Whitney $U$ test. The distribution of the weighted normalized $U_\circ^\star$ can be obtained via bootstrap. However, it is always nice if we can come up with an exact solution for the statistic distribution or at least provide reasonable approximations. In this post, we start exploring this distribution.
The Cucconi test is a nonparametric two-sample test that compares both location and scale. It is a classic example of the family of tests that perform such a comparison simultaneously instead of combining the results of a location test and a scale test. Intuitively, such an approach should fit well unimodal distributions. Moreover, it has the potential to outperform more generic nonparametric…
Recently, I started writing about defensive statistics . The methodology allows having parametric assumptions, but it adjusts statistical methods so that they continue working even in the case of huge deviations from the declared assumptions. This idea sounds quite similar to nonparametric and robust statistics. In this post, I briefly explain the difference between different statistical…
Recently, I was rereading [[hampel-robust-statistics]] and I found this quote about the difference between robust and nonparametric statistics (page 9): Robust statistics considers the effects of only approximate fulfillment of assumptions, while nonparametric statistics makes rather weak but nevertheless strict assumptions (such as continuity of distribution or independence). This statement may…
Robust statistics is a practical and pragmatic branch of statistics. If you want to design reliable and trustworthy statistical procedures, the knowledge of robust statistics is essential. Unfortunately, it’s a challenging topic to learn. In this post, I share my favorite books on robust statistics. I cannot pick my favorite one: each book is good in its own way, and all of them complement…
This post investigates effect size measures for multimodal distributions, where standard approaches based on mean differences fail. In Pragmastat, Disparity provides a robust effect size normalized by pooled Spread . Confidence intervals are available via DisparityBounds . pragmastat.dev github.com/AndreyAkinshin/pragmastat When we want to express the difference between two samples or…
In R, we have functions to calculate the density, distribution function, and quantile function of the Wilcoxon Signed Rank statistic distribution: dsignrank , psignrank , and qsignrank . All the functions use exact calculations of the target functions (the R 4.3.1 implementation can be found here ). The exact approach works excellently for small sample sizes. Unfortunately, for large sample sizes,…