maths and data
Let \(f\) be a real function. For \(\lambda > 0\), we are interested in the equally-spaced summation \(\sum_{k=-\infty}^{+\infty} f(.+k\lambda)\), that can be interpreted depending on the context as a periodic mixture or as a wrapped distribution. For some specific functions such as the Gaussian density, we derive expressions, evaluations, and approximations of the sum, further accompanied with…
Let \(Z\) be a \(\text{Exp}(1)\) random variable. For \(\alpha_1, \ldots \alpha_N \in \mathbb{R}_{+}^{*}\), we are looking for variables \(X_1, \ldots X_N\) independent and following the same distribution such that: \[Z = \sum_{j=1}^{N} \alpha_j X_j.\] For the special case where all \(\alpha_j\) are equal, this corresponds to the problem of infinite divisibility of the exponential distribution,…
This tutorial highlights structure of common RNN algorithms by following and understanding computations carried out by each model. It is intended for anyone knowing the general deep learning workflow, but without prior understanding of RNN. If you really never heard about RNN, you can read this post of Christopher Olah first. The present post focuses on understanding computations in each model…
[This tutorial has been written for answering a stackoverflow post, and has been used later in a real-world context]. This tutorial provides a complete introduction of time series prediction with RNN. In part A, we predict short time series using stateless LSTM. Computations give good results for this kind of series. In part B, we try to predict long time series using stateless LSTM. In that case,…
Coal is a small software dedicated to automatize composition of linear functions. For example, it can compute slope, intercept and fixed point of gfg given f: x ↦ ax+b and g: x ↦ cx+d. Here, a and b can be formal letters, rational values or real values.
Let \(x_1, x_2, x_3\) be three points in a plane. We define \(x_4\) the point on the ray \([x_3 x_1)\) located at a distance \(1\) of \(x_3\). It is as \(x_1\) has been attracted to \(x_3\) but kept at distance. We continue by defining \(x_5\) the point on the ray \([x_4 x_2)\) located at a distance \(1\) of \(x_4\). On the whole, we define from \((x_1, x_2, x_3)\) a recurrent sequence taking…
Many methods exist to visualize high-dimensional data through a two-dimensional map. Those include linear techniques such as PCA and MDS; as well as nonlinear ones such as Isomap, LLE, SNE and t-SNE (resp. Principal Component Analysis, 1933; MultiDimensional Scaling, 1952; Isomap, 2000; Locally Linear Embedding, 2000; Stochastic Neighbor Embedding, 2002; t-Distributed Stochastic Neighbor…
This post explains why maximizing likelihood is equivalent to minimizing KL-divergence. This can already be found here and here, but I restate this in my “own” words. More generally, I encourage you to read Section 3.13 of Deep Learning book for insights on information theory. Let \(\mathbf{x} = (x_1, \ldots x_n)\) a dataset of \(n\) elements. We assume that each \(x_i\) has been sampled…
A Gaussian Mixture Model (GMM) models data as a finite mixture of Gaussians. It is often used to perform clustering. In this case, the number of Gaussians \(K\) is the number of clusters and is initially selected. Compared to \(K\)-means, GMM allows clusters with uneven variance and density. Parameters are usually estimated using an Expectation-Maximization (EM) algorithm, its aim being to…
Expectation-Maximization algorithm (EM) is an algorithm used to estimate parameters of statistical models comprising latent variables. It is applied in the case where direct calculation of maximum likelihood estimation (MLE) is impractical. EM updates parameters of the model iteratively, in order to increase likelihood of the set. It generally gives good estimation results, although there is no…