Z-Score¶
Status: available on rasters and scatter plots, and as a flag filter.
What it is¶
A robust statistical score of how unusual each sample is for its own
baseline, in the style of CASA’s rflag. It is computed jointly from
the real and imaginary parts, against a per-baseline reference that uses
the median and the median absolute deviation (MAD). It is always 0 or
greater: it measures how anomalous a sample is, not in which direction.
What it is for¶
It finds outliers without one threshold for every baseline. A short baseline on a bright source and a long, noisy one are each judged against themselves. On a Baseline × Time raster, isolated bright cells are outliers in time.
How it behaves¶
Already-flagged samples are left out of the reference population, so existing flags do not distort it.
On a raster, a cell that covers several samples shows the largest score among them, so a single outlier is not averaged away.
On a scatter plot it can be plotted as the Y value, or used to color an amplitude or phase layer, so the outliers stand out in an ordinary plot.
As a flag filter, a flag box then takes only the samples above a chosen score (see Flagging).
How it is calculated¶
For each reference population (below), take the medians of the real and imaginary parts separately, \(m_R\) and \(m_I\), of its unflagged samples. For a sample \(V\):
where \(\operatorname{median}(r)\) is taken over the reference population.
What the numbers mean. If the samples are complex Gaussian noise about a constant value, \(Z\) follows a Rayleigh distribution with \(P(Z > t) = e^{-t^2/2}\); the factor \(\sqrt{2\ln 2}\) is chosen so that this holds. So:
the median \(Z\) is 1.18;
\(Z > 3\) happens for about 1 % of samples;
\(Z > 3.5\) (the default cutoff of the Z-Score flag filter, after Iglewicz and Hoaglin) happens for about 0.2 %.
Because the reference uses medians, a minority of bad samples does not move it.
The reference population:
On a scatter plot it is one baseline and one correlation, over the whole current selection: all times, channels and spectral windows selected.
On a raster it is one baseline, one correlation and one spectral window. A sample can therefore score differently on a raster than on a scatter plot when several spectral windows are selected.
In either case the score depends on the selection: narrowing the selection changes the reference, and with it every score.
Blank scores. A population with no spread, such as a single sample, gives no score.
A raster cell shows the largest \(Z\) among its samples, so a single outlier is not averaged away. The largest of many samples is naturally higher than one sample’s value. For that reason, the Z-Score flag filter working on whole cells raises its cutoff with the number of samples \(N\) per cell, so that a cell of pure noise is taken no more often than a single sample would be. The per-sample tail probability \(p\) is set so that \(1-(1-p)^N = e^{-c^2/2}\), where \(c\) is the per-sample cutoff (a Šidák correction). With 384 channels per cell, a cutoff of 3.5 becomes about 4.9.
From Python¶
visplot(vis="obs.ms", preset="zscore")
visplot(vis="obs.ms", scatter_x="TIME", scatter_y="Z_SCORE")
Coming from plotms or AIPS? See Coming from plotms and Coming from AIPS.