Nyquist–Shannon sampling theorem

The Nyquist–Shannon sampling theorem is a fundamental theorem in the field of information theory, in particular telecommunications. In addition to Claude Shannon and Harry Nyquist, it is also attributed to Whittaker and Kotelnikov, and sometimes simply referred to as the sampling theorem.

Sampling is the process of converting a signal (e.g., a function of continuous time) into a numeric sequence (a function of discrete time). The process is also called analog-to-digital conversion, or simply digitizing. The theorem states conditions under which the samples represent no loss of information and can therefore be used to reconstruct the original signal with arbitrarily good fidelity. It states that the signal must be bandlimited and that the sampling frequency must be at least twice the signal bandwidth.

More precisely, analog-to-digital conversion actually consists of the combination of two processes: sampling, which involves converting the domain of the signal from continuous-time to discrete-time, and quantization, which involves converting the amplitude of the signal from a continuous range of values to a finite set of discrete values.

A signal that is bandlimited is constrained in terms of how fast it can change and therefore how much detail it can convey in between discrete moments of time. The sampling theorem means that the discrete samples are a complete representation of the signal if the highest frequency component is less than half the sample-rate, which is referred to as the Nyquist frequency. Frequency components that are above the Nyquist frequency are subject to a phenomenon called aliasing, which is undesirable in most applications. The severity of the problem depends on the relative strength of the aliased components.

To formalize these concepts, let $x(t)\,$ represent a real-valued continuous-time signal and let $X(f)\,$ represent its unitary Fourier transform (to the domain of ordinary frequency, Hz). I.e.:

$X(f)={\mathcal {F}}\{x(t)\}=\int _{-\infty }^{\infty }x(t)\ e^{-j2\pi ft}\ dt$

Spectrum of a bandlimited signal as a function of frequency

The figure depicts a bandlimited $X(f)\,$ whose highest frequency is $f_{H}\,$.   I.e.:

$X(f)=0\quad {\mbox{ for }}\quad |f|>f_{H}\,$

Then the condition for alias-free sampling at rate $f_{s}\,$ (in samples per second) is:

$f_{s}>2f_{H}\,$   (Nyquist rate)

or equivalently:

$f_{H}<f_{s}/2\,$   (Nyquist frequency)

The time interval between successive samples is a constant, referred to as sampling interval. It is given, in seconds, by:

$T={1 \over f_{s}}\,$

And the samples of $x(t)\,$ are denoted by:

$x_{n}:=x\left({n \over f_{s}}\right)=x(nT)\,$

for a sequence of integers denoted by n.

Introduction to sampling

From a signal processing perspective, the theorem describes two processes; a sampling process, in which a continuous or analog signal is converted into a discrete or digital signal, and a reconstruction process, in which the continuous/analog signal is recovered from the discrete/digital signal.

Let us assume that the continuous/analog signal varies over time and that the sampling process is done by simply measuring the continuous/analog signal's value every T seconds, which is called the sampling interval (in practice, the sampling interval is typically quite small, on the order of milliseconds or even microseconds). This results in a sequence of numbers which can be said to represent the original signal in one way or another. Let us call the elements of this sequence samples. Notice that each sample is associated to the specific point in time where it was measured. Notice also that 1/T can be interpreted as a sampling frequency, which is often represented by the symbol f_s and measured in samples per second, or equivalently, hertz.

Let us also assume that the reconstruction process is done by somehow interpolating a continuous/analog signal from the samples.

A very practical question would be to ask: under what circumstances is it possible to reconstruct the original signal completely and exactly (perfect reconstruction)?

The answer is provided by the sampling theorem. In fact, it states two things:

The sinc-function, showing the central location at x/π = 0, and periodicity of zero-crossings at non-zero, integer multiples of x/π.

• Each sample should be multiplied by a particular function, called a sinc-function. The width of each half-period of the sinc-function is scaled to match the sampling frequency, and the location of the sinc-function's central point is shifted to the time of that sample. All of these shifted and scaled functions are then added together to recover the original signal. Recall that a sinc-function is continuous/analog, which means that the result of this operation is indeed a continuous/analog signal. This procedure derives from the Nyquist-Shannon interpolation formula.

• In order to obtain the original signal after this reconstruction process, we must also observe a critical condition on the sampling frequency. It must be at least twice as large as the highest frequency component of the original signal, also measured in hertz.

Sometimes, the sampling theorem refers only to the last statement, but you need also the first one to put things into the right context.

A few practical conclusions can be drawn from the theorem:

• If it is known that the signal which we sample has a certain highest frequency, the theorem gives us the lowest possible sampling frequency to assure perfect reconstruction. This minimum value of the sampling frequency is called the Nyquist rate, or f_N.

• If instead the sampling frequency is known, the theorem gives us an upper bound for the frequencies of the signal to assure perfect reconstruction.

• Both of these cases imply that the signal to be sampled should be bandlimited, i.e., any component of this signal which has a frequency above a certain bound should be zero, or at least sufficiently close to zero to allow us to neglect its influence on the resulting reconstruction. In the first case the condition of bandlimitation of the sampled signal can be accomplished by assuming a model of the signal which can be analysed in terms of the frequency components it contains, e.g., sounds which are made by a speaking human normally contains very small frequency components above 5 kHz and it is then sufficient to sample such an audio signal with a sampling frequency of at least 10 kHz. For the second case, we have to assure that the sampled signal is bandlimited such that frequency components above half of the sampling frequency can be neglected. This is usually accomplished by means of a suitable low-pass filter.

• In practice, neither of the two statements of the sampling theorem described above can be completely satisfied. The reconstruction process which involves the sinc-functions can be described as ideal. It cannot be realized in practice since it implies that each sample contributes to the reconstructed signal at almost all time points. Instead some type of approximations of the sinc-functions which are truncated to limited intervals have to be used. The error which corresponds to the sinc-function approximation is referred to as interpolation error. Furthermore, in practice the sampled signal can never be exactly bandlimited. This means that even if an ideal reconstruction could be made, the reconstructed signal would not be exactly the sampled signal. The error which corresponds to the failure of bandlimitation is referred to as aliasing.

• The sampling theorem implies that someone who is going to design a system which deals with sampling and reconstruction processes needs a thorough understanding of the signal to be sampled, in particular its frequency content, the sampling frequency, how is the signal reconstructed in terms of interpolation, and what are the requirement of the total reconstruction error, including aliasing and interpolation error. In simple terms, all these properties and parameters have to be carefully tuned in order to obtain a useful system.

Aliasing

If the sampling condition is not satisfied, then frequencies will overlap (see the proof below). This overlap is called aliasing.

To prevent aliasing, two things can readily be done

1. Increase the sampling rate
2. Introduce an anti-aliasing filter or make anti-aliasing filter more stringent

The anti-aliasing filter is to restrict the bandwidth of the signal to satisfy the sampling condition. This holds in theory, but is not satisfiable in reality. It is not satisfiable in reality because a signal will have some energy outside of the bandwidth. However, the energy can be small enough that the aliasing effects are negligible.

Application to multivariable signals and images

Subsampled image showing a Moiré pattern

See for full size image

The sampling theorem is usually formulated for functions of a single variable. Consequently, the theorem is directly applicable to time-dependent signals and is normally formulated in that context. However, the sampling theorem can be extended in a straightforward way to functions of arbitrarily many variables. Grayscale images, for example, are often represented as two-dimensional arrays (or matrices) of real numbers representing the relative intensity of each pixel located at the intersection of a single row and a single column. As a result, grayscale images require two independent variables, or indices, to specify each pixel uniquely – one for the row, and one for the column.

Color images typically consist of a composite of three separate grayscale images, one to represent each of the three primary colors – red, green, and blue, or RGB for short. So color images actually require three independent indices, the first two specifiy the pixel location, and the third specifies one of the three colors.

Similar to one-dimensional discrete-time signals, images can also suffer from aliasing if the sampling resolution, or pixel density, is inadequate. For example, a digital photograph of a striped shirt with high frequencies (in other words, the distance between the stripes is small), can cause aliasing between the shirt and the camera's sensor array. The aliasing appears as a Moiré pattern. The "solution" to higher sampling in the spatial domain for this case would be to move closer to the shirt or use a higher resolution sensor (for example, a CCD).

Another example is shown to the right in the brick patterns. The top image shows the effects when the sampling theorem is not followed. When software rescales an image (the same process that creates the thumbnail shown in the bottom image) it, in effect, runs the image through a low-pass filter first and then downsamples the image to result in a smaller image that does not exhibit the Moiré pattern. The top image is what happens when the image is downsampled without low-pass filtering and aliasing results.

The top image was created by zooming out in GIMP and then taking a screenshot of it. Likely reason that this works is that the zooming feature simply downsamples without low-pass filtering (probably for performance reasons) since the zoomed image is for on-screen display instead of printing or saving.

The application of the sampling theorem on images should not be made without care. For example, the sampling process in any standard image sensor (CCD or CMOS camera) is relatively far from the ideal sampling which would measure the image intensity at a single point. Instead these devices have a relatively large sensor area at each sample point in order to obtain sufficient amount of light. Also, it is not obvious that the analog image intensity function which is sampled by the sensor device is bandlimited. It should be noted, however, that the non-ideal sampling in itself implies some type of low-pass filtering, although far from one that effectively removes high frequency components. Furthermore, since the intensity function in practice is zero outside the actual sensor chip, it cannot be bandlimited. Despite that images have these problems in relation to the sampling theorem, it can be used to describe the basic aspects of down and up sampling of images, but only sufficiently far from the image boundaries.

Downsampling

When a signal is downsampled, the theorem must still be satisfied in order to avoid aliasing. To meet the requirements of the theorem, the signal must pass through a low-pass filter of appropriate cutoff frequency prior to the downsampling operation. The low-pass filter, which prevents aliasing, is called an anti-aliasing filter.

Critical frequency

The critical frequency is defined as twice the bandwidth of the signal. If the sampling frequency is at the critical frequency, exactly twice the highest frequency of the input signal, then phase mismatches between the sampler and the signal will distort the signal. For example, sampling $\cos(\pi t)$ at $t=0,1,2,\dots$ will give you the discrete signal $\cos(\pi n)$, as desired. However, sampling the same signal at $t=0.5,1.5,2.5,\dots$ will give you a constant zero signal. These two sets of samples, which differ only in phase and not frequency, give dramatically different results because they sample at the critical frequency, instead of strictly above the critical frequency. This result provides an rationale for the strict inequality of the sampling condition, and why the sampling rate must exceed the critical frequency.

Consequences of the theorem

A well-known consequence of the sampling theorem is that a signal cannot be both bandlimited and time-limited. To see why, assume that such a signal exists, and sample it faster than the Nyquist frequency. This finite number of time-domain coefficients should define the entire signal. Equivalently, the entire spectrum of the bandlimited signal should be expressible in terms of the finite number of time-domain coefficients obtained from sampling the signal. Mathematically this is equivalent to requiring that a (trigonometric) polynomial can have infinitely many zeros in bounded intervals since the bandlimited signal must be zero on an interval beyond a critical frequency which has infinitely many points. However, it is well-known that polynomials do not have more zeros than their orders due to the fundamental theorem of algebra. This contradiction shows that our original assumption that a time-limited and bandlimited signal exists is incorrect.

Mathematical basis for the theorem

The Fourier transform of the discrete-time sequence is called the discrete-time Fourier transform (DTFT), and it is easily shown to be periodic, with period $f_{s}\,$. As shown below, the DTFT can be constructed by placing copies (aka aliases) of $X(f)\,$ at intervals of $f_{s}\,$ and summing them all together. Clearly, if the original and the copies are bandlimited, and $f_{s}\,$ is sufficiently large to prevent overlap, the original signal can be recovered by simply filtering out the aliases. That is the essence of the sampling theorem.

There are also situations where overlap is allowed to occur, and due to the particular shape of $X(f)\,$ the aliases coincide with null regions of $X(f)\,$. Then neither the alias nor the original spectrum is irreparably affected. See Undersampling.

A mathematical model of the sampling process is to multiply $x(t)\,$ by the Dirac comb function, which is an infinite sequence of impulses occurring at integer multiples of the sampling interval $T={1 \over f_{s}}$:

$x(t)\,\sum _{n=-\infty }^{\infty }\delta (t-nT)\quad =\sum _{n=-\infty }^{\infty }\ x(t){\bigg |}_{t=nT}\cdot \delta (t-nT)\quad =\sum _{n=-\infty }^{\infty }x(nT)\,\delta (t-nT)$

The Fourier transform of the comb is another comb:

$\sum _{n=-\infty }^{\infty }\delta (t-nT)\quad \Longleftrightarrow \quad {1 \over T}\sum _{k=-\infty }^{\infty }\delta \left(f-{k \over T}\right)\ ={1 \over T}\sum _{k=-\infty }^{\infty }\delta (f-kf_{s})$

So the transform of the product is this convolution:

$X_{dtft}(f)=X(f)*{1 \over T}\sum _{k=-\infty }^{\infty }\delta (f-kf_{s})={1 \over T}\sum _{k=-\infty }^{\infty }X(f-kf_{\mathrm {s} })$

which utilizes the shifting property of the Dirac delta under convolution. $X(f)\,$ is proportional to the k=0 term. So when $f_{s}>2f_{H}\,$ as previously discussed, we may conclude by inspection:

$X(f)=X_{dtft}(f)\cdot T\cdot \operatorname {rect} \left({f \over f_{s}}\right)\quad =X_{dtft}(f)\cdot T\cdot \operatorname {rect} \left(Tf\right)\,$

where $T\cdot \operatorname {rect} \left(Tf\right)\,$ is a rectangle function, whose inverse transform is $\operatorname {sinc} \left(\pi t/T\right)\equiv {\frac {\sin(\pi t/T)}{\pi t/T}}\,$

And that essentially proves the theorem, because $x(t)\,$ is just the inverse transform:

$x(t)\,$	$=\left(\sum _{n=-\infty }^{\infty }x(nT)\,\delta (t-nT)\right)*\operatorname {sinc} \left(\pi t/T\right)\,$
	$=\sum _{n=-\infty }^{\infty }x(nT)\cdot \operatorname {sinc} \left(\pi (t-nT)/T\right)$

which is known as the Nyquist–Shannon interpolation formula.

The original proof

The original basis developed by Shannon is also brief, but it offers less insight into the subtleties of aliasing, both unintentional and intentional. The Fourier transform $X(f)\,$ of a bandlimited function $x(t)\,$ has a finite width of $2f_{H}\,$, as defined earlier. Therefore, it can be represented as a Fourier series expansion, using any period $P>2f_{H}\,$. Since $P\,$ is actually a frequency interval, it corresponds to a time interval $T={1 \over P}<{\frac {1}{2f_{H}}}$. In terms of $T\,$, the expansion can be written:

$X(f)=\sum _{n=-\infty }^{\infty }A_{n}\cdot e^{j\,(2\pi f)nT}$,   where $A_{k}=T\int _{-{1 \over 2T}}^{1 \over 2T}X(f)\cdot e^{-j\,(2\pi f)kT}\;df$.

The bandlimited property also reduces the inverse transform to this form, where again we rely on ${1 \over 2T}>f_{H}\,$:

$x(t)={\mathcal {F}}^{-1}\{X(f)\}=\int _{-f_{H}}^{f_{H}}X(f)\cdot e^{j(2\pi f)t}\;df=\int _{-{1 \over 2T}}^{1 \over 2T}X(f)\cdot e^{j(2\pi f)t}\;df$

Therefore, by comparison we can observe that: $A_{n}=T\cdot x(-nT)\,$, which shows that $X(f)\,$ can be fully represented by just the discrete time samples of $x(t)\,$. Of course, that means $x(t)\,$ can also be represented by its samples.  (Q.E.D.)

And of course that representation is again the Nyquist–Shannon interpolation formula. To derive it, we substitute $A_{n}\,$ into the Fourier series and substitute the series for $X(f)\,$ as shown below. The apparent additional complexity is because embedded in the manipulations is also the derivation of the $\operatorname {rect} \Leftrightarrow \operatorname {sinc}$ transform pair.

$x(t)\,$	$=\int _{-{1 \over 2T}}^{1 \over 2T}{\bigg (}\sum _{n=-\infty }^{\infty }T\cdot x(-nT)\cdot e^{j\,(2\pi f)nT}{\bigg )}\cdot e^{j(2\pi f)t}\;df$
	$=\sum _{n=-\infty }^{\infty }x(nT)\cdot T\int _{-{1 \over 2T}}^{1 \over 2T}e^{-j\,(2\pi f)nT}\cdot e^{j(2\pi f)t}\;df$
	$=\sum _{n=-\infty }^{\infty }x(nT){\frac {T}{2\pi (t-nT)}}\left.{\bigg (}\sin(2\pi f(t-nT))-j\cos(2\pi f(t-nT)){\bigg )}\right|_{-{1 \over 2T}}^{f={1 \over 2T}}$
	$=\sum _{n=-\infty }^{\infty }x(nT)\cdot {\frac {\sin(\pi (t-nT)/T)}{\pi (t-nT)/T}}=\sum _{n=-\infty }^{\infty }x(nT)\cdot \operatorname {sinc} \left[\pi (t-nT)/T\right]\,$

Undersampling

When sampling a non-baseband signal, the theorem must be restated as follows. Let $0<f_{\mathrm {L} }<f_{\mathrm {H} }$ be the lower and higher boundaries of a frequency band and $W=f_{\mathrm {H} }-f_{\mathrm {L} }$ be the bandwidth. Then there is a non-negative integer N with

$N\leq {f_{\mathrm {L} } \over W}<(N+1)\leq {f_{\mathrm {H} } \over W}$

In addition, we define the remainder r as

$r:=f_{\mathrm {L} }-NW\in [0,f_{\mathrm {L} }]$.

Any real-valued signal x(t) with a spectrum limited to this frequency band, that is with

$X(f)={\mathcal {F}}\{x\}(f)=0$ for $|f|\,$ outside the interval $[f_{\mathrm {L} },f_{\mathrm {H} }]\,$,

is uniquely determined by its samples obtained at a sampling rate of $f_{\mathrm {s} }$, if this sampling rate satisfies one of the following conditions:

• $2\left(W+{\frac {nW+r}{N-n+1}}\right)\leq f_{\mathrm {s} }\leq 2\left(W+{\frac {nW+r}{N-n}}\right)$ for one value of n = { 0, 1, ..., N-1 }

- OR -

• $2f_{\mathrm {H} }\leq f_{\mathrm {s} }$.

If $N>0$, then the first conditions result in a sampling rate less than the Nyquist frequency $2f_{\mathrm {H} }$ obtained from the upper bound of the spectrum. If the so obtained sampling rates are still too high, the intuitive sampling-by-taking-values has to be replaced by sampling-by-taking-scalar-products, as is (implicitly) the case in Frequency-division multiplexing.

Example: Consider FM radio to illustrate the idea of undersampling.

In the US, FM radio operates on the frequency band from $f_{L}$ = 88 MHz to $f_{H}$ = 108 MHz. The bandwidth is given by

$W=f_{H}-f_{L}=108\ \mathrm {MHz} -88\ \mathrm {MHz} =20\ \mathrm {MHz}$

The sampling conditions are satisfied for

$N<4.4={88\ \mathrm {MHz} \over 20\ \mathrm {MHz} }<N+1$

Therefore

N=4, r=8MHz and $n=0,1,2,3$.

The value $n=0$ gives the lowest sampling frequencies interval $43.2\ \mathrm {MHz} <f_{\mathrm {s} }<44\ \mathrm {MHz}$ and this is a scenario of undersampling.

Note that when undersampling a real-world signal, the sampling circuit must be fast enough to capture the highest signal frequency of interest. Theoretically, each sample should be taken during an infinitesimally short interval, but this is not practically feasible. Instead, the sampling of the signal should be made in a short enough interval that it can represent the instantaneous value of the signal with the highest frequency. This means that in the FM radio example above, the sampling circuit must be able to capture a signal with a frequency of 110 MHz, not 43.2 MHz. Thus, the sampling frequency may be only a little bit greater than 43.2 MHz, but the input bandwidth of the system must be at least 110 MHz.

If the theorem is misunderstood to mean twice the highest frequency, then the sampling rate would assumed to need to be greater than the Nyquist-frequency 216 MHz.

While this does satisfy the last condition on the sampling rate, it is grossly over sampled.

Note that if the FM radio band is sampled, e.g., at $43.2\ \mathrm {MHz} <f_{\mathrm {s} }<44\ \mathrm {MHz}$, then a band-pass filter is required for the anti-aliasing filter.

In certain problems, the frequencies of interest are not an interval of frequencies, but perhaps some more interesting set F of frequencies. Again, the sampling frequency must be proportional to the size of F. For instance, certain domain decomposition methods fail to converge for the 0th frequency (the constant mode) and some medium frequencies. Then the set of interesting frequencies would be something like 10 Hz to 100 Hz, and 110 Hz to 200 Hz. In this case, one would need to sample at a data rate of 360 Hz — i.e. at a sampling rate of 20 Hz with 18 real values in each sample — not 400 Hz, to fully capture these signals.

As we have seen, the normal condition for reversible sampling is that $X(f)=0\,$ outside the interval: $\left[-{\frac {1}{2}}f_{\mathrm {s} },{\frac {1}{2}}f_{\mathrm {s} }\right]$

And the reconstructive interpolation function is $\operatorname {sinc} \left(\pi t/T\right)$.

To accommodate undersampling, the generalized condition is that $X(f)=0\,$ outside the union

$\left[-{\frac {N+1}{2}}f_{\mathrm {s} },-{\frac {N}{2}}f_{\mathrm {s} }\right]\cup \left[{\frac {N}{2}}f_{\mathrm {s} },{\frac {N+1}{2}}f_{\mathrm {s} }\right]$ for some $N\in \mathbb {N}$.

which includes the normal condition as case N=0.

And the corresponding interpolation function is:

$(N+1)\operatorname {sinc} \left[{\frac {\pi (N+1)t}{T}}\right]-N\operatorname {sinc} \left[\pi \left({\frac {Nt}{T}}\right)\right]$.

Historical background

The theorem was first formulated by Harry Nyquist in 1928 ("Certain topics in telegraph transmission theory"), but was only formally proven by Claude E. Shannon in 1949 ("Communication in the presence of noise"). Kotelnikov published in 1933, Whittaker in 1915 (E.T.) and 1935 (J.M.), and Gabor in 1946.

See also

• Aliasing
• Anti-aliasing filter: low-pass filter, band-pass filter
• Dirac comb
• Nyquist–Shannon interpolation formula
• Sampling (information theory)
• Signal (information theory)
• Reconstruction from Zero Crossings

References

• E. T. Whittaker, "On the Functions Which are Represented by the Expansions of the Interpolation Theory," Proc. Royal Soc. Edinburgh, Sec. A, vol.35, pp.181-194, 1915
• H. Nyquist, "Certain topics in telegraph transmission theory," Trans. AIEE, vol. 47, pp. 617-644, Apr. 1928.
• V. A. Kotelnikov, "On the carrying capacity of the ether and wire in telecommunications," Material for the First All-Union Conference on Questions of Communication, Izd. Red. Upr. Svyazi RKKA, Moscow, 1933 (Russian).
• C. E. Shannon, "Communication in the presence of noise", Proc. Institute of Radio Engineers, vol. 37, no.1, pp. 10-21, Jan. 1949.

External links

• Learning by Simulations Interactive simulation of the effects of inadequate sampling
• Undersampling and an application of it