The inner product (or ``dot product'', or ``scalar product'') is an operation on two vectors which produces a scalar. Defining an inner product for a Banach space specializes it to a Hilbert space (or ``inner product space''). There are many examples of Hilbert spaces, but we will only need for this book (complex length vectors, and complex scalars).
The inner product between (complex) -vectors and is defined by5.9
Note that the inner product takes to . That is, two length complex vectors are mapped to a complex scalar.
Any function of a vector (which we may call an operator on ) is said to be linear if for all and , and for all scalars and in ,
The inner product is linear in its first argument, i.e., for all , and for all ,
The inner product is also additive in its second argument, i.e.,
The inner product is strictly linear in its second argument with respect to real scalars and :
Since the inner product is linear in both of its arguments for real scalars, it may be called a bilinear operator in that context.
The Cauchy-Schwarz Inequality (or ``Schwarz Inequality'') states that for all and , we have
We can quickly show this for real vectors , , as follows: If either or is zero, the inequality holds (as equality). Assuming both are nonzero, let's scale them to unit-length by defining the normalized vectors , , which are unit-length vectors lying on the ``unit ball'' in (a hypersphere of radius ). We have
The triangle inequality states that the length of any side of a triangle is less than or equal to the sum of the lengths of the other two sides, with equality occurring only when the triangle degenerates to a line. In , this becomes
A useful variation on the triangle inequality is that the length of any side of a triangle is greater than the absolute difference of the lengths of the other two sides:
Proof: By the triangle inequality,
Interchanging and establishes the absolute value on the right-hand side.
The Cauchy-Schwarz Inequality can be written
Note that if and are real and orthogonal, the cosine of the angle between them is zero. In plane geometry (), the angle between two perpendicular lines is , and , as expected. More generally, orthogonality corresponds to the fact that two vectors in -space intersect at a right angle and are thus perpendicular geometrically.
Let and , as shown in Fig.5.8.
The inner product is . This shows that the vectors are orthogonal. As marked in the figure, the lines intersect at a right angle and are therefore perpendicular.
This relationship generalizes to dimensions, as we can easily show:
If , then and Eq.(5.1) holds in dimensions.
Note that the converse is not true in . That is, does not imply in . For a counterexample, consider , , in which case
For real vectors , the Pythagorean theorem Eq.(5.1) holds if and only if the vectors are orthogonal. To see this, note that, from Eq.(5.2), when the Pythagorean theorem holds, either or is zero, or is zero or purely imaginary, by property 1 of norms (see §5.8.2). If the inner product cannot be imaginary, it must be zero.
Note that we also have an alternate version of the Pythagorean theorem:
The orthogonal projection (or simply ``projection'') of onto is defined by
Motivation: The basic idea of orthogonal projection of onto is to ``drop a perpendicular'' from onto to define a new vector along which we call the ``projection'' of onto . This is illustrated for in Fig.5.9 for and , in which case
Signal Reconstruction from Projections