Hidden Linear Algebra in DSP

Sami AldalahmehJune 17, 20105 comments

Linear algebra (LA) is usually thought of as a blunt theoretical subject. However, LA is found hidden in many DSP algorithms used widely in practice.

An obvious clue in finding LA in DSP is the linearity assumption used in theoretical analysis of systems for modelling or design. A standard modelling example for this case would be linear time invariant (LTI) systems. LTI are usually used to model flat wireless communication channels. LTI systems are also used in the design of digital filter such as FIR and IIR filters.

Linear systems are often introduced in standard system theory textbooks, informally, as systems that preserve additive inputs up to a scaling factor. The emphasis, however, is usually on the impulse response, which completely characterizes the system, and convolution that is used to find the output to any input signal. This emphasis is justly in its place, but let us deviate from the conventional treatment and look at it from LA point of view. If we are examining a linear system, LA says that we can represent it in matrix form, i.e. if we have a (Nx1) input vector signal (x) and we desire a (Mx1) vector output signal (y), the system can be characterized by the standard LA form (y = A*x) , where (A) is a (MxN) matrix. In case of LTI systems the matrix is Toeplitz, a structure repeatedly seen in DSP e.g. in wide sense stationary covariance matrices. What is interesting is that LA gives the advantage of geometric visualization of the system. We can think of the input signal (x) as a vector in N dimensional space and of the system matrix (A) as a linear transformation operator., so the system is just transforming one vector in space, x, to another vector, y. LA also provides further insight into the system by inspecting intrinsic matrix characteristics such as eigenvalues and eigenvectors.

This approach is of interest by itself (if you get in touch with your mathematical side) or can be of practical value in the future. In both cases, using the vast and well developed area of LA in DSP will surely have significant benefits.

[ - ]
Comment by October 2, 2011
I took a look on the book by Moon and Striling, the least that I can say is that this book is a "must have" for any engineer who wants to know the why and the how of signal processing. It tries to break the barrier between engineering analysis and rigorous mathematical analysis. Thanks alexpazh!
[ - ]
Comment by Mich0361June 24, 2010
[ - ]
Comment by Mich0361June 24, 2010
I like numbers and logic. Even I like playing with numbers. Linear Algebra has significant role in understanding any analysis courses.
[ - ]
Comment by alexpazhSeptember 20, 2011
Interesting point. For anyone who wants to learn more about this "hidden in DSP linear algebra" I suggest to read unique book on this subject: "Mathematical Methods and Algorithms for Signal Processing" by Todd K. Moon and Wynn C. Stirling, especially part II of this book (Vector spaces and linear algebra). This book provides good insight on subject and is full of examples how Linear algebra is connected with signal processing. And for anyone who wants to learn more about linear algebra as interesting subject with many different applications I would like to suggest to read books by Gilbert Strang: "Linear Algebra and Its Applications" and "Introduction to Applied Mathematics" (or newer book "Computational Science and Engineering"), or watch his lectures at MIT on MIT Open Course Ware site: http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/ http://ocw.mit.edu/courses/mathematics/18-085-computational-science-and-engineering-i-fall-2008/video-lectures/ http://ocw.mit.edu/courses/mathematics/18-086-mathematical-methods-for-engineers-ii-spring-2006/video-lectures/ And one more very good book on applied linear algebra is "Matrix analysis and applied linear algebra" by Carl D. Meyer, which can be found here: http://www.matrixanalysis.com/DownloadChapters.html Personally, after reading these books I can not think about Linear Algebra as "blunt theoretical subject" anymore.
[ - ]
Comment by julespenkJanuary 12, 2022

for those starting out - or wanting a refresh and beautiful 'real time' visualisations of anything geometric/LA, then I don't believe there is any better resource than first watching and working through 3b1b's first course on LA:

it is simply excellent.


To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.

Please login (on the right) if you already have an account on this platform.

Otherwise, please use this form to register (free) an join one of the largest online community for Electrical/Embedded/DSP/FPGA/ML engineers: