Music/Audio Signal Processing

Julius Orion Smith IIISeptember 5, 20087 comments


This is my blog from the point of view of a music/audio DSP research engineer / educator. It is informal and largely nontechnical because nearly everything I have to say about signal processing is (or will be) somewhere in my four-book series: Mathematics of DFT with Audio Applications, Introduction to Digital Filters, Physical Audio Signal Processing and Spectral Audio Signal Processing

I thought I would start out with a short history of how I got involved in music/audio DSP, and perhaps thereby provide some insights into that field. After that, I will say a few words about what I consider to be interesting current developments related to music/audio DSP.

This article is available in PDF format for easy printing

One Path to a Music/Audio Signal Processing Research Career

My story is basically that of an engineer who is passionate about music. I know there are a lot of you out there, and I get to know quite a few among the CCRMA graduate students. In high school, I was a pretty serious musician while keeping my grades up in math and science. At Rice, I worked harder as a student, but I kept up my music side playing semi-professionally in my spare time. There was no connection between music and engineering through this time --- music was something I pursued on my own time, and at school I simply worked at being a good student. A very significant influence at Rice was the DSP and Control curricula there, especially the courses of C. Sidney Burrus and J. Boyd Pearson. That technical foundation served me very well in later years.

Upon graduation, I took a job at ESL in Sunnyvale CA working on digital signal processing applied to communications (mostly at sea). (ESL regularly recruited at Rice University at that time.) This was a great learning experience for me, but still it had nothing (directly) to do with music. I found other musicians through the Bay Area Music guide, and played in several bands for fun and occasional profit. I also built a few electronic effects projects (all analog at that time). ESL had an Honors Coop program that allowed me to take one or two courses per quarter at Stanford toward an MS/EE degree and beyond.

CCRMA co-founder James A. Moorer presented one of the EE 201 seminars, and that motivated me to visit CCRMA (then just a small project at the Artificial Intelligence Lab) to see how I might get involved. I volunteered hardware and software assistance in return for a free computer account at the AI Lab for purposes of learning about computer music. In parallel I was taking 300-level courses in signal processing in EE at Stanford. There was still no formal connection between my academic life and music life, although my music side had taken an academic turn. I did less playing as a musician and more studies related to computer music in my spare time (although I kept playing solo classical guitar, and was an avid Bach student).

After two years at ESL, I applied for and was awarded a Hertz fellowship which enabled me to go back to school full time. For the next five years I was a full-time graduate student (except for summer jobs related to signal processing, which I highly recommend to all graduate students). The greatest thing about fellowship support is that one can choose any reasonable research topic. I therefore chose a problem of importance in music technology: violin synthesis. At the time, one could not purchase a good bowed-string synthesizer for any amount of money, so I figured it must be a problem hard enough to warrant a research effort. In fact, it is hard enough to warrant quite a few dissertations!

The first two years or so of the PhD/EE program were dominated by taking courses. Then and before at Rice, I chose courses that I thought could be useful for music/audio technology down the road. In other words, potential music technology applicability was my most consistent criterion for deciding what courses to take and which research topics to pursue. (I also took some courses out of pure interest, of course.) The most valuable courses for me at Stanford were those put together by Tom Kailath and Joe Goodman. Prof. Kailath's research seminars were especially enlightening.

During and especially after taking courses, one has to master a branch of the current research literature (which involves reading a lot of papers and misc. books in the field). My area of emphasis was digital filter design and system identification (with violin modeling firmly in the back of my mind). In my third year, music applications of DSP were finally starting to pop up in my research (you get ideas while reading the literature, and sometimes a promising idea remains unexplored after you seem to have finished reading all relevant literature). So, at this point, I started to publish conference papers and go to one or so conferences per year (sometimes self-funded - ouch!).

Shortly before graduating, I took a job in the Adaptive Signal Processing Dept. of Systems Control Technology (SCT), working for Ben Friedlander, a well known researcher in statistical signal processing. I met Ben when he was doing a little teaching at Stanford in the area of statistical signal processing (a former Kailath student). This was another great learning experience for me, and I considered it more of a "postdoc" than a job. There was no direct connection with music, but I continued to publish regularly in the computer music literature on the side.

After about two years working at SCT, I was offered a part-time research associate position at CCRMA, funded by a System Development Foundation grant. (I'm told John Pierce had a lot to do with getting that five-year grant.) During that time I started my signal processing courses for music research, and continued my earlier research on violin modeling (which led to the development of waveguide synthesis). Finally I had a real position that was officially in service to music! The funding was temporary, so I kept my job at SCT going at the half-time level.

After another couple of years, I received a call from Steve Jobs saying that the NeXT computer was going to have a Motorola DSP56001 chip on every motherboard, and that they were looking for someone to work on software for signal processing, and he was especially interested in having some good music capabilities. I attribute my "visibility" to being one of the very few PhD/EEs publishing in the music technology field at the time. I started at NeXT in 1986 as the 25th employee, and was the founding member of the NeXT sound, music, and signal processing group. (My official title was simply "Software Engineer", like everyone else in the NeXT software division.) Thus, my NeXT position replaced my SCT position as a music-related engineering (and management) position. During that time, I went down to one day per week at CCRMA, just enough to keep my courses going.

In 1989, I was promoted to Associate Professor (Research) at CCRMA, thanks entirely to the efforts of John Chowning (founding director of CCRMA). As music became less important at NeXT, I transitioned back to CCRMA full time where I have been ever since. CCRMA's financial health at that time and since was primarily due to the success of the FM synthesis patents by John Chowning and Yamaha Corporation.

So, what can one glean from this story in terms of music-technology career advice? First of all, opportunities are unpredictable! I did not foresee any of the career opportunities I ultimately pursued. My "big picture" was pretty simple at all times. First, I was simply getting the best education I could, choosing courses according to "likely future music technology relevance". Next, I was working in the best job I could find that could help me develop more skills that seemed applicable to music/audio research. After the PhD, it was also important that the job allow me to publish at least annually, and that it support some conference attendance. In parallel, I was pursuing music in my own way (with or without much technology).

There are of course many more opportunities in music/audio DSP today than there were when I was a student. It is even possible to develop and sell plugins over the Internet as an individual these days. However, I still recommend pursuit of relationships with the best known institutions in one's chosen field. As the Internet grows, it becomes more and more important to have name/brand recognition, simply because there is too much out there to look at. Already the number of free LADSPA plugins is more than one can try out in a reasonable length of time.

Exciting Recent Developments

On the recent developments front, I have been having a very good time working in the Faust programming language. See, for example,




Another exciting recent development for me is Octave 3.0. It is now amazingly more compatible with Matlab (including handle graphics), and Octave Forge has continued to develop strongly as well. I believe it is now true that I am using 100% free open-source software for all of my research and publication.

Why is using free open-source software important to me? Well, since this is my blog, I'll tell you: Today's engineer needs to be able to "tool up and produce" on an extremely wide variety of fronts in a short time. The monolithic software purchase model inhibits this, while the free, open-source model supports it. For example, I may need a "spread sheet" 3 or 4 times a year. I shouldn't have to spend the same amount of money on Microsoft Excel as someone who uses it every day. I don't mind paying for software, but I feel I should pay in proportion to how much I use it.

About a decade ago, there was a movement toward "micropayment". One of the companies was "Millicent" who claimed that purchases down to a tenth of a cent were economically practical. I don't know why this did not take hold, but I think it is the way to go for commercial software and information dissemination. I think the reason may be simply that the "subscription model" is more profitable, because most subscribers pay even when they don't use the tool or service. Companies know this and they are likely resisting micropayment models as a result.

Of course, in advertising, micropayment has taken off in a big way (witness Google). As Google provides more and more online applications (Google Office, etc.), they will effectively convert software to the micropayment model (supported by ads), and that will be pretty good. However, I would rather give metered micropayment and save the ads for when I'm actually shopping. Also, I don't want to run programs over the Web. I want them installed on my own recently upgraded machine, and I want them to be blazingly fast and instantly responsive. I don't want to depend on a network connection involving several routers and servers in the path.

Fortunately, the free, open-source software revolution is filling the bill. With Fedora Red Hat Linux, the GNU tools, Octave, Pd, Faust, JACK, Ardour, Rosegarden, Open Office, TeX, LaTeX, and the huge variety of additional packages available (the Planet CCRMA distribution for Fedora/RedHat contains many of the music/audio related packages), I can function without having to pay exorbitant subscription fees for things I rarely use. When I need to solve a new problem, I search the Web, download new tools, study the documentation as needed, and keep things moving in one sitting (without entering my credit-card number all the time). Such accelerated, cross-disciplinary problem-solving increases overall productivity, and in my opinion this should be a top priority until everyone in the world has a decent standard of living and population growth is slower than economic growth. In other words, Job 1 for all of us should be "creating wealth" in our own way, and free open-source software contributes significantly to this goal for software engineers.



[ - ]
Comment by superhotmuffinOctober 29, 2012
So, I finally run across the man, the myth and the legend :) My name is Michael and I must say that I am deeply indebted to you, David Yeh and others who have published works at CCRMA and the DAFX conferences as after, literally, years of studying the math and other concepts in these papers I was quite easily able to whip up my own digital guitar amp models - about as fast as I could type the code. I even have a little paper entitled "Real Time Implementation of the '59 Fender Bassman Tone Stack" that I'd like to unleash on the world. Hmm, I wonder where I might've gotten the idea for that??? Anyway, i just wanted to say thanks for all the research that you and your organization have provided to the hacks of the world, such as myself, so that we may continue enjoying the art of annoying people at bars & clubs with our LOUD rock 'n roll. Thank You again.
[ - ]
Comment by aravindhaMay 21, 2009
Insightful... Continue blogging.
[ - ]
Comment by superhotmuffinDecember 30, 2012
I've recently discovered that oversampling has a (very) significant impact on the quality and quantity of distortion produced by non-linearities in my guitar amp models. Changing the the oversampling ratio and, more significantly, the transition bandwidth in the wsinc anti-imaging and anti-aliasing filters, significantly impacts the amount and quality of the overdrive produced by the non-linearity. I can literally take a model that sounds like a Mesa Boogie Triple Rectifier (super metal head amp) and turn it into a Fender Twin by changing nothing more than the filter BW. Hmm, go figure!
[ - ]
Comment by graycNovember 16, 2010
Haven't read the whole blog yet. I'm stuck on a couple class questions and need some help from the pro's. Not sure what 'stereo linking' refers to. Or 'Gain Makeup'. We're using Hubers Modern Recording techniques and I haven't been able to find anything! Help please! Thanks.
[ - ]
Comment by Gaurav HansdaJanuary 15, 2011
Really good blog. I have exactly same interest as yours. I am currently pursuing Masters in EE. I would like to involve more with Audio/Video signal processing. Could you please guide me?
[ - ]
Comment by deepanwitasarMay 12, 2012
really good blog. sir, i am student of m.tech(ece). i am working in software defined radio in sysgen xillinx 12.3. can you help me sir, how i eleminate the noise after geatting the final output analog siganl.can you suggest any block or procedure of eleminate the noise from analog signal...its urgents.....could you please guide me???????????
[ - ]
Comment by 15MT000048September 5, 2016
sir, i am pursuing M.Tech in ECE. My project topic is implementation of code excited linear prediction (CELP) in real time using 6416 DSP KIT. i am unable to understand vector quantization and codebook design.please help me.
Thanking you sir.

To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.

Please login (on the right) if you already have an account on this platform.

Otherwise, please use this form to register (free) an join one of the largest online community for Electrical/Embedded/DSP/FPGA/ML engineers: