Music/Audio Signal Processing

• Audio DSP
• Academia / Research

Greetings,

This is my blog from the point of view of a music/audio DSP research engineer / educator. It is informal and largely nontechnical because nearly everything I have to say about signal processing is (or will be) somewhere in my four-book series: Mathematics of DFT with Audio Applications, Introduction to Digital Filters, Physical Audio Signal Processing and Spectral Audio Signal Processing

I thought I would start out with a short history of how I got involved in music/audio DSP, and perhaps thereby provide some insights into that field. After that, I will say a few words about what I consider to be interesting current developments related to music/audio DSP.

One Path to a Music/Audio Signal Processing Research Career

My story is basically that of an engineer who is passionate about music. I know there are a lot of you out there, and I get to know quite a few among the CCRMA graduate students. In high school, I was a pretty serious musician while keeping my grades up in math and science. At Rice, I worked harder as a student, but I kept up my music side playing semi-professionally in my spare time. There was no connection between music and engineering through this time --- music was something I pursued on my own time, and at school I simply worked at being a good student. A very significant influence at Rice was the DSP and Control curricula there, especially the courses of C. Sidney Burrus and J. Boyd Pearson. That technical foundation served me very well in later years.

Upon graduation, I took a job at ESL in Sunnyvale CA working on digital signal processing applied to communications (mostly at sea). (ESL regularly recruited at Rice University at that time.) This was a great learning experience for me, but still it had nothing (directly) to do with music. I found other musicians through the Bay Area Music guide, and played in several bands for fun and occasional profit. I also built a few electronic effects projects (all analog at that time). ESL had an Honors Coop program that allowed me to take one or two courses per quarter at Stanford toward an MS/EE degree and beyond.

CCRMA co-founder James A. Moorer presented one of the EE 201 seminars, and that motivated me to visit CCRMA (then just a small project at the Artificial Intelligence Lab) to see how I might get involved. I volunteered hardware and software assistance in return for a free computer account at the AI Lab for purposes of learning about computer music. In parallel I was taking 300-level courses in signal processing in EE at Stanford. There was still no formal connection between my academic life and music life, although my music side had taken an academic turn. I did less playing as a musician and more studies related to computer music in my spare time (although I kept playing solo classical guitar, and was an avid Bach student).

After two years at ESL, I applied for and was awarded a Hertz fellowship which enabled me to go back to school full time. For the next five years I was a full-time graduate student (except for summer jobs related to signal processing, which I highly recommend to all graduate students). The greatest thing about fellowship support is that one can choose any reasonable research topic. I therefore chose a problem of importance in music technology: violin synthesis. At the time, one could not purchase a good bowed-string synthesizer for any amount of money, so I figured it must be a problem hard enough to warrant a research effort. In fact, it is hard enough to warrant quite a few dissertations!

The first two years or so of the PhD/EE program were dominated by taking courses. Then and before at Rice, I chose courses that I thought could be useful for music/audio technology down the road. In other words, potential music technology applicability was my most consistent criterion for deciding what courses to take and which research topics to pursue. (I also took some courses out of pure interest, of course.) The most valuable courses for me at Stanford were those put together by Tom Kailath and Joe Goodman. Prof. Kailath's research seminars were especially enlightening.

During and especially after taking courses, one has to master a branch of the current research literature (which involves reading a lot of papers and misc. books in the field). My area of emphasis was digital filter design and system identification (with violin modeling firmly in the back of my mind). In my third year, music applications of DSP were finally starting to pop up in my research (you get ideas while reading the literature, and sometimes a promising idea remains unexplored after you seem to have finished reading all relevant literature). So, at this point, I started to publish conference papers and go to one or so conferences per year (sometimes self-funded - ouch!).

Shortly before graduating, I took a job in the Adaptive Signal Processing Dept. of Systems Control Technology (SCT), working for Ben Friedlander, a well known researcher in statistical signal processing. I met Ben when he was doing a little teaching at Stanford in the area of statistical signal processing (a former Kailath student). This was another great learning experience for me, and I considered it more of a "postdoc" than a job. There was no direct connection with music, but I continued to publish regularly in the computer music literature on the side.

After about two years working at SCT, I was offered a part-time research associate position at CCRMA, funded by a System Development Foundation grant. (I'm told John Pierce had a lot to do with getting that five-year grant.) During that time I started my signal processing courses for music research, and continued my earlier research on violin modeling (which led to the development of waveguide synthesis). Finally I had a real position that was officially in service to music! The funding was temporary, so I kept my job at SCT going at the half-time level.

After another couple of years, I received a call from Steve Jobs saying that the NeXT computer was going to have a Motorola DSP56001 chip on every motherboard, and that they were looking for someone to work on software for signal processing, and he was especially interested in having some good music capabilities. I attribute my "visibility" to being one of the very few PhD/EEs publishing in the music technology field at the time. I started at NeXT in 1986 as the 25th employee, and was the founding member of the NeXT sound, music, and signal processing group. (My official title was simply "Software Engineer", like everyone else in the NeXT software division.) Thus, my NeXT position replaced my SCT position as a music-related engineering (and management) position. During that time, I went down to one day per week at CCRMA, just enough to keep my courses going.

In 1989, I was promoted to Associate Professor (Research) at CCRMA, thanks entirely to the efforts of John Chowning (founding director of CCRMA). As music became less important at NeXT, I transitioned back to CCRMA full time where I have been ever since. CCRMA's financial health at that time and since was primarily due to the success of the FM synthesis patents by John Chowning and Yamaha Corporation.

So, what can one glean from this story in terms of music-technology career advice? First of all, opportunities are unpredictable! I did not foresee any of the career opportunities I ultimately pursued. My "big picture" was pretty simple at all times. First, I was simply getting the best education I could, choosing courses according to "likely future music technology relevance". Next, I was working in the best job I could find that could help me develop more skills that seemed applicable to music/audio research. After the PhD, it was also important that the job allow me to publish at least annually, and that it support some conference attendance. In parallel, I was pursuing music in my own way (with or without much technology).

There are of course many more opportunities in music/audio DSP today than there were when I was a student. It is even possible to develop and sell plugins over the Internet as an individual these days. However, I still recommend pursuit of relationships with the best known institutions in one's chosen field. As the Internet grows, it becomes more and more important to have name/brand recognition, simply because there is too much out there to look at. Already the number of free LADSPA plugins is more than one can try out in a reasonable length of time.

Exciting Recent Developments

On the recent developments front, I have been having a very good time working in the Faust programming language. See, for example,

https://ccrma.stanford.edu/~jos/aspf/

and

http://ccrma.stanford.edu/realsimple/faust_strings/

Another exciting recent development for me is Octave 3.0. It is now amazingly more compatible with Matlab (including handle graphics), and Octave Forge has continued to develop strongly as well. I believe it is now true that I am using 100% free open-source software for all of my research and publication.

Why is using free open-source software important to me? Well, since this is my blog, I'll tell you: Today's engineer needs to be able to "tool up and produce" on an extremely wide variety of fronts in a short time. The monolithic software purchase model inhibits this, while the free, open-source model supports it. For example, I may need a "spread sheet" 3 or 4 times a year. I shouldn't have to spend the same amount of money on Microsoft Excel as someone who uses it every day. I don't mind paying for software, but I feel I should pay in proportion to how much I use it.

About a decade ago, there was a movement toward "micropayment". One of the companies was "Millicent" who claimed that purchases down to a tenth of a cent were economically practical. I don't know why this did not take hold, but I think it is the way to go for commercial software and information dissemination. I think the reason may be simply that the "subscription model" is more profitable, because most subscribers pay even when they don't use the tool or service. Companies know this and they are likely resisting micropayment models as a result.

Of course, in advertising, micropayment has taken off in a big way (witness Google). As Google provides more and more online applications (Google Office, etc.), they will effectively convert software to the micropayment model (supported by ads), and that will be pretty good. However, I would rather give metered micropayment and save the ads for when I'm actually shopping. Also, I don't want to run programs over the Web. I want them installed on my own recently upgraded machine, and I want them to be blazingly fast and instantly responsive. I don't want to depend on a network connection involving several routers and servers in the path.

Fortunately, the free, open-source software revolution is filling the bill. With Fedora Red Hat Linux, the GNU tools, Octave, Pd, Faust, JACK, Ardour, Rosegarden, Open Office, TeX, LaTeX, and the huge variety of additional packages available (the Planet CCRMA distribution for Fedora/RedHat contains many of the music/audio related packages), I can function without having to pay exorbitant subscription fees for things I rarely use. When I need to solve a new problem, I search the Web, download new tools, study the documentation as needed, and keep things moving in one sitting (without entering my credit-card number all the time). Such accelerated, cross-disciplinary problem-solving increases overall productivity, and in my opinion this should be a top priority until everyone in the world has a decent standard of living and population growth is slower than economic growth. In other words, Job 1 for all of us should be "creating wealth" in our own way, and free open-source software contributes significantly to this goal for software engineers.

Cheers,

Julius

Sign in