Forums

Interesting Audio Processing

Started by d99n April 3, 2008
Hi,

I've always been interested in DSP methods to decompose a piece of music
into discrete instruments.  I came across this new tool that's pretty
fascinating:

http://www.celemony.com/cms/index.php?id=dna&L=0

How do you think something like this works?  Considering an acoustic
guitar chord and all its intermixed, varying and slightly non-integer
harmonics, how would one go about separating such a signal into individual
notes that actually sound pretty clean?

/Dan


On Apr 3, 6:20&#2013266080;am, "d99n" <dan_led...@yahoo.com> wrote:
> Hi, > > I've always been interested in DSP methods to decompose a piece of music > into discrete instruments. &#2013266080;I came across this new tool that's pretty > fascinating: > > http://www.celemony.com/cms/index.php?id=dna&L=0 > > How do you think something like this works? &#2013266080;Considering an acoustic > guitar chord and all its intermixed, varying and slightly non-integer > harmonics, how would one go about separating such a signal into individual > notes that actually sound pretty clean? > > /Dan
cross posted to rec.audio.pro Mark
> Hi, > > I've always been interested in DSP methods to decompose a piece of music > into discrete instruments. I came across this new tool that's pretty > fascinating: > > http://www.celemony.com/cms/index.php?id=dna&L=0
Again! This is all coming across like an advert, which is losing integrity by the minute! : )
> How do you think something like this works? Considering an acoustic > guitar chord and all its intermixed, varying and slightly non-integer > harmonics, how would one go about separating such a signal into individual > notes that actually sound pretty clean? > > /Dan
cross posted to rec.audio.pro Mark
On Apr 3, 6:20 am, "d99n" <dan_led...@yahoo.com> wrote:
> Hi, > > I've always been interested in DSP methods to decompose a piece of music > into discrete instruments. I came across this new tool that's pretty > fascinating: > > http://www.celemony.com/cms/index.php?id=dna&L=0 > > How do you think something like this works? Considering an acoustic > guitar chord and all its intermixed, varying and slightly non-integer > harmonics, how would one go about separating such a signal into individual > notes that actually sound pretty clean?
of course i don't know exactly how they did it. i doubt that they are assuming any non-integer harmonics. if it were non-real-time, i could imaging processing the audio with a set of cascaded, adjustable comb- filters (where the notches are at the harmonic frequencies) until they get something that is residual. dealing with the attacks is another problem. dealing with harmonics of two different notes that land on top of each other is another problem. lot'sa problems. dunno how they dealt with them all. but, from what i have seen in Electronic Musician, i think this is no bogus product. i think it's real. i'm looking forward to finding their booth at the AES show in October (they better the hell be there). r b-j
robert bristow-johnson wrote:
> On Apr 3, 6:20 am, "d99n" <dan_led...@yahoo.com> wrote: >> Hi, >> >> I've always been interested in DSP methods to decompose a piece of music >> into discrete instruments. I came across this new tool that's pretty >> fascinating: >> >> http://www.celemony.com/cms/index.php?id=dna&L=0 >> >> How do you think something like this works? Considering an acoustic >> guitar chord and all its intermixed, varying and slightly non-integer >> harmonics, how would one go about separating such a signal into individual >> notes that actually sound pretty clean? > > of course i don't know exactly how they did it. i doubt that they are > assuming any non-integer harmonics. if it were non-real-time, i could > imaging processing the audio with a set of cascaded, adjustable comb- > filters (where the notches are at the harmonic frequencies) until they > get something that is residual. dealing with the attacks is another > problem. dealing with harmonics of two different notes that land on > top of each other is another problem. > > lot'sa problems. dunno how they dealt with them all. but, from what > i have seen in Electronic Musician, i think this is no bogus product. > i think it's real. i'm looking forward to finding their booth at the > AES show in October (they better the hell be there).
I'd agree with your assessment of "lot'sa problems" if I hadn't heard the short talk by the guy behind it. He makes it sound like something clicked in his mind, and made him realise that something which was not rigorous was going to fail with sufficient rarity for it to useful. He makes it sound like the magic trick was one key insight. Steve
>>> Hi, >>> >>> I've always been interested in DSP methods to decompose a piece of music >>> into discrete instruments. I came across this new tool that's pretty >>> fascinating: >>> >>> http://www.celemony.com/cms/index.php?id=dna&L=0 >>> >>> How do you think something like this works? Considering an acoustic >>> guitar chord and all its intermixed, varying and slightly non-integer >>> harmonics, how would one go about separating such a signal into >>> individual >>> notes that actually sound pretty clean? >> >> of course i don't know exactly how they did it. i doubt that they are >> assuming any non-integer harmonics. if it were non-real-time, i could >> imaging processing the audio with a set of cascaded, adjustable comb- >> filters (where the notches are at the harmonic frequencies) until they >> get something that is residual. dealing with the attacks is another >> problem. dealing with harmonics of two different notes that land on >> top of each other is another problem. >> >> lot'sa problems. dunno how they dealt with them all. but, from what >> i have seen in Electronic Musician, i think this is no bogus product. >> i think it's real. i'm looking forward to finding their booth at the >> AES show in October (they better the hell be there). > > I'd agree with your assessment of "lot'sa problems" if I hadn't heard the > short talk by the guy behind it. He makes it sound like something clicked > in his mind, and made him realise that something which was not rigorous > was going to fail with sufficient rarity for it to useful. He makes it > sound like the magic trick was one key insight.
Well, you've got to start somewhere. I've been looking at Continuous Wavelet's, and as the data is pre-processed I guess you could do a lot in the analysis stage. How about a CWT at note resolution pitches? It would give you the fundamentals, and a nice hi-res spectrum. If it's designed mainly for one instrument, it could use different number of notes played at once to figure out the harmonics of each through-out the piece. But like someone above said, there's a lot of potential breakages, and it completely depends on the source material. VC
Steve Underwood wrote:
> robert bristow-johnson wrote: >> On Apr 3, 6:20 am, "d99n" <dan_led...@yahoo.com> wrote: >>> >>> I've always been interested in DSP methods to decompose a piece of >>> music into discrete instruments. I came across this new tool that's >>> pretty fascinating: >>> >>> http://www.celemony.com/cms/index.php?id=dna&L=0 >>> >>> How do you think something like this works? Considering an acoustic >>> guitar chord and all its intermixed, varying and slightly non-integer >>> harmonics, how would one go about separating such a signal into >>> individual notes that actually sound pretty clean? >> >> of course i don't know exactly how they did it. i doubt that they are >> assuming any non-integer harmonics. [...] >> >> lot'sa problems. dunno how they dealt with them all. but, from what i >> have seen in Electronic Musician, i think this is no bogus product. i >> think it's real. [...] > > I'd agree with your assessment of "lot'sa problems" if I hadn't heard > the short talk by the guy behind it. He makes it sound like something > clicked in his mind, and made him realise that something which was not > rigorous was going to fail with sufficient rarity for it to useful. He > makes it sound like the magic trick was one key insight.
I watched the video too, and didn't get that impression at all. Here's my take on it, based on what is said and shown in the video, combined with some educated guessing on my part: He said that what got him started is that he realized that the problem was impossible in theory (and I agree), but in practice, he could adopt a looser, less disciplined approach, and actually "solve" the problem. I at least like his pragmatic, "cut the Gordian knot" attitude. I've solved several Rubik's Cubes simply by taking them apart and putting them back together. It's a lot easier! ;-) And that may be somewhat like the developer's approach. I'm guessing the method is roughly: 1. FFT analysis of the track. 2. Heuristic separation of the components and assignment of them to notes. (Note: doesn't require assumptions about integral harmonics, it just lumps all the arbitrarily-separated time-domain data together in groups using an arbitrary method.) 3. Allow editing. 4. Resynthesis. Ever looked at one of those 3-D FFT plots, and noticed how you can see the harmonics of the note, or generally, the components of the sound? In theory, there's no absolute way to define any of that to an original, single sound source. But, what if you just guess? Maybe there's a method of guessing that works well enough most of the time -- at least well enough to produce a useful end result. I think this problem is conceptually not completely unlike selecting objects in Photoshop. There are several different plugins available to help with it, in addition to Photoshop's own selection functions. Some work better than others, on different images. There is ongoing development in the area. None of them work perfectly all the time, and all involve some "destruction" of part of the image (at the edges of the selection). BUT, graphic designers can do amazing things with the functionality, and the end results validate the methods. At the end of the video and on the web page, they say the Direct Note Access feature will be introduced "Coming Soon" in August 2008. It looks slick in the demo, but Aug. 2008 is several months away, and we all know what "coming soon" means in the field of software development: possible vaporware, will probably be delayed, and in any case, the first version may look and act more like a proof of concept, to be followed by many updates. I don't expect a lot from the first release, but at some point, I can imagine myself using it and doing really cool things! Jay Ts -- To contact me, use this web page: http://www.jayts.com/contact.php
"VelociChicken" <bob@yahoob.com> wrote in
news:0GmJj.25$Wf3.4@newsfe16.ams2: 

>>>> Hi, >>>> >>>> I've always been interested in DSP methods to decompose a piece of >>>> music into discrete instruments. I came across this new tool that's >>>> pretty fascinating: >>>> >>>> http://www.celemony.com/cms/index.php?id=dna&L=0 >>>> >>>> How do you think something like this works? Considering an acoustic >>>> guitar chord and all its intermixed, varying and slightly non-integer >>>> harmonics, how would one go about separating such a signal into >>>> individual >>>> notes that actually sound pretty clean? >>> >>> of course i don't know exactly how they did it. i doubt that they are >>> assuming any non-integer harmonics. if it were non-real-time, i could >>> imaging processing the audio with a set of cascaded, adjustable comb- >>> filters (where the notches are at the harmonic frequencies) until they >>> get something that is residual. dealing with the attacks is another >>> problem. dealing with harmonics of two different notes that land on >>> top of each other is another problem. >>> >>> lot'sa problems. dunno how they dealt with them all. but, from what >>> i have seen in Electronic Musician, i think this is no bogus product. >>> i think it's real. i'm looking forward to finding their booth at the >>> AES show in October (they better the hell be there). >> >> I'd agree with your assessment of "lot'sa problems" if I hadn't heard >> the short talk by the guy behind it. He makes it sound like something >> clicked in his mind, and made him realise that something which was not >> rigorous was going to fail with sufficient rarity for it to useful. He >> makes it sound like the magic trick was one key insight. > > Well, you've got to start somewhere. I've been looking at Continuous > Wavelet's, and as the data is pre-processed I guess you could do a lot > in the analysis stage. How about a CWT at note resolution pitches? It > would give you the fundamentals, and a nice hi-res spectrum. > If it's designed mainly for one instrument, it could use different > number of notes played at once to figure out the harmonics of each > through-out the piece. But like someone above said, there's a lot of > potential breakages, and it completely depends on the source material. > VC > > > >
I have never tried to solve this problem, but it seems to me a cepstrum approach might be useful. A cepstrum (spectrum spelled sideways), is a forward FFT of an FFT power spectrum. The bins represents harmonics of the original FFT. It's been a long time since I looked at cepstrum, but I know that they have been used to help isolate components in mechanical systems like gear boxes. I think their might be some old Bruel & Kjaer application notes on this idea. Al Clark Danville Signal Processing, Inc.
Al Clark wrote:
> "VelociChicken" <bob@yahoob.com> wrote in > news:0GmJj.25$Wf3.4@newsfe16.ams2: > >>>>> Hi, >>>>> >>>>> I've always been interested in DSP methods to decompose a piece of >>>>> music into discrete instruments. I came across this new tool that's >>>>> pretty fascinating: >>>>> >>>>> http://www.celemony.com/cms/index.php?id=dna&L=0 >>>>> >>>>> How do you think something like this works? Considering an acoustic >>>>> guitar chord and all its intermixed, varying and slightly non-integer >>>>> harmonics, how would one go about separating such a signal into >>>>> individual >>>>> notes that actually sound pretty clean? >>>> of course i don't know exactly how they did it. i doubt that they are >>>> assuming any non-integer harmonics. if it were non-real-time, i could >>>> imaging processing the audio with a set of cascaded, adjustable comb- >>>> filters (where the notches are at the harmonic frequencies) until they >>>> get something that is residual. dealing with the attacks is another >>>> problem. dealing with harmonics of two different notes that land on >>>> top of each other is another problem. >>>> >>>> lot'sa problems. dunno how they dealt with them all. but, from what >>>> i have seen in Electronic Musician, i think this is no bogus product. >>>> i think it's real. i'm looking forward to finding their booth at the >>>> AES show in October (they better the hell be there). >>> I'd agree with your assessment of "lot'sa problems" if I hadn't heard >>> the short talk by the guy behind it. He makes it sound like something >>> clicked in his mind, and made him realise that something which was not >>> rigorous was going to fail with sufficient rarity for it to useful. He >>> makes it sound like the magic trick was one key insight. >> Well, you've got to start somewhere. I've been looking at Continuous >> Wavelet's, and as the data is pre-processed I guess you could do a lot >> in the analysis stage. How about a CWT at note resolution pitches? It >> would give you the fundamentals, and a nice hi-res spectrum. >> If it's designed mainly for one instrument, it could use different >> number of notes played at once to figure out the harmonics of each >> through-out the piece. But like someone above said, there's a lot of >> potential breakages, and it completely depends on the source material. >> VC >> > > I have never tried to solve this problem, but it seems to me a cepstrum > approach might be useful. > > A cepstrum (spectrum spelled sideways), is a forward FFT of an FFT power > spectrum. The bins represents harmonics of the original FFT. > > It's been a long time since I looked at cepstrum, but I know that they have > been used to help isolate components in mechanical systems like gear boxes. > I think their might be some old Bruel & Kjaer application notes on this > idea.
Cepstrums are widely used in speech analysis for isolating components, too. Thats not a million kilometres away from music analysis, so cepstral analysis might well be involved. Steve
Jay Ts wrote:
> Steve Underwood wrote: >> robert bristow-johnson wrote: >>> On Apr 3, 6:20 am, "d99n" <dan_led...@yahoo.com> wrote: >>>> I've always been interested in DSP methods to decompose a piece of >>>> music into discrete instruments. I came across this new tool that's >>>> pretty fascinating: >>>> >>>> http://www.celemony.com/cms/index.php?id=dna&L=0 >>>> >>>> How do you think something like this works? Considering an acoustic >>>> guitar chord and all its intermixed, varying and slightly non-integer >>>> harmonics, how would one go about separating such a signal into >>>> individual notes that actually sound pretty clean? >>> of course i don't know exactly how they did it. i doubt that they are >>> assuming any non-integer harmonics. [...] >>> >>> lot'sa problems. dunno how they dealt with them all. but, from what i >>> have seen in Electronic Musician, i think this is no bogus product. i >>> think it's real. [...] >> I'd agree with your assessment of "lot'sa problems" if I hadn't heard >> the short talk by the guy behind it. He makes it sound like something >> clicked in his mind, and made him realise that something which was not >> rigorous was going to fail with sufficient rarity for it to useful. He >> makes it sound like the magic trick was one key insight. > > I watched the video too, and didn't get that impression at all. > > Here's my take on it, based on what is said and shown in the > video, combined with some educated guessing on my part: > > He said that what got him started is that he realized that the > problem was impossible in theory (and I agree), but in practice, > he could adopt a looser, less disciplined approach, and actually > "solve" the problem. > > I at least like his pragmatic, "cut the Gordian knot" attitude. > I've solved several Rubik's Cubes simply by taking them apart > and putting them back together. It's a lot easier! ;-) > > And that may be somewhat like the developer's approach. I'm guessing > the method is roughly: > > 1. FFT analysis of the track. > 2. Heuristic separation of the components and assignment > of them to notes. (Note: doesn't require assumptions about > integral harmonics, it just lumps all the arbitrarily-separated > time-domain data together in groups using an arbitrary method.) > 3. Allow editing. > 4. Resynthesis. > > Ever looked at one of those 3-D FFT plots, and noticed how you > can see the harmonics of the note, or generally, the components > of the sound? In theory, there's no absolute way to define any > of that to an original, single sound source.
I think anyone who has been processing signals for a long time has many times stared at a visual representation of a signal and thought "if the damn thing looks so bloody obvious there, why I am I struggling so hard to come up with an algorithm to detect or extract it reliably". :-)
> But, what if you just guess? Maybe there's a method of guessing > that works well enough most of the time -- at least well enough > to produce a useful end result. > > I think this problem is conceptually not completely unlike > selecting objects in Photoshop. There are several different > plugins available to help with it, in addition to Photoshop's > own selection functions. Some work better than others, on > different images. There is ongoing development in the area. > None of them work perfectly all the time, and all involve > some "destruction" of part of the image (at the edges of > the selection). BUT, graphic designers can do amazing > things with the functionality, and the end results > validate the methods. > > At the end of the video and on the web page, they say the > Direct Note Access feature will be introduced "Coming Soon" > in August 2008. It looks slick in the demo, but Aug. 2008 > is several months away, and we all know what "coming soon" > means in the field of software development: possible vaporware, > will probably be delayed, and in any case, the first version > may look and act more like a proof of concept, to be followed > by many updates. > > I don't expect a lot from the first release, but at some > point, I can imagine myself using it and doing really > cool things!
There appear to be impartial reports that this is the real deal, demonstrable, and in the final tidying stages for market. Steve