Compared to several years ago on comp.dsp I see a huge increase in people asking for help (and code) who don't have a clue what they are doing. If someone gives you code that you don't understand, how do you know if it is right? Do you just believe it? How do you test it? How do you know if your final implementation is working as well as it is supposed to? I have run into DSP code in fielded and about to be fielded systems (including pilot rescue radio and sonar processing system), written by people who didn't know enough about what they were doing to know how to test code that people's lives depended on. The implementations had major problems with them that would not show up on a test bench at high SNR, but exhibited UNRECOGNIZED impaired performance in the field. comp.dsp is not the first place people should go to solve their problems. Comments? Dirk
How do you know it is right?
Started by ●August 23, 2005
Reply by ●August 23, 20052005-08-23
dbell wrote:> Compared to several years ago on comp.dsp I see a huge increase in > people asking for help (and code) who don't have a clue what they are > doing. If someone gives you code that you don't understand, how do you > know if it is right? Do you just believe it? How do you test it? How do > you know if your final implementation is working as well as it is > supposed to? I have run into DSP code in fielded and about to be > fielded systems (including pilot rescue radio and sonar processing > system), written by people who didn't know enough about what they were > doing to know how to test code that people's lives depended on. The > implementations had major problems with them that would not show up on > a test bench at high SNR, but exhibited UNRECOGNIZED impaired > performance in the field. > > comp.dsp is not the first place people should go to solve their > problems. > > Comments? > > Dirk >I was going to say that I wouldn't use code that I didn't thoroughly understand, but then I realized that I do all the time -- I use RTOS code, matrix manipulation code, serial libraries, etc., that I don't know personally. However, when I use such code I _do_ understand what it's _supposed_ to do, so I can write some test vectors for it and I can know when I'm using it if it's misbehaving. Certainly in the case of signal processing algorithms I'm going to know what to look for and in my arrogance about my abilities in this area I'm probably going to feel most comfortable with the code if I've either written it or thoroughly analyzed it. If you _don't_ know enough about some piece of code to adequately test it then you should realize this. Often I don't think that people's imaginations can stretch to the point of believing that there's a difference between "working kinda on the bench" and "working adequately in the field". I think it's fine for a toaster or an iPod to not know how to test what you're working on, but for life-critical stuff you should _know_. I'm somewhat dismayed by the state of life-critical software verification. Avionics, medical systems, some automotive systems (brake-by-wire, anyone?), and other things like your pilot locators can all be life critical, yet there is no consistent, across-the-board set of standards for life-critical software testing. I feel that such testing should be more unified, so that no matter what the application there is a consistent way of categorizing the impact of a bug, determining a level of criticality, and therefore determining the level of scrutiny required. As to your last question: I think that newsgroups are a good place to go for ideas and clarifications, but you shouldn't just cobble a product together from things you read in newsgroups. Whatever notions you may get from a newsgroup you should fully absorb and understand before you go using it for anything. So this may be the first place to look for information, but it shouldn't be the last word. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com
Reply by ●August 23, 20052005-08-23
On Tue, 23 Aug 2005 12:52:10 -0700, Tim Wescott <tim@seemywebsite.com> wrote:>dbell wrote:...>> comp.dsp is not the first place people should go to solve their >> problems. >> >> Comments? >> >> Dirk...>I'm somewhat dismayed by the state of life-critical software >verification. Avionics, medical systems, some automotive systems >(brake-by-wire, anyone?), and other things like your pilot locators can >all be life critical, yet there is no consistent, across-the-board set >of standards for life-critical software testing.... I used to work on avionics for airliners, and that's some of the most heavily tested (formally) and verified software there is. And, at least at that time, there were different levels that distinguished higher standards for life-critical or otherwise. IMHO it was a good process, and some of the best test methodology training I ever got was for that application.>As to your last question: I think that newsgroups are a good place to >go for ideas and clarifications, but you shouldn't just cobble a product >together from things you read in newsgroups. Whatever notions you may >get from a newsgroup you should fully absorb and understand before you >go using it for anything. So this may be the first place to look for >information, but it shouldn't be the last word.Agreed. Anybody who just comes to comp.dsp, rips out some publically offered code and puts it in a product has some serious process issues that they need to iron out. Advice from comp.dsp is free, and while the SNR is quite high in this particular newsgroup any free advice should be taken for what it's worth. And not testing or verifying code that comes from the internet and goes in a product is, in my personal opinion, essentially negligence. Not to mention that not testing and verifying code in general is pretty dumb, again, IMHO. Eric Jacobsen Minister of Algorithms, Intel Corp. My opinions may not be Intel's opinions. http://www.ericjacobsen.org
Reply by ●August 23, 20052005-08-23
If you want to read about something really scary about using only software. Read the following account http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.html Clay "dbell" <dirkman@erols.com> wrote in message news:1124823549.425187.88220@z14g2000cwz.googlegroups.com...> Compared to several years ago on comp.dsp I see a huge increase in > people asking for help (and code) who don't have a clue what they are > doing. If someone gives you code that you don't understand, how do you > know if it is right? Do you just believe it? How do you test it? How do > you know if your final implementation is working as well as it is > supposed to? I have run into DSP code in fielded and about to be > fielded systems (including pilot rescue radio and sonar processing > system), written by people who didn't know enough about what they were > doing to know how to test code that people's lives depended on. The > implementations had major problems with them that would not show up on > a test bench at high SNR, but exhibited UNRECOGNIZED impaired > performance in the field. > > comp.dsp is not the first place people should go to solve their > problems. > > Comments? > > Dirk >
Reply by ●August 23, 20052005-08-23
in article Q3NOe.49462$rp.41853@bignews1.bellsouth.net, Clay S. Turner at Physics@Bellsouth.net wrote on 08/23/2005 18:06:> If you want to read about something really scary about using only software. > Read the following account > > http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.htmli remember when that happened. i remember reading something about it at the time (i sure thought it was in some IEEE mag) and i think the article was entitled: "Death by Shared Variable" very memorable title, IMO. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●August 23, 20052005-08-23
Eric Jacobsen wrote:> On Tue, 23 Aug 2005 12:52:10 -0700, Tim Wescott <tim@seemywebsite.com> > wrote: > > >>dbell wrote: > > > ... > > >>>comp.dsp is not the first place people should go to solve their >>>problems. >>> >>>Comments? >>> >>>Dirk > > > ... > > >>I'm somewhat dismayed by the state of life-critical software >>verification. Avionics, medical systems, some automotive systems >>(brake-by-wire, anyone?), and other things like your pilot locators can >>all be life critical, yet there is no consistent, across-the-board set >>of standards for life-critical software testing. > > > ... > > I used to work on avionics for airliners, and that's some of the most > heavily tested (formally) and verified software there is. And, at > least at that time, there were different levels that distinguished > higher standards for life-critical or otherwise. IMHO it was a good > process, and some of the best test methodology training I ever got was > for that application. >The FAA probably does the best job with DO-178. The FDA does pretty well, but they use a completely different set of rules to achieve the same thing. While there's no regulation at all for antilock brakes the car manufacturer's know which side their bread is buttered on and they have their own set of self-imposed rules which are different from FAA or FDA. What irks me is that it's all a hodge-podge, and the standard isn't "if your software can kill this is what you do" it's more "if you make this then do this, if you make that then do that, if you make this other thing then no one is going to pay any attention at all until men, women and children start dying". I know I'm ranting, but still. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com
Reply by ●August 24, 20052005-08-24
robert bristow-johnson wrote:> in article Q3NOe.49462$rp.41853@bignews1.bellsouth.net, Clay S. Turner at > Physics@Bellsouth.net wrote on 08/23/2005 18:06: > > >>If you want to read about something really scary about using only software. >>Read the following account >> >>http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.html > > > i remember when that happened. i remember reading something about it at the > time (i sure thought it was in some IEEE mag) and i think the article was > entitled: > > "Death by Shared Variable" > > very memorable title, IMO.I'm too lazy to read the whole article now. I too remember when it happened, and I learned from it. The problem boiled down to losing sync between the displayed dose and the internally programmed dose because the buggy keypad editing routine changed the first without changing the second. I learned from that In a furnace controller, I use a read-back LCD display. The keypad entry and editing is done on the display only. When the operator says "go", the processor reads the display, converts the digits to binary, and uses the number. There is nothing to get out of sync, and each step is easily tested in isolation. Even if the entry and editing routines were buggy, the operator gets what he sees. (Displayed characters are returned as ASCII and interpreted by a state machine. Disallowed characters abort the operation.) Of course the code should be free of bugs, but it's important that the design not respond inappropriately if there is one. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●August 24, 20052005-08-24
dbell wrote:> Compared to several years ago on comp.dsp I see a huge increase in > people asking for help (and code) who don't have a clue what they are > doing. If someone gives you code that you don't understand, how do you > know if it is right? Do you just believe it? How do you test it? How do > you know if your final implementation is working as well as it is > supposed to? I have run into DSP code in fielded and about to be > fielded systems (including pilot rescue radio and sonar processing > system), written by people who didn't know enough about what they were > doing to know how to test code that people's lives depended on. The > implementations had major problems with them that would not show up on > a test bench at high SNR, but exhibited UNRECOGNIZED impaired > performance in the field. > > comp.dsp is not the first place people should go to solve their > problems.There are categories that look similar. 1. "My boss said I had to program in speaker recognition. I know diddly about DSP. What should I do?" Sometimes this is disguised by a little prior research, so that it's sprinkled with buzzwords. It's not too hard to see through when those are strung together in a meaningless order. 2. "I wanna make cool FX. What should I do?" An amateur who thinks that such things are easier than they are, but might be up to them with some guidance (that he doesn't know enough to ask about specifically). 3a. A student who just wants to get a grade so he can get a job or 3b. one of those students who got a job and now has to deliver. I fear for the next generation. Be warned: I've been saying that for fifty years. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
Reply by ●August 24, 20052005-08-24
in article c8idnWOwoJpLHJHeRVn-1Q@rcn.net, Jerry Avins at jya@ieee.org wrote on 08/24/2005 10:18:> robert bristow-johnson wrote: >> in article Q3NOe.49462$rp.41853@bignews1.bellsouth.net, Clay S. Turner at >> Physics@Bellsouth.net wrote on 08/23/2005 18:06: >> >> >>> If you want to read about something really scary about using only software. >>> Read the following account >>> >>> http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.html >> >> >> i remember when that happened. i remember reading something about it at the >> time (i sure thought it was in some IEEE mag) and i think the article was >> entitled: >> >> "Death by Shared Variable" >> >> very memorable title, IMO. > > I'm too lazy to read the whole article now. I too remember when it > happened, and I learned from it. The problem boiled down to losing sync > between the displayed dose and the internally programmed dose because > the buggy keypad editing routine changed the first without changing the > second. I learned from thati also thought that one of the lessons is that for every editable variable, there should be only one method that "owns" it and may write to it. other dudes can read it (and in fact should read it every time it's updated) but only one dude may write it. i thought there was something about that important variable getting written over by method #2 and method #1 (just previously having written to that variable itself) was oblivious of the change and something bad happened. i dunno. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."
Reply by ●August 24, 20052005-08-24
robert bristow-johnson wrote:> in article c8idnWOwoJpLHJHeRVn-1Q@rcn.net, Jerry Avins at jya@ieee.org wrote > on 08/24/2005 10:18: > > >>robert bristow-johnson wrote: >> >>>in article Q3NOe.49462$rp.41853@bignews1.bellsouth.net, Clay S. Turner at >>>Physics@Bellsouth.net wrote on 08/23/2005 18:06: >>> >>> >>> >>>>If you want to read about something really scary about using only software. >>>>Read the following account >>>> >>>>http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.html >>> >>> >>>i remember when that happened. i remember reading something about it at the >>>time (i sure thought it was in some IEEE mag) and i think the article was >>>entitled: >>> >>>"Death by Shared Variable" >>> >>>very memorable title, IMO. >> >>I'm too lazy to read the whole article now. I too remember when it >>happened, and I learned from it. The problem boiled down to losing sync >>between the displayed dose and the internally programmed dose because >>the buggy keypad editing routine changed the first without changing the >>second. I learned from that > > > i also thought that one of the lessons is that for every editable variable, > there should be only one method that "owns" it and may write to it. other > dudes can read it (and in fact should read it every time it's updated) but > only one dude may write it. > > i thought there was something about that important variable getting written > over by method #2 and method #1 (just previously having written to that > variable itself) was oblivious of the change and something bad happened. i > dunno.Maybe I'm too belt-and-suspenders, but only the keyboard may write to the displayed number with my way, and when it comes time for use, nobody can write and only the controller routine can read and transcribe it. Jerry -- Engineering is the art of making what you want from things you can get. �����������������������������������������������������������������������
