In January, Erik de Castro Lopo started a long thread here on comp.dsp by calling for signal processing papers to include sufficient source code for the reproduction of results. ICASSP is one of the main conferences associated with the IEEE Signal Processing Society. It happened two weeks ago, and there was a special session on this topic of reproducibility. I've included the list of papers below, along with URLs for those papers which I've found on the public web. I was particularly interested in the paper by Prof. Jelena Kovacevic, a past editor-in-chief of IEEE Transactions on Image Processing. She makes concrete proposals about changing reviewing standards in the Signal Processing Society to place more emphasis on reproducibility. Now is a good time for those of you who'd like to see this to make your voices heard, because the ICASSP special session has put a spotlight on this issue. Personally, I would be glad to sign an open letter or petition in support of placing more emphasis on reproducibility in the review process, and in support of using her proposals as the starting point for discussion of how to do this. SS-L5: Reproducible Signal Processing Research SS-L5.1: EXPERIENCES WITH REPRODUCIBLE RESEARCH IN VARIOUS FACETS OF SIGNAL PROCESSING RESEARCH Patrick Vandewalle, Guillermo Barrenetxea, Ivana Jovanovic, Andrea Ridolfi, Martin Vetterli, Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland http://infoscience.epfl.ch/getfile.py?recid=97195 SS-L5.2: REPRODUCIBLE COMPUTATIONAL EXPERIMENTS USING SCONS Sergey Fomel, University of Texas at Austin, United States; Gilles Hennenfent, University of British Columbia, Canada http://slim.eos.ubc.ca/Publications/Public/Conferences/ICASSP/2007/fomel07icassp.pdf SS-L5.3: PUTTING REPRODUCIBLE SIGNAL PROCESSING INTO PRACTICE: A CASE STUDY IN WATERMARKING Mauro Barni, University of Siena, Italy; Fernando Perez-Gonzalez, Pedro Comesana, University of Vigo, Spain; Guido Bartoli, University of Siena, Italy SS-L5.4: REPRODUCIBLE RESEARCH: CASE STUDY ON SAMPLING SIGNALS WITH FINITE RATE OF INNOVATION Pina Marziliano, Nanyang Technological University, Singapore SS-L5.5: STATE OF THE ART AND EVOLUTION IN PUBLIC DATA SETS AND COMPETITIONS FOR SYSTEM IDENTIFICATION, TIME SERIES PREDICTION AND PATTERN RECOGNITION Joos Vandewalle, Johan Suykens, Bart De Moor, Katholieke Universiteit Leuven, Belgium; Amaury Lendasse, Helsinki University of Technology, Finland ftp://ftp.esat.kuleuven.ac.be/pub/SISTA/ida/reports/06-167.pdf SS-L5.6: HOW TO ENCOURAGE AND PUBLISH REPRODUCIBLE RESEARCH Jelena Kovacevic, Carnegie Mellon University, United States http://andrew.cmu.edu/user/jelenak/Repository/07_04_Icassp_Kovacevic.pdf
Papers are better with code: ICASSP session
Started by ●April 30, 2007
Reply by ●April 30, 20072007-04-30
On Apr 30, 2:29 pm, David Gelbart <for-spam-o...@mailinator.com> wrote:> In January, Erik de Castro Lopo started a long thread here on > comp.dsp by calling for signal processing papers to include sufficient > source code for the reproduction of results. > > ICASSP is one of the main conferences associated with the IEEE Signal > Processing Society. It happened two weeks ago, and there was a > special session on this topic of reproducibility. I've included the > list of papers below, along with URLs for those papers which I've > found on the public web. > > I was particularly interested in the paper by Prof. Jelena Kovacevic, > a past editor-in-chief of IEEE Transactions on Image Processing. > She makes concrete proposals about changing reviewing standards in the > Signal Processing Society to place more emphasis on reproducibility. > Now is a good time for those of you who'd like to see this to make > your voices heard, because the ICASSP special session has put a > spotlight on this issue. Personally, I would be glad to sign an open > letter or petition in support of placing more emphasis on > reproducibility in the review process, and in support of using her > proposals as the starting point for discussion of how to do this. > > SS-L5: Reproducible Signal Processing Research > > SS-L5.1: EXPERIENCES WITH REPRODUCIBLE RESEARCH IN VARIOUS FACETS OF > SIGNAL PROCESSING RESEARCH > Patrick Vandewalle, Guillermo Barrenetxea, Ivana Jovanovic, Andrea > Ridolfi, Martin Vetterli, Ecole Polytechnique Federale de Lausanne > (EPFL), Switzerlandhttp://infoscience.epfl.ch/getfile.py?recid=97195 > > SS-L5.2: REPRODUCIBLE COMPUTATIONAL EXPERIMENTS USING SCONS > Sergey Fomel, University of Texas at Austin, United States; Gilles > Hennenfent, University of British Columbia, Canadahttp://slim.eos.ubc.ca/Publications/Public/Conferences/ICASSP/2007/fo... > > SS-L5.3: PUTTING REPRODUCIBLE SIGNAL PROCESSING INTO PRACTICE: A CASE > STUDY IN WATERMARKING > Mauro Barni, University of Siena, Italy; Fernando Perez-Gonzalez, > Pedro Comesana, University of Vigo, Spain; Guido Bartoli, University > of Siena, Italy > > SS-L5.4: REPRODUCIBLE RESEARCH: CASE STUDY ON SAMPLING SIGNALS WITH > FINITE RATE OF INNOVATION > Pina Marziliano, Nanyang Technological University, Singapore > > SS-L5.5: STATE OF THE ART AND EVOLUTION IN PUBLIC DATA SETS AND > COMPETITIONS FOR SYSTEM IDENTIFICATION, TIME SERIES PREDICTION AND > PATTERN RECOGNITION > Joos Vandewalle, Johan Suykens, Bart De Moor, Katholieke Universiteit > Leuven, Belgium; Amaury Lendasse, Helsinki University of Technology, > Finlandftp://ftp.esat.kuleuven.ac.be/pub/SISTA/ida/reports/06-167.pdf > > SS-L5.6: HOW TO ENCOURAGE AND PUBLISH REPRODUCIBLE RESEARCH > Jelena Kovacevic, Carnegie Mellon University, United Stateshttp://andrew.cmu.edu/user/jelenak/Repository/07_04_Icassp_Kovacevic.pdfDavid, I used to attend ICASSP. There were some real impressive papers and demonstration, so I am not slamming ICASSP. However, there were too many audio presentations where people were "putting their best foot forward", presenting their best results as typical, and there was also some methodology in some of the audio presentations that made the performance appear much better than it was. One method I saw was playing the same audio over and over again while 'reducing the quality' so that by training the listeners the quality of the final version sounded much better than if it had been played first; in one case the final result was not actually intelligible, but the audience "understood" it (I covered my ears for all but the final playing). Sometimes the results were so specific to the audio they used that the method was of no interest, but that was not ever stated. I asked one of the authors of such a paper about the paper during a job interview years ago, he actually blushed bright red, admitted that it didn't work well on audio, said it worked on what they were interested in, and ... I didn't get an offer. My point is, if you require supplying the code, you also need to require people to supply a copy of any data that is processed by the code for the paper. This requirement will probably discourage more than a few authors. I don't think that is a bad thing. Dirk
Reply by ●April 30, 20072007-04-30
> My point is, if you require supplying the code, you also need to > require people to supply a copy of any data that is processed by the > code for the paper.Dirk, I agree that data sharing is also part of reproducible research. The papers in the special session share this view, and I should have made that more clear in my post. (My description of Prof. Kovacevic's proposals as being only about 'changing reviewing standards' was also incomplete. Her full paper is publicly available for those who want the details.) Because of the intellectual property issues that many authors face, I do not think that the Signal Processing Society should require that every paper share all code and data. I do think that it's practical and desirable for there to be much more sharing than there is today. Regards, David
Reply by ●May 1, 20072007-05-01
Dude, you are naive You proposal is dead in the water - those conferences are just public hangouts, paid up vacations for most of the attendees, with some goodies given out to the members of the "inner" circle I suggest a little experiment: come up with some truly original idea (like FFT or cepstrum,,, just joking) write a paper, develop a working software demo (e.g. in Matlab), submit it to ICASSP with the faked author name and unknown affiliation, then see what happens. Regardsless of the merits of your paper your chances of acceptance are less than 25%... On Apr 30, 2:29 pm, David Gelbart <for-spam-o...@mailinator.com> wrote:> In January, Erik de Castro Lopo started a long thread here on > comp.dsp by calling for signal processing papers to include sufficient > source code for the reproduction of results. > > ICASSP is one of the main conferences associated with the IEEE Signal > Processing Society. It happened two weeks ago, and there was a > special session on this topic of reproducibility. I've included the > list of papers below, along with URLs for those papers which I've > found on the public web. > > I was particularly interested in the paper by Prof. Jelena Kovacevic, > a past editor-in-chief of IEEE Transactions on Image Processing. > She makes concrete proposals about changing reviewing standards in the > Signal Processing Society to place more emphasis on reproducibility. > Now is a good time for those of you who'd like to see this to make > your voices heard, because the ICASSP special session has put a > spotlight on this issue. Personally, I would be glad to sign an open > letter or petition in support of placing more emphasis on > reproducibility in the review process, and in support of using her > proposals as the starting point for discussion of how to do this. > > SS-L5: Reproducible Signal Processing Research > > SS-L5.1: EXPERIENCES WITH REPRODUCIBLE RESEARCH IN VARIOUS FACETS OF > SIGNAL PROCESSING RESEARCH > Patrick Vandewalle, Guillermo Barrenetxea, Ivana Jovanovic, Andrea > Ridolfi, Martin Vetterli, Ecole Polytechnique Federale de Lausanne > (EPFL), Switzerlandhttp://infoscience.epfl.ch/getfile.py?recid=97195 > > SS-L5.2: REPRODUCIBLE COMPUTATIONAL EXPERIMENTS USING SCONS > Sergey Fomel, University of Texas at Austin, United States; Gilles > Hennenfent, University of British Columbia, Canadahttp://slim.eos.ubc.ca/Publications/Public/Conferences/ICASSP/2007/fo... > > SS-L5.3: PUTTING REPRODUCIBLE SIGNAL PROCESSING INTO PRACTICE: A CASE > STUDY IN WATERMARKING > Mauro Barni, University of Siena, Italy; Fernando Perez-Gonzalez, > Pedro Comesana, University of Vigo, Spain; Guido Bartoli, University > of Siena, Italy > > SS-L5.4: REPRODUCIBLE RESEARCH: CASE STUDY ON SAMPLING SIGNALS WITH > FINITE RATE OF INNOVATION > Pina Marziliano, Nanyang Technological University, Singapore > > SS-L5.5: STATE OF THE ART AND EVOLUTION IN PUBLIC DATA SETS AND > COMPETITIONS FOR SYSTEM IDENTIFICATION, TIME SERIES PREDICTION AND > PATTERN RECOGNITION > Joos Vandewalle, Johan Suykens, Bart De Moor, Katholieke Universiteit > Leuven, Belgium; Amaury Lendasse, Helsinki University of Technology, > Finlandftp://ftp.esat.kuleuven.ac.be/pub/SISTA/ida/reports/06-167.pdf > > SS-L5.6: HOW TO ENCOURAGE AND PUBLISH REPRODUCIBLE RESEARCH > Jelena Kovacevic, Carnegie Mellon University, United Stateshttp://andrew.cmu.edu/user/jelenak/Repository/07_04_Icassp_Kovacevic.pdf
Reply by ●May 1, 20072007-05-01
fizteh89 <dt@soundmathtech.com> writes:> Dude, you are naive > > You proposal is dead in the water - those conferences are just public > hangouts, paid up vacations for most of the attendees, with some > goodies given out to the members of the "inner" circle > > I suggest a little experiment: come up with some truly original idea > (like FFT or cepstrum,,, just joking) > write a paper, develop a working software demo (e.g. in Matlab), > submit it to ICASSP with the faked author name and unknown > affiliation, then see what happens. > Regardsless of the merits of your paper your chances of acceptance are > less than 25%...If what you say is true (and I suspect it may be), it is a sad state of affairs. I wonder if a conference would ever be organized in which innovation (and/or just plain love of signal processing) is espoused above pigskin. Maybe an independent conference? Maybe a *comp.dsp* conference??? ... -- % Randy Yates % "Remember the good old 1980's, when %% Fuquay-Varina, NC % things were so uncomplicated?" %%% 919-577-9882 % 'Ticket To The Moon' %%%% <yates@ieee.org> % *Time*, Electric Light Orchestra http://home.earthlink.net/~yatescr
Reply by ●May 1, 20072007-05-01
On 1 May 2007 08:02:19 -0700, fizteh89 <dt@soundmathtech.com> wrote:> I suggest a little experiment: come up with some truly original idea >(like FFT or cepstrum,,, just joking) >write a paper, develop a working software demo (e.g. in Matlab), >submit it to ICASSP with the faked author name and unknown >affiliation, then see what happens. >Regardsless of the merits of your paper your chances of acceptance are >less than 25%...I'm not quite as cynical as your post suggests you may be, but I think there's some truth to this point, and I think it applies to many transactions journals as well as conferences. Eric Jacobsen Minister of Algorithms Abineau Communications http://www.ericjacobsen.org
Reply by ●May 1, 20072007-05-01
> You proposal is dead in the waterWhile organizations as large as the IEEE Signal Processing Society have some amount of inertia, I think it is premature to say that change is impossible. The near-term goal Prof. Kovacevic lays out is not to have every single paper achieve some gold standard of reproducibility, but to increase the percentage of papers that do. Given her experience as an editor, a professor, a staff member at Bell Labs, and co founder of a startup, I think she is in a good position to craft proposals that have a chance of winning support.
Reply by ●May 1, 20072007-05-01
On May 1, 1:58 pm, David Gelbart <for-spam-o...@mailinator.com> wrote:> Given her experience as an editor, a professor, a staff member at Bell > Labs, and co founder of a startup,and a pretty nice woman too... I am already running to cast my vote for Jelena Kovacevic's proposal Oops, forgot that I am not an IEEE member...
Reply by ●May 1, 20072007-05-01
On May 1, 11:21 am, fizteh89 <d...@soundmathtech.com> wrote:> Oops, forgot that I am not an IEEE member...I think the opinion of anyone who reads IEEE papers is worth counting, whether or not they are a member of the IEEE. Regarding reviewing: I am not sure whether or not there is a problem. The sort of experiment you suggest would be valuable but it's difficult to conduct. There are major conferences in computer science which have "double-blind' reviewing in which the identity of the authors is hidden from the reviewers. I have the impression that people are more likely to want double-blind reviewing when the conference acceptance rate is low, because the fairness of the process becomes a bigger concern. ICASSP acceptance rates seem to be dropping over time; maybe this will lead to double-blind reviewing at ICASSP.
Reply by ●May 1, 20072007-05-01
"fizteh89" <dt@soundmathtech.com> wrote in message news:1178031739.439709.320720@y80g2000hsf.googlegroups.com... ......> You proposal is dead in the water - those conferences are just public > hangouts, paid up vacations for most of the attendees, .....Very true and pretty much as one *should* expect. After all, the *first* definition of the word "Symposium" in the Merriam-Webster on-line dictionary is 1 a : a convivial party (as after a banquet in ancient Greece) with music and conversation b : a social gathering at which there is free interchange of ideas 2 a : a formal meeting at which several specialists deliver short addresses on a topic or on related topics
