IPOL: Image Processing On Line Gabriele Facciolo Nicolas Limare 10.6084/m9.figshare.679852.v1 https://figshare.com/articles/presentation/IPOL_Image_Processing_On_Line/679852 <p>Presented at the Oxford Future of Science conference "Rigour and Openness in 21st Century Science", 2013-04-11, Oxford, UK</p> <p> </p> <p>It’s clear that software is important in modern science. It’s used to SOLVE real scientific problems in a variety of fields ranging from physics, to medicine and economics. However research software is not like general purpose equipment (like a telescopes or compilers). Research software is usually made by scientist for scientists, tailored to a concrete experimental process. So this code is not meant to be released nor published and even less reviewed. Nevertheless the research code is very important, since it embodies the process itself. So the research software is key for attaining a truly reproducible research.</p> <p>In this talk I’m going to share our experience with IPOL: an on-line journal for image processing, a journal that publishes algorithm descriptions along with the code that implement them. IPOL follows the open access and reproducible research models. Since it is a journal, the code is also peer-reviewed to verify that it corresponds to the algorithm description.</p> <p>The distinguishing characteristic of IPOL articles is that each article also includes an on-line demo that permits to easily test the code with new data and parameters from the web, and a public archive that stores all original test data used in the demo. Both the demo and the archive provide great insight into the algorithm inner workings and limitations, facilitating the experimentation and leading in the end to a stricter verification of the algorithm.</p> <p> </p> <p>Derived from a presentation of N. Limare: http://dx.doi.org/10.6084/m9.figshare.155649</p> <p> </p> <p> </p> <p><br>---------------------------------------------------------------------------<br>Speech (RAW):<br>---------------------------------------------------------------------------<br>1. In this talk I’m going to share our experience with IPOL: an on-line journal for image processing, which follows the open access and reproducible research models.</p> <p>2. Software. It’s clear that software is important in modern science. It’s used to SOLVE real scientific problems in a variety of fields ranging from physics, to medicine and economics.</p> <p>3. However research software is not like general purpose equipment (like a telescope).<br>Research software is usually made by scientist for scientists, tailored to a concrete experimental process. So this code is not meant to be released nor published and even less reviewed.</p> <p>4. (REPRODUCIBLE RESEARCH RECOGNIZES THE IMPORTANCE OF SOFTWARE)<br>But the research code is very important, since it embodies the process itself so the results are linked to it. So the research software is key for attaining a truly reproducible research.<br>Note that theoretical sciences have proofs, experimental sciences procedures but computational sciences have:<br>(in many occasions) incomplete descriptions, missing steps and/or data and missing pre/post processes.</p> <p>5. These observations led to the emergence of many reproducible research initiatives. Which are aimed at making available to the community the missing data and/or code.</p> <p>6. And the science code manifesto proposes five principles concerning: copyright, citation, credit and curation of code, which should be followed when software is used in (open) science.</p> <p>7. And there’s the open access, which is important by itself. However I will not expand on it. Let me just recommend a 9-minute video from Jorge Cham (the author of phdcomics) in which is explained what open access is about. And it is very clear about it.</p> <p>8-11. Now, what’s wrong with traditional articles? In the field of image processing (which is the one I know) the articles usually describe some algorithm and then show some results. But rarely provide a way to reproduce these results (the code). As you can image this is suspicious.<br>But what are the reasons to hide the software?<br>I can only imagine 3 reasons.<br>* The first is to keep a competitive “advantage” with respect to other researchers. But in that case the optimal strategy whould be not publishing at all.<br>* The second is that the author believes the code is not proper enough or not ready to be seen.<br>* And the third is to prevent the incorrect use by miss-setting the parameters, and then claiming that it doesn’t work.</p> <p>12-13. So the question is: Can we convince researchers to share their code? The key is to understand why researchers publish in the first place. Ideally research should be as depicted on the left: “sharing for the greater good”. But in practice we know that the depiction of the right is more faithful to reality. The circumstances that got us here aren’t important now.<br>I just want to remark that a way to convince the researchers is here.</p> <p>14-15. (CREATE INCENTIVE)<br>With a less dramatic representation, we can depict the traditional feedback loop in which the researcher publishes its findings and the community “rewards” him with cites.<br>However until recently there were no incentive for the researcher to publish any code.<br>Can we promote a similar feedback process for the research code?<br>The anwert is Yes! (maybe) Following the ideas of the manifesto. First we need to be able to cite the code, and then we need to “convince” the community that code is also a form of publication.</p> <p>16. IPOL implements the first step. It is a research journal for image processing that publishes algorithm descriptions along with the code that implement them.<br>Since it is a journal, the code is also peer-reviewed to verify that it corresponds to the algorithm description. And to verify that follows some basic guidelines for ensuring correctness documentation and portability.</p> <p>17. The goal of IPOL is to provide reference implementations for classical and new image processing algorithms.<br>IPOL is not a prototype, in fact it’s been running since 2011.<br>IPOL is a true journal with DOI ISSN and international editorial committee, we hace a partnership with a SIAM journal for publishing on both journals.</p> <p>Lastly, let me say that IPOL came into existence because no other journal did it.</p> <p>18-20. So, What’s in the box? Each ipol article comprehends three parts:<br>* an algorithm description, and the code. The description is like a standard pdf article and the code has been peer-reviewed to assure that corresponds to the description.<br>* the second part is an on-line demo, this interface permits to easily test the code with new data and parameters from the web.<br>* and last but not least ipol stores a public archive of original test data used in the demo.<br>This archive provides a great insight into the applicability of the algorithm and its limitations, which are usually spotted looking at the experiments made by other users.</p> <p>21-22. Producing the on line demo requires some effort but it’s worth it. As it facilitates the experimentation and in the end leads to stricter verification of the claims about the algorithm. Sometimes it’s hard to compile, setup and run the code, while with the demo that’s immediate.</p> <p>23. As for 2013 IPOL has 30 published articles, and 20 more in preparation. It has a sustained activity and its building momentum.</p> <p>24. On the good side IPOL:<br>* provides reference implementations,<br>* And since it implies a deep analysis of the algorithm it usually leads to improvements,<br>* and the reviewers usually contribute to improve code and algorithms</p> <p>25-26. (AWARENESS OF LIMITATIONS)<br>The challenges. It remains an asymmetry between the effort needed to publish the code and the reward. To address this we need on one hand, to facilitate the preparation of articles (which is technical), and on the other improve the community response (which is social).<br>Also there exists the issue that reproducible is not necessarily reusable. Reproducible code is not aimed at reusing. A possible way to reduce the complexity associated with the preparation of the demo is to team-up with web designers or visualization scientists.</p> <p>27-28. If you want a single take home message from IPOL let it be:<br>"the on-line demo and achieve allow for a faster experimentation and stricter validation of the results.</p> <p>Thank you for your attention.</p> <p><br>---------------------------------------------------------------------------<br>Presentation points:<br>---------------------------------------------------------------------------</p> <p>SOFTWARE IS EVERYWHERE</p> <p>REPRODUCIBLE RESEARCH RECOGNIZES THE IMPORTANCE OF SOFTWARE</p> <p>NOT THE FIRST TIME THAT THIS HAS BEEN NOTICED (OTHER INITIATIVES)</p> <p>WHAT’S WRONG WITH TRADITIONAL RESEARCH AND THE CODE</p> <p>RESEARCH OBJECTIVES : Researchers are not interested in sharing code</p> <p>RESEARCH OBJECTIVES: CREATE INCENTIVE FOR SHARING THE CODE</p> <p>IPOL: PUBLISHES ALGORITHMS (3 pieces): [DESCRIPTION+CODE]+DEMO+ARCHIVE</p> <p>IPOL: GROWING</p> <p>OPEN ACCESS BET</p> <p>CHALLENGES: DEMO PRODUCTION, REUSABILITY, CODE REVIEW C</p> <p>CHALLENGES: CREATE THE HABIT OF PUBLISHING THE CODE<br>---------------------------------------------------------------------------<br>---------------------------------------------------------------------------</p> <p> </p> 2013-04-11 22:10:45 software engineering image processing computer vision reproducible-research open science rigor Software Engineering Science Policy Applied Computer Science