How do I ensure the reproducibility of data produced by my software?
Generally, there are two approaches to this, but they ultimately contradict each other. On one hand, there is the preservation of the program code as unchanged as possible. With this approach, there is practically no risk that the data produced by the software will change, but long-term executability is not guaranteed. The other approach is the continuous maintenance of the software, which at least requires adaptations to changing operating systems and runtime environments; ideally, this also involves further development.
Preservation of the Program Code
This approach may also be necessary for documentation purposes (e.g., if the software is part of a publication).
Archiving
- Containers (Docker incl. Docker Hub, podman): Programming should be done with the use of containers in mind from the beginning; adapting already completed software can be difficult
- VM: works largely without issues with existing software, possibility to preserve (old) operating systems along with the software
- Long-term reliability also depends on the existence of companies (e.g., Docker)
- What if packages that need to be installed into the base image are no longer available? -> This should at least be guaranteed with Docker.
- More information in Konrad Hinsen’s blog:
- Two (relatively old - why is there nothing newer?) articles on the topic of software preservation including operating systems:
How do I ensure that my source code remains readable (and executable) for 10 years?
- TECHNOLOGY FEATURE: Challenge to scientists: does your ten-year-old code still run?
- Application Preservation of the Humanities Data Centre (Göttingen)
- Examples include a software project of the MPI-MMG (the PIDs are unfortunately no longer valid)
- Ensure that data formats will remain readable in the future (Are Research Datasets FAIR in the Long Run?)
- See also in the Pad FAQs on PIDs and citation: How does software become persistent?
Continuous Maintenance
For this, reviewing, consistent testing (also automated), and documentation (also in the form of version control) are absolutely necessary.
What standards are there for reviewing scientific software (field-specific or generic)?
- Pair-programming
- AI (Co-piloting)?
- JOSS Reviewer Guidelines
Examples
- MPI Psychiatry Code Club = local solution!
- Code review checklist from MPI Psychiatry Code Club
- INRIA software self-assessment
- MaRDI TA1 technical review sheet (math, better link TODO)
- ADHO DHTech Code Review
Consistent Testing
How can regression tests be automated?
Three evaluations of corresponding software (https://www.rainforestqa.com/blog/regression-testing-tools, https://www.softwaretestinghelp.com/regression-testing-tools/, https://theqalead.com/tools/best-regression-testing-tools/). The intersection results in (in alphabetical order):
- IBM Rational Functional Tester
- Katalon Studio
- Selenium
- Watir
- Not included in the evaluations: KD-Executor
None of these are currently available through the MPDL license list (SoLi), possibly submit suggestions
In any case, indispensable: Documentation (software and hardware!)
- In the end, only text-based documentation of the algorithm, independent of the programming language, remains (but this would be extremely labor-intensive)
- Compromise: Source code including all relevant libraries also as source code (especially for mathematical calculations)
- Regarding documentation tools for software, see also the FAQ on Documentation.