“It’s not a baseline, if it hasn’t been reproduced by others!”

ECIR also strongly encourages the submission of papers that repeat, reproduce, generalize, and analyze prior work which leads to novel insights, lessons learned, and/or validation and verification. In particular, we solicit replicability (different team, different experimental setup) and reproducibility (different team, same experimental setup) papers. Submissions from the same authors – i.e., repeatability (same team, same experimental setup) papers – of the reproduced experiments will generally not be considered.

Reproducibility is key for establishing research to be reliable, referenceable, and extensible for the future. Emphasize your motivation for selecting the paper/papers, the process of how results have been attempted to be reproduced (successful or not), the communication that was necessary to gather all information, the potential difficulties encountered, and the result of the process. A successful reproduction of the work is not a requirement, but it is crucial to provide a precise and rigid evaluation of the process to allow lessons to be learned for the future.

Submission Guidelines

• Reproducibility papers are up to 12 pages in length plus additional pages for references. Appendices count toward the pagelimit.
• Reproducibility papers will be refereed through double-blind peer review, with an initial first stage review followed with a second stage of discussion led by a meta-reviewer.

Authors should consult Springer’s authors’ guidelines and use their proceedings templates, either for LaTeX or for Word (to be found at https://www.springer.com/gp/computer-science/lncs/conference-proceedings-guidelines), for the preparation of their papers.  Springer encourages authors to include their ORCIDs in their papers (https://www.springer.com/gp/authors-editors/orcid). 

All submissions must be written in English. All papers should be submitted electronically through the EasyChair submission system (https://easychair.org/conferences/?conf=ecir23). 

In addition, the corresponding author of each accepted paper, acting on behalf of all of the authors of that paper, must complete and sign a Consent-to-Publish form. The corresponding author signing the copyright form should match the corresponding author marked on the paper. Once the paper has been submitted, changes relating to its authorship cannot be made.

Accepted papers will be published in the conference proceedings in the Springer Lecture Notes in Computer Science series. The proceedings will be distributed to all delegates at the conference. Accepted papers will have to be presented at the conference – and at least one author will be required to register. ECIR intends to publish the proceedings as open access if sufficient funding is acquired.

Dual submission policy

Papers submitted to ECIR 2022 should be substantially different from papers that have been previously published, or accepted for publication, or that are under review at other venues. Exceptions to this rule are:

• Submission is permitted for papers presented or to be presented at conferences or workshops without proceedings.
• Submission is permitted for papers that have previously been made available as a technical report (e.g., in institutional archives or preprint archives like arXiv). However, we discourage this since it places anonymity at risk; in particular, please do not publish your paper at arXiv and submit to ECIR at the same time, some days before, or during the reviewing period of ECIR.
• If your paper already is available as a technical report:
  • You might not want to use the exact same title and abstract for your ECIR submission (in case of acceptance at ECIR, the title of your submission still might be changed “back”).
  • Please do not cite your technical report and make some effort to avoid any issues that may harm the double-blindness of your submission. Reviewers will receive guidance that ask them to refrain from trying to break blindness if at all possible too, but be aware that the availability of an available technical report for an ECIR submission can cause issues.

Ethics and professional conduct

ECIR 2023 expects authors (as well as the PC, and the organizing committee) to adhere to accepted standards on ethics and professionalism, such as:

• The ACM's Code of Ethics and Professional Conduct,
• The ACM's Conflict of Interest Policy.
• The ACM's Policy on Plagiarism, Misrepresentation, and Falsification.
• The ACM's Policy Against Harassment.
• or similar other standards of professional conduct.

Review Criteria

All reproducibility-track papers will be evaluated along with the following criteria (when applicable):

Novelty

• Were there key practical information (algorithms, parameter settings, software libraries, data collections) not reported in the original paper?
• Was the original work not supported from the theoretical point of view?
• Were the original experiments not clear about important points or lacking confirmation for some of the original claims?
• Are there new baselines and experiments presented in the reproduced paper?
• Is the reproduced paper proposing new evaluation criteria (new measures, statistical tests, …)?

Impact

• How important is the reproduction of the experiments to the community?
• How obvious are the conclusions achieved?
• Do the reproduced prior works, if validated, advance a central topic to information retrieval (a topic with broad applicability or focused on a hot research area)?
• What is the impact of the original paper? Is it central or marginal to the community?

Reliability

• Is the evaluation methodology in line with the research challenges addressed by the reproduced experiment?
• Are the selected baselines representative of the several algorithm types and techniques available?
• Is the parameter/hyperparameter setting properly described?
• Are algorithms and baselines adequately tuned?

Availability

• Are the code and datasets used to reproduce the experiments available to the reviewers at the time of review?
• Is the shared material released in a permanent repository for easy access by researchers?
• Are the reproduced experiments well documented, with all the details required for other researchers to reproduce the experiments?
• Are there discrepancies between what is described in the paper and what is available in the shared material?
• Is the shared material complete with everything you need to replicate the experiments exactly?

Please refer to the ACM “Artifact Review and Badging” guidelines for consistent use of the terminology, which is heterogeneous across disciplines.

Reproducibility paper track dates

• Reproducibility paper submission: October 21, 2022, 11:59pm (AoE)
• Reproducibility paper notification: December 12, 2022
• Main conference: April 3-5, 2023

Reproducibility paper track chairs

• Leif Azzopardi (University of Strathclyde, UK)
• Contact: ecir23-reproducibility at easychair.org

“... and no, I didn’t try your method on Arxiv that you put up last week!”