Call for Reproducibility papers
“It’s not a baseline, if it hasn’t been reproduced by others!”
ECIR also strongly encourages the submission of papers that repeat, reproduce, generalize,
and analyze prior work which leads to novel insights, lessons learned, and/or
validation and verification. In particular, we solicit replicability (different team,
different experimental setup) and reproducibility (different team, same experimental
setup) papers. Submissions from the same authors – i.e., repeatability (same team,
same experimental setup) papers – of the reproduced experiments will generally not be
Reproducibility is key for establishing research to be reliable, referenceable, and
extensible for the future. Emphasize your motivation for selecting the paper/papers, the
process of how results have been attempted to be reproduced (successful or not), the
communication that was necessary to gather all information, the potential difficulties
encountered, and the result of the process. A successful reproduction of the work is not a
requirement, but it is crucial to provide a precise and rigid evaluation of the process to
allow lessons to be learned for the future.
- Reproducibility papers are up to 12 pages in length plus additional
pages for references. Appendices count toward the pagelimit.
- Reproducibility papers will be refereed through double-blind peer
review, with an initial first stage review followed with a second
stage of discussion led by a meta-reviewer.
Authors should consult Springer’s authors’ guidelines and use their proceedings templates,
either for LaTeX or for Word (to be found at https://www.springer.com/gp/computer-science/lncs/conference-proceedings-guidelines),
for the preparation of their papers. Springer encourages authors to include their
ORCIDs in their papers (https://www.springer.com/gp/authors-editors/orcid).
All submissions must be written in English. All papers should be submitted electronically
through the EasyChair submission system (https://easychair.org/conferences/?conf=ecir23).
In addition, the corresponding author of each accepted paper, acting on behalf of all of the
authors of that paper, must complete and sign a Consent-to-Publish form. The corresponding
author signing the copyright form should match the corresponding author marked on the paper.
Once the paper has been submitted, changes relating to its authorship cannot be made.
Accepted papers will be published in the conference proceedings in the Springer Lecture Notes
in Computer Science series. The proceedings will be distributed to all delegates at the
conference. Accepted papers will have to be presented at the conference – and at least one
author will be required to register. ECIR intends to publish the proceedings as open access
if sufficient funding is acquired.
Dual submission policy
Papers submitted to ECIR 2022 should be substantially different from papers that have
been previously published, or accepted for publication, or that are under review at
other venues. Exceptions to this rule are:
- Submission is permitted for papers presented or to be presented at conferences or
workshops without proceedings.
- Submission is permitted for papers that have previously been made available as a
technical report (e.g., in institutional archives or preprint archives like arXiv).
However, we discourage this since it places anonymity at risk; in particular, please
do not publish your paper at arXiv and submit to ECIR at the same time, some days
before, or during the reviewing period of ECIR.
- If your paper already is available as a technical report:
- You might not want to use the exact same title and abstract for your ECIR
submission (in case of acceptance at ECIR, the title of your submission
still might be changed “back”).
- Please do not cite your technical report and make some effort to avoid any
issues that may harm the double-blindness of your submission. Reviewers will
receive guidance that ask them to refrain from trying to break blindness if
at all possible too, but be aware that the availability of an available
technical report for an ECIR submission can cause issues.
Ethics and professional conduct
ECIR 2023 expects authors (as well as the PC, and the organizing committee) to adhere
to accepted standards on ethics and professionalism, such as:
All reproducibility-track papers will be evaluated along with the following criteria
- Were there key practical information (algorithms, parameter settings, software
libraries, data collections) not reported in the original paper?
- Was the original work not supported from the theoretical point of view?
- Were the original experiments not clear about important points or lacking
confirmation for some of the original claims?
- Are there new baselines and experiments presented in the reproduced paper?
- Is the reproduced paper proposing new evaluation criteria (new measures,
statistical tests, …)?
- How important is the reproduction of the experiments to the community?
- How obvious are the conclusions achieved?
- Do the reproduced prior works, if validated, advance a central topic to
information retrieval (a topic with broad applicability or focused on a hot
- What is the impact of the original paper? Is it central or marginal to the
- Is the evaluation methodology in line with the research challenges addressed by
the reproduced experiment?
- Are the selected baselines representative of the several algorithm types and
- Is the parameter/hyperparameter setting properly described?
- Are algorithms and baselines adequately tuned?
- Are the code and datasets used to reproduce the experiments available to the
reviewers at the time of review?
- Is the shared material released in a permanent repository for easy access by
- Are the reproduced experiments well documented, with all the details required
for other researchers to reproduce the experiments?
- Are there discrepancies between what is described in the paper and what is
available in the shared material?
- Is the shared material complete with everything you need to replicate the
Please refer to the ACM
“Artifact Review and Badging” guidelines for consistent use of the
terminology, which is heterogeneous across disciplines.
Reproducibility paper track dates
- Reproducibility paper submission: October 21, 2022,
- Reproducibility paper notification: December 12, 2022
- Main conference: April 3-5, 2023
Reproducibility paper track chairs
- Leif Azzopardi (University of Strathclyde, UK)
- Contact: ecir23-reproducibility at easychair.org
“... and no, I didn’t try your method on Arxiv that you put up last week!”