Registered Reports

Jump to:

The principled idea of Registered Reports can be summarised as follows: Perform peer-review before results are known, in order to align scientific values and practices. BIOLINGUISTICS has been one of the first linguistics journals to adopt Registered Reports as a new article format, alongside regular articles, briefs, reviews, and forum contributions.

Graphic by Center for Open Science.

This page offers a guide to submitting a Registered Report for prospective authors, as well as guidance for reviewers. The general way in which Registered Reports have been implemented at BIOLINGUISTICS has been adapted from the recommendations of the Center for Open Science.

Guidelines for authors

Registered Reports are a form of empirical article in which the methods and proposed analyses are pre-registered and reviewed prior to research being conducted. This format is designed to minimise bias in deductive science, while also allowing complete flexibility to conduct exploratory (unregistered) analyses and report serendipitous findings.

The cornerstone of the Registered Reports format is that a significant part of the manuscript will be assessed prior to data collection, with the highest quality submissions accepted in advance. Initial submissions will include a description of the key research question and background literature, hypotheses, experimental procedures, analysis pipeline, a statistical power analysis (or Bayesian equivalent), and pilot data (where applicable).

Initial submissions will be triaged by the Editor in Chief for suitability. Those that pass triage will then be assigned to an Associate Editor and sent for in-depth peer review (Stage 1). This process follows the general double-blind peer-review process implemented at BIOLINGUISTICS for all original articles. Following review, the article will then be either rejected or accepted in principle for publication. Following in principle acceptance (IPA), the authors will then proceed to conduct the study, adhering exactly to the peer-reviewed procedures. When the study is complete the authors will submit their finalised manuscript for re-review (Stage 2) and will upload their raw data, digital study materials, and laboratory log to a publicly accessible file-sharing service. Pending quality checks and a sensible interpretation of the findings, the manuscript will be published regardless of the results.

Schematic of the review process for Registered Reports.

Stage 1: Initial Manuscript Submission and Review

Stage 1 submissions should include the manuscript (details below) and a brief Comment to the Editor. Please note that the Associate Editors will not agree to send manuscripts for in-depth review until a complete Stage 1 submission has been considered. The Stage 1 Comments to the Editor should include:
  • A brief scientific case for consideration. Replication studies are welcome in addition to novel studies.
  • A statement confirming that all necessary support (e.g. funding, facilities) and approvals (e.g. ethics) are in place for the proposed research.
  • An anticipated timeline for completing the study if the initial submission is accepted.
  • A statement confirming that the authors agree to share their raw data, any digital study materials, and analysis code as appropriate.
  • A statement confirming that, following Stage 1 in principle acceptance, the authors agree to register their approved protocol on the Open Science Framework or other recognised repository, either publicly or under private embargo until submission of the Stage 2 manuscript.
  • A statement confirming that if the authors later withdraw their paper, they agree to BIOLINGUISTICS publishing a short summary of the pre-registered study under a sub-section Withdrawn Registrations in the section Registered Reports.
Manuscripts for initial Stage 1 submissions should include the following sections:
  • Introduction: A review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. Please note that following IPA, the Introduction section cannot be altered apart from correction of factual errors, typographic errors and altering of tense from future to past (see below).
  • Methods:
    • Full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g. outlier extraction). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.
    • A description of experimental procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly in the subsequent experiments or any Stage 2 manuscript can be rejected.
    • Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Only pre-planned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned exploratory analyses will be admissible in a separate section of the Results (see below).
    • Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.
    • Any description of prospective methods or analysis plans should be written in future tense.
  • Pilot Data (optional): Optional. Can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Any pilot experiments will be published with the final version of the manuscript as an Appendix.

Stage 1 submissions that are judged by the Editor-in-Chief and the assigned Associate Editor to be of sufficient quality and within journal scope will be sent for in-depth peer review to two external reviewers and, if necessary, to a member of the BIOLINGUISTICS Editorial Board or Advisory Board for additional evaluation.

Following Stage 1 peer review, manuscripts will be rejected outright, offered the opportunity to revise, or accepted. Proposals that exceed the highest standards of importance and scientific rigour will be issued an in principle acceptance (IPA), indicating that the article will be published pending completion of the approved methods and analytic procedures, passing of all pre-specified quality checks, and a defensible interpretation of the results. Stage 1 protocols are not published by the journal following IPA. Instead they are registered by the authors in a recognised repository (either publicly or under embargo until Stage 2) and then integrated into a single completed article following approval of the final Stage 2 manuscript.

Authors are reminded that any deviation from the stated experimental procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2. In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the Associate Editor immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted per editorial discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the experimental procedures more substantially following IPA but still wish to publish their article as a Registered Report then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript (see below).

Stage 2: Full Manuscript Review

Once the study is complete, authors prepare and resubmit their manuscript for full review, with the following additions:

  • Comments to the Editor: The Stage 2 Comments to the Editor must confirm that the manuscript (1) includes a link to the public archive containing anonymized study data, digital materials/code and the laboratory log. (2) That the manuscript contains a link to the approved Stage 1 protocol on the Open Science Framework or other recognised repository.
  • Submission of anonymised raw data, digital study materials, and laboratory log:
    • Anonymised raw data and digital study materials must be made freely available in a public repository/archive with a link provided within the Stage 2 manuscript. Authors are free to use any repository that renders data and materials freely and publicly accessible and provides a digital object identifier (DOI) to ensure that the data remain persistent, unique and citable. For a comprehensive list of available data repositories, see http://www.re3data.org/
    • Data files should be appropriately time stamped to show that data was collected after IPA and not before. Other than pre-registered and approved pilot data, no data acquired prior to the date of IPA is admissible in the Stage 2 submission. Raw data must be accompanied by guidance notes, where required, to assist other scientists in replicating the analysis pipeline. Authors are required to upload any relevant analysis scripts and other digital experimental materials that would assist in replication.
    • Any supplementary figures, tables, or other text (such as supplementary methods) can either be included as standard Appendix that accompanies the paper, or they can be archived together with the data. Please note that the raw data itself should be archived (see above) rather than submitted to the journal.
    • A basic laboratory log must also be provided outlining the range of dates during which data collection took place. This log should be uploaded to the same public archive as the data and materials.
    • The Stage 2 manuscript must also contain a link to the registered protocol (deposited following IPA) on the Open Science Framework or other recognised repository.
  • Background, Rationale and Methods: Apart from minor stylistic revisions, the Introduction cannot be altered from the approved Stage 1 submission, and the stated hypotheses cannot be amended or appended. At Stage 2, any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense. Any textual changes to the Introduction or Methods (e.g. correction of typographic errors) must be clearly marked in the Stage 2 submission. Any relevant literature that appeared following the date of IPA should be covered in the Discussion.
  • Results and Discussion: The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but omitted with justification from the Results. It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc analyses.

The resubmission will most likely be considered by the same reviewers as in Stage 1, but could also be assessed by new reviewers.

Reviewers are informed that editorial decisions will not be based on the perceived importance, novelty or conclusiveness of the results. Thus, while reviewers are free to enter such comments on the record, they will not influence editorial decisions. Reviewers at Stage 2 may suggest that authors report additional post hoc tests on their data; however, authors are not obliged to do so unless such tests are necessary to satisfy one or more of the Stage 2 review criteria.

Guidelines for reviewers

Registered Reports are a form of empirical article in which the methods and proposed analyses are pre-registered and reviewed prior to research being conducted. This format of article seeks to neutralise a variety of inappropriate research practices, including inadequate statistical power, selective reporting of results, and publication bias.

The review process for Registered Reports is divided into two stages. In Stage 1, reviewers assess study proposals before data is collected. In Stage 2, reviewers consider the full study, including results and interpretation.

Stage 1 manuscripts will include only an Introduction, Methods (including proposed analyses), and Pilot Data (where applicable). In considering papers at Stage 1, reviewers will be asked to assess:

  1. The importance of the research question(s).
  2. The logic, rationale, and plausibility of the proposed hypotheses.
  3. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
  4. Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.
  5. Whether the authors have pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks.

Following Stage 1 peer review, manuscripts will be accepted, offered the opportunity to revise, or rejected outright. Manuscripts that pass peer review will be issued an in principle acceptance (IPA), indicating that the article will be published pending successful completion of the study according to the exact methods and analytic procedures outlined, as well as a defensible and evidence-bound interpretation of the results.

Following completion of the study, authors will complete the manuscript, including Results and Discussion sections. These Stage 2 manuscripts will more closely resemble a regular article format. The manuscript will then be returned to the reviewers, who will be asked to appraise:

  1. Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls)
  2. Whether the Introduction, rationale and stated hypotheses are the same as the approved Stage 1 submission (required)
  3. Whether the authors adhered precisely to the registered experimental procedures
  4. Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative
  5. Whether the authors’ conclusions are justified given the data

Please note that editorial decisions will not be based on the perceived importance, novelty, or clarity of the results.