conference post alt text.

Oct 26, 2026

EMNLP Paper Integrity Policy

Challenge #1: Systematic abuse of community resource

  • There were organized efforts to submit fully Al-generated slop to EMNLP by the dozens - a waste of human review effort.
  • The review system in its current form does not address this issue
    • We are seeing scores of 3/3.5/4's given to many of such slop
    • (more on reviews soon)
  • There is no existing mechanism to automatically detect, or systematically penalize such abuse.

Challenge #2: Unverifiable citation

  • Unverifiable citation counts do not always correlate with Al slop
    • It does not identify paper mills
  • Took ~10 people days to manually verify, not including AC effort, which took away valuable time for other organizational duties.
  • We do not believe that citation checks, though with its own merit, is scalable, or effective for catching slop.

Problems with Al reviews

  • Qualitatively, we observe the following issues with Al reviews at EMNLP and our Al review experiment partner from AAAI:
    • The number of weaknesses is much larger than human reviews
    • No notion of salience: which errors are major ones vs. minor ones
    • Lack of "research taste"
    • Does not call out slop papers

Lack of Engagement during Rebuttal

  • Reviewers are not responding to authors during the response period
  • Histogram of reviewer replies across ARR May 2026 by submission
    • The vast majority of submissions received 0 replies (nearly 10,000), with the frequency sharply decreasing to around 4,300 for 1 reply, 2,200 for 2 replies, and dropping close to zero for 3 or more replies.

Other Compounding Challenges

  • Reciprocal review requirements leading to authors "signing up" for their co-authors to review 20+ papers!
  • We need a mechanism to handle frustrating behavior from program committee members
    • sloppy Al reviews or meta reviews
    • ghosting
    • late responses and endless chasing of reviewers / ACS

ARR volunteers take on an enormous service load

  • ARR May cycle received 3353 tickets in the May cycle
  • EMNLP PCs received > 10,000 emails in the last year
  • ARR team are volunteers who take on unimaginable service load
    • While handling ARR tickets, we observed several emails with verbally abusive and intimidating language
  • Running the infrastructure as-is requires full time tech team
  • Running the submission cycle as-is is a full time job

Community overload: Challenges

  • Double cycles across clearly distinct conferences (EACL/ACL, AACL/EMNLP, NAACL/COLING)
    • Makes SAC recruitment harder and spreads their expertise more thinly. We handled this by strictly separating commitments.
    • Tight, rigid cycles that don't work with physical times for conferences
      • Directly leads to low quality reviewing

Our Suggestions: Community

  • Clearer agreement between Program Chairs and ACL/ARR about decision making and policy ownership; eg. early communication about shared cycles and policies on Al usage.
  • ARR timelines should be shifted to better work with the physical constraints of the conferences and to improve review quality
  • Fewer submission cycles to reduce community overload:
    • Avoid shared cycles to preserve conference identity
    • Creates essential time for authors and volunteers to take breathing room and to work on policy changes to stay up-to-date.


Never miss a post

Get new writing sent to your inbox.

Subscribe