conference post alt text.
EMNLP Paper Integrity Policy
Challenge #1: Systematic abuse of community resource
- There were organized efforts to submit fully Al-generated slop to EMNLP by the dozens - a waste of human review effort.
- The review system in its current form does not address this issue
- We are seeing scores of 3/3.5/4's given to many of such slop
- (more on reviews soon)
- There is no existing mechanism to automatically detect, or systematically penalize such abuse.
Challenge #2: Unverifiable citation
- Unverifiable citation counts do not always correlate with Al slop
- It does not identify paper mills
- Took ~10 people days to manually verify, not including AC effort, which took away valuable time for other organizational duties.
- We do not believe that citation checks, though with its own merit, is scalable, or effective for catching slop.
Problems with Al reviews
- Qualitatively, we observe the following issues with Al reviews at EMNLP and our Al review experiment partner from AAAI:
- The number of weaknesses is much larger than human reviews
- No notion of salience: which errors are major ones vs. minor ones
- Lack of "research taste"
- Does not call out slop papers
Lack of Engagement during Rebuttal
- Reviewers are not responding to authors during the response period
- Histogram of reviewer replies across ARR May 2026 by submission
- The vast majority of submissions received 0 replies (nearly 10,000), with the frequency sharply decreasing to around 4,300 for 1 reply, 2,200 for 2 replies, and dropping close to zero for 3 or more replies.
Other Compounding Challenges
- Reciprocal review requirements leading to authors "signing up" for their co-authors to review 20+ papers!
- We need a mechanism to handle frustrating behavior from program committee members
- sloppy Al reviews or meta reviews
- ghosting
- late responses and endless chasing of reviewers / ACS
ARR volunteers take on an enormous service load
- ARR May cycle received 3353 tickets in the May cycle
- EMNLP PCs received > 10,000 emails in the last year
- ARR team are volunteers who take on unimaginable service load
- While handling ARR tickets, we observed several emails with verbally abusive and intimidating language
- Running the infrastructure as-is requires full time tech team
- Running the submission cycle as-is is a full time job
Community overload: Challenges
- Double cycles across clearly distinct conferences (EACL/ACL, AACL/EMNLP, NAACL/COLING)
- Makes SAC recruitment harder and spreads their expertise more thinly. We handled this by strictly separating commitments.
- Tight, rigid cycles that don't work with physical times for conferences
- Directly leads to low quality reviewing
Our Suggestions: Community
- Clearer agreement between Program Chairs and ACL/ARR about decision making and policy ownership; eg. early communication about shared cycles and policies on Al usage.
- ARR timelines should be shifted to better work with the physical constraints of the conferences and to improve review quality
- Fewer submission cycles to reduce community overload:
- Avoid shared cycles to preserve conference identity
- Creates essential time for authors and volunteers to take breathing room and to work on policy changes to stay up-to-date.
Write a noteā¦Log in