System Description Papers

Participants who submitted a system to the shared task and filled the form about their systems are invited to write a system description paper, to be published in the proceedings of WASSA 2018 (part of EMNLP proceedings).

Please submit your paper for review until 6th of August at the WASSA submission page and select “Click HERE to make a new Submission to WASSA shared task.”

Reviews will be between participants, i.e., each participating team will review the submissions of other teams.

The camera ready version of the shared task paper is now available here.

Important notes

  • You are not obligated to submit a system-description paper, however, we strongly encourage all participating teams to do so.
  • You cannot submit essentially the same work in more than one form to WASSA-2018. For example, you cannot submit essentially the same work as a system description paper and also as a main workshop long paper. You have to choose one.
  • There is no difference in how a paper will be cited whether it is in the shared task track, or in the main workshop. Of course, only a small proportion of the papers submitted to the main workshop track will be accepted, whereas almost all papers submitted to the shared task track will be accepted.
  • You are free to submit an extended version of the WASSA shared task paper to another future venue (workshop, conference, or journal). You may want to check the policies of the venue for submissions that overlap with previously published papers.

Specifications of the system-description paper:

  • The paper is to be four to six pages long, plus references. We encourage writing at least four pages to make sure you have included sufficient details so that others can replicate your system, and so that the paper includes sufficient experimental analysis to make the paper interesting for readers. Even though you can use up to six pages, we encourage being concise and not using pages 5 and 6 unless the work and analysis warrants it. The papers are to follow the format and style files provided by EMNLP-2018. The review process is single-blind. Submissions are not anonymous. You are to enter your names and affiliations on the paper. Paper titles are required to follow this format: “[Insert System Name Here] at IEST 2018: [Insert Suitable Paper Title Text Here]”

What to include in a system-description paper?

  • Replicability: present all details that will allow someone else to replicate your system
  • Analysis: focus more on results and analysis and less on discussing rankings; report results on several variants of the system (even beyond the official submission); present sensitivity analysis of your system’s parameters, network architecture, etc.; present ablation experiments showing usefulness of different features and techniques; show comparisons with baselines. The labels of the test data are available now in the data section of this homepage. We will award the best system analysis based on the system description paper, independent of the score of the submission!
  • Avoid Duplication: Cite the task description paper that will be written by the task organizers (a draft is available here, version of Friday, the 27th of July). This paper will provide details of the task, summary of data creation, competition results, and a summary of participating submissions. You can avoid repeating details of the task and data in your paper, however, briefly outlining the task and relevant aspects of the data is a good idea.
@InProceedings{Klinger2018x,
  author =     {Roman Klinger and Orph\'ee de Clercq and Saif
                  M. Mohammad and Alexandra Balahur},
  title =     {IEST: WASSA-2018 Implicit Emotions Shared Task},
  booktitle =     {Proceedings of the 9th Workshop on Computational
                  Approaches to Subjectivity, Sentiment and Social
                  Media Analysis},
  year =     {2018},
  address =     {Brussels, Belgium},
  month =     {November},
  organization = {Association for Computational Linguistics}
}
  • It may also be helpful to look at some of the papers from past SemEval competitions, e.g., from https://aclweb.org/anthology/S/S16/.

My system did not get a good rank. Should I still write a system-description paper?

We encourage all participants to submit a system description paper. The goal is to record all the approaches that were used and how effective they were. Do not dwell too much on rankings. Focus instead on analysis and the research questions that your system can help address. What has not worked is also useful information.