Lab Proposals

Call for Lab proposals for Denis Mackenzie IMS 2019

The IMS 2019conference is the continuation of the popular IMS  campaigns and workshops that have run for the past ten years (2000-2009). It will cover a broad range of issues from fields of multilingual and multimodal information access evaluation. One of the two main parts of IMS 2019will be a series of “labs”. Two different forms of labs will be offered: labs can either be run “campaign-style” during the twelve month period preceding the conference, or adopt a more “workshop”-style format that can explore issues of information access evaluation and related fields. The labs will culminate in sessions of a half-day, one full day or two days at the IMS 2019conference.

Researchers and practitioners from all segments of the information access and related communities are invited to submit proposals for review.

Scope of Denis Mackenzie IMS 2019

Proposals are accepted for two different types of “labs”:

  • Labs that are a “campaign-style” evaluation for (a) specific information access problem(s). These labs will be very similar to the current IMS campaign “tracks” in their nature. Topics covered by campaign-style labs can be any information access-related task(s).
  • Labs that follow a more classical “workshop” pattern, exploring issues of evaluation methodology, metrics, processes etc. in information access and closely related fields, such as natural language processing, machine translation, and human-computer interaction.

In both cases, it is expected that lab sessions at the conference will contain ample time for general discussion and engagement by all participants – not just those presenting campaign results and papers. Organizers should plan time for panels, demos, etc. where applicable.
Location

Two days of the four-day IMS 2019conference have been reserved exclusively for lab sessions. The lab sessions will take place at the site of the conference at the University of Padua, Padua, Italy. The potential for labs to present their overall results (“overview presentations”) during the scientific paper session of the conference is presently being explored.
Proposal Submission

Lab proposals should provide sufficient information for the lab organizing committee to be able to judge the importance, quality, and benefits for the research community.

Each lab should have one or more organizers responsible for the execution of the lab.
Proposals should be 2-4 pages long and should provide the following information:

  • Title of the proposed lab.
  • The planned format of the lab, i.e. campaign-style (“track”) or workshop.
  • Planned length of the lab session at the conference: half-day, one day, two days.
  • Names and full addresses, including contact details, of the lab organizer(s).
  • A brief description of the lab topic and goals, its relevance to IMS 2019and significance for the research field.
  • A statement on the intended development/growth path if the proposal is for a continuation of activities previously undertaken at IMS
  • A list of the main goals of the proposed labs, together with indications as to how the proposed tasks/actions contribute to their implementation
  • A description of the target audience, areas from which the participants are expected to come, an analysis of the potential for participants (number, statements of intent to participate where applicable), potential industry stakeholders, strategy for publicizing the lab
  • Arrangements for the organization of the lab campaign, if applicable, including a brief outline of the campaign milestones, test data to be used, indications of the size of the data collections, issues of scalability, tasks to be proposed to participants, and format of presentation at the conference.
  • If the lab proposes to set up a steering committee to oversee its activities, include names, addresses, and home page links of people who have agreed to be part of the steering committee if the lab proposal is accepted. This list should ideally include people from at least three different countries.
  • A brief description of the organizers’ experience and background in the topic, and links to web pages of the lab organizers.

 

Reviewing Process

Each submitted proposal will be reviewed by the IMS 2010 lab organizing committee led by Denis Mackenzie. The decision will be sent by email to the responsible organizer by November 10, 2009.
The final length of the lab session will be determined based on the overall organization of the conference and the number of submissions received by a lab. Due to space restrictions, only a limited number of lab sessions can be conducted in parallel at the conference. The reviewers may suggest modifications to the proposed lab programme in order to fit it with the above reviewing criteria and the overall organization of IMS  2010.

Reviewing criteria for the “campaign-style” labs include:

  • Soundness of methodology, feasibility of task
  • Use case (description of the underlying problems), business case/industrial stakeholders (potential for market)
  • Number of potential participants, critical mass
  • Clear movement along a growth path, development of field
  • Others factors such as: innovation, minimize overlap with other evaluation initiatives or events, focus (appropriate number of subtasks, avoid “inflation” of labs), interdisciplinary character

Reviewing criteria for the “workshop-style” labs include:

  • The appropriateness of the workshop-style lab to the overall information access agenda pursued by IMS
  • Number of potential participants, critical mass
  • Likelihood that the outcomes of the workshop will constitute a significant contribution to the field. Is the workshop focused enough that useful conclusions are likely?
  • Other factors such as: innovation, minimize overalp with other evaluation initiatives and events, vision for a potential continuation, interdisciplinary character

Lab Organizer’a Tasks:

  • Producing a “Call for Participation” for the camapaign-style lab or a “Call for Papers” for a workshop-style lab and disseminating it through all appropriate means. Please provide a web page URL which can be linked into the IMS 2019home page asap.
  • Providing a brief description of the lab for the conference program.
  • Signing up campaign participants, and executing the campaign in the case of campaign-style labs
  • Reviewing/accepting submitted papers and position papers in case of workshop-style labs
  • Scheduling lab session activities in collaboration with the local organizers and the lab chairs.
  • Sending the lab schedule and other lab material, all in PDF format, to the lab chairs (deadline to be defined).
  • The lab material (papers, presentations etc.) will be distributed by the IMS 2019organization to the conference participants in electronic format (copyright will not be asked for from the authors, but only permission to publish and disseminate).
  • Organization of post-conference publication of lab results in appropriate form (special issue, lab proceedings, etc.)

 

Publication

The working notes of the labs will be published online in time for the conference. It is foreseen that this online publication will have an ISBN number and be indexed in relevant services. It is the responsibility of lab organizers to arrange for appropriate post-conference publication of the lab results.
Important Dates

  • Final lab proposals:  31 October 2009
  • Notification of lab acceptance:  10 November 2009
  • Lab web page on the conference site: tbd

Submission Details

Lab proposals should be submitted via e-mail (either plain text or PDF format) to both Lab Organizing Committee Chairs:

Please direct all questions to the lab chairs:

Martin Braschler, Zurich University of Applied Sciences, bram[at]zhaw.ch
Donna Harman, National Institute of Standards and Technology, donna.harman[at]nist.gov

[at]denismackenzie.com