Query Reformulation Experiment

Emily Hill, Lori Pollock, K. Vijay-Shanker
Natural Language Program Analysis (NLPA) Group
Department of Computer & Information Sciences
University of Delaware

 

Quick Navigation:
[Overview]   [Example Concerns]   [Part I Concerns]   [Part II Concerns]

 

Thank you for agreeing to particpate in our experiment. The study should take approximately 1 hour, with a maximum of 2 hours.

 

Overview

In this study, you will use two different searching mechanisms to locate 14 concerns, i.e., features, in Java code. A concern is some aspect of a software system of interest to a stakeholder, such as a developer or end user. For example, one concern of an ATM machine is the deposit capability. The implementation of the deposit concern would include the deposit menu as well as the back end logic that allows users to make deposits.

In this study you will be given 14 concerns, defined either by a snippet of documentation or a screenshot of the concern being executed. For each concern, you will formulate an initial query and view the results. You may continue to reformulate your query based on the results until you are satisfied with the query's ability to identify program elements in the concern's implementation. When reformulating the query you will be given access to method and field signatures (i.e., headers), but not the full code. A method signature will be displayed as the declaring class followed by the return type, the method name, and a list of the types and names for the parameters. Similarly, a field signature will be formatted as the declaring class, the type, followed by the field name. Each searching mechanism you use will display the results differently.

Your Task: Formulate a query that identifies a list of signatures that you believe has the best chance of implementing the concern. In other words, you are looking for a list of signatures that you, as a developer, would start exploring the code from to locate all the code for the concern. We provide two example concerns below to give you a better idea of the task.

After reading the instructions for each part of the experiment below, you may begin by clicking on a link to a concern. I would open the link in a new window, or have an extra window open with the concern description so you can refer back to the concern description if necessary and so you do not loose your place in the list of concerns once you've finished.

When finished with the study, please don't forget to complete the exit survey.

Example Concerns

9.5_-_ToInt32

The first example concern is for the Rhino program, a JavaScript/ECMAScript interpreter and compiler written in Java. For each concern, you are given the portion of the documentation that describes the concern.

From this snippet of documentation, I notice that the words "convert", "to", "int", "integer", and "32" look important. I begin my search by first entering in the query "convert":

However, there are just 8 results, and none of them look like they have anything to do with integers, numbers, or math. Seeing that the abbreviation for integer is used frequently in this snippet, I try "int" as a query, but that query returns 53 results, more than I want to look through. Therefore, I add the word "to" to the query, since I suspect that "to" and "int" are likely to occur together when describing this concern:


But I still get 30 results, many of which have nothing to do with converting integers. However, in scrolling down the list I see that the singatures "toInt32" look promising, so I reformulate my query to be "to int 32" instead:


And I find 3 method signatures that look like promising candidates for further investigation.

jBidWatcher_D

The next concern is for jBidWater, an auction bidding, sniping, and tracking tool for online auction sites such as eBay or Yahoo. jBidwatcher includes a unique and powerful multisniping feature. For this concern, you are given a screenshot of the concern being executed.

It appears that the screen shot is depicting the action of deleting an auction. The first query I formulate is "delete auction":

The query returns two results that appear to be highly relevant. But with just 2 results, I wonder if I could be missing other relevant pieces of code? Therefore, I reformulate my query to be just "delete" instead:

This query returns 19 results, and seems more comprehensive. After skimming the results, I see one or two signatures that do not look relevant, such as deleteSearch or DeleteComment. But overall, I think this is a more accurate list of the methods I will need to investigate to locate the concern, in contrast to the more specific query "delete auction".

Part I

In this part of the study, you will formulate queries for locating 7 concerns. Before describing the concerns, we will introduce the search tool you will be using.

Verb Direct-Object Search Tool

The Verb Direct-Object (Verb-DO) search tool takes verb and direct object queries as input and outputs a list of signatures. The query consists of a single verb, followed by a direct object which may be multiple words. Examples include "remove item" and "lookup performance event". The tool will always treat the first word of the query as a verb.

The image above shows a screen shot of the Verb-DO search tool in action. To use the tool:

  1. Enter your first name into the box. If you are not comfortable giving your name, you may use some other uniquely identifying word, but please be consistent across the entire experiment.
  2. Enter a query in the form of a verb followed by some object, and hit 'search'.
  3. View the results of the search.
  4. If you are not satisfied with the results of the search, scroll below to see possible alternatives for your search. The column to the left, labelled "Verbs", shows a list of the verbs that appear with the direct object in the query, sorted by frequeny. Similarly, the column on the right displays all the direct objects that appear with the noun in the query. In this example, "get" is the verb and "connection" is the object. Based on the verb recommendation list, I might decide that "get" is not the best verb, and that "use" may give more accurate results for the concern. I would reformulate the query as "use connection" and view the search results before hitting the 'Save and go to Next Concern' button.
  5. Repeat Steps 2-4 until you are satisfied with your query. Then hit the 'Save and go to Next Concern' button in the top left of the page. This will output your search results (so we can analyze the data later) and bring you back to this page so you can continue with the experiment.

Concerns

Rhino

The first 5 concerns in this set are for the Rhino program, a JavaScript/ECMAScript interpreter and compiler written in Java. For each concern, you are given the portion of the documentation that describes the concern (you may also browse the entire documentation). After reading the documentation, click on the link below each description and try to formulate a query that gives the most accurate list of signatures that you would need to investigate to locate the concern

9.2_-_ToBoolean

Formulate a query for 9.2_-_ToBoolean

 

10.2.1_-_Global_Code

Formulate a query for 10.2.1_-_Global_Code

 

15.4.2.1_-_new_Array

Formulate a query for 15.4.2.1_-_new_Array

 

15.9.5.26_-_getTimezoneOffset

Formulate a query for 15.9.5.26_-_getTimezoneOffset

 

15.11.4.3_-_message

Formulate a query for 15.11.4.3_-_message

 

iReport

The next concern is for the iReport program, a powerful, intuitive and easy to use visual report builder/designer for JasperReports written in java. iReport allows users to visually edit complex reports with charts, images, and subreports.

For this concern, you are given a screenshot of the concern being executed. After looking at the image, click on the link below and try to formulate a query that gives the most accurate list of signatures that you would need to investigate to locate the concern.

iReport_C

Formulate a query for iReport_C

 

jBidWatcher

The next concern is for jBidWater, an auction bidding, sniping, and tracking tool for online auction sites such as eBay or Yahoo. jBidwatcher includes a unique and powerful multisniping feature.

For this concern, you are given a screenshot of the concern being executed. After looking at the image, click on the link below and try to formulate a query that gives the most accurate list of signatures that you would need to investigate to locate the concern.

jBidWatcher_A

Formulate a query for jBidWatcher_A

 

Part II

In this part of the study, you will formulate queries for locating 7 more concerns. Before describing the concerns, we will introduce the search tool you will be using.

Context Search Tool

The Context search tool takes a query as input and outputs the search results as a hierarchy of signatures, thus providing some context for the results. The query mechanism is like a space-sensitive version of Google. For example, "getconnection", "GetConnection", and "GETCONNECTION" will all return the same results, but will return different results from "get connection". All queries are converted into lower case, and therefore the query is not case-sensitive.

When formulating queries, try to order the query terms the way you would expect them to occur naturally. The query "execute statement" will return different results from "statement execute".

The image above shows a screen shot of the Context search tool in action. To use the tool:

  1. Enter your first name into the box. If you are not comfortable giving your name, you may use some other uniquely identifying word, but please be consistent across the entire experiment.
  2. Enter a query and hit 'search'.
  3. View the results of the search by clicking on a phrase to see other matching phrases and signatures.

    Results are listed by most frequent subphrases to least frequent, followed by exact matches (listed last for when there are many exact matches). Numbers after a phrase indicate the number of matches to that phrase, which may be more than the number of signature hits (since a signature may map to multiple phrases).

  4. Repeat Steps 2-3 until you are satisfied with your query. Don't forget to reformulate your query so that only the relevant signatures are displayed. Then hit the 'Save and go to Next Concern' button in the top left of the page. This will output your search results (so we can analyze the data later) and bring you back to this page so you can continue with the experiment.

Try to order the query terms the way you would expect them to occur: "execute statement" will return different results from "statement execute".

Concerns

Rhino

The first 4 concerns in this set are for the Rhino program, a JavaScript/ECMAScript interpreter and compiler written in Java. For each concern, you are given the portion of the documentation that describes the concern (you may also browse the entire documentation). After reading the documentation, click on the link below each description and try to formulate a query that gives the most accurate list of signatures that you would need to investigate to locate the concern

12.6.3_-_for_Statement

Formulate a query for 12.6.3_-_for_Statement

 

15.2.4.6_-_isPrototypeOf

Formulate a query for 15.2.4.6_-_isPrototypeOf

 

15.8.1.8_-_SQRT2

Formulate a query for 15.8.1.8_-_SQRT2

 

15.10.7.4_-_multiline

Formulate a query for 15.10.7.4_-_multiline

 

Jajuk

The next concern is for Jajuk, a Java music organizer. The main goal of Jajuk is to provide a fully-featured application to advanced users with large or scattered music collections. Jajuk supports MP3, OGG Vorbis, AU, AIFF, WAV and SPEEX audio formats.

For this concern, you are given a screenshot of the concern being executed. After looking at the image, click on the link below and try to formulate a query that gives the most accurate list of signatures that you would need to investigate to locate the concern.

jajuk_P

Formulate a query for jajuk_P

 

javaHMO

The next concern is for javaHMO, which implements a media server for the Home Media Option from TiVo and adds new innovative features beyond the existing functionality provided by TiVo's implementation.

For this concern, you are given a screenshot of the concern being executed. After looking at the image, click on the link below and try to formulate a query that gives the most accurate list of signatures that you would need to investigate to locate the concern.

javaHMO_L

Formulate a query for javaHMO_L

 

jBidWatcher

The next concern is for jBidWater, an auction bidding, sniping, and tracking tool for online auction sites such as eBay or Yahoo. jBidwatcher includes a unique and powerful multisniping feature.

For this concern, you are given a screenshot of the concern being executed. After looking at the image, click on the link below and try to formulate a query that gives the most accurate list of signatures that you would need to investigate to locate the concern.

jBidWatcher_E

Formulate a query for jBidWatcher_E

 

Proceed to exit survey