Homework #1 [Mostly due Sep 15]

[Thanks to James Allen and Jimmy Lin]

Please Read Chapter 3 on Evaluation.

Problem 1 (20 points) [DUE SEP 8 !!!!]

Run the following two queries on Google (http://google.com) and on Teoma (http://www.teoma.com/) (do not run the explanation; just run the query). You will judge 10 documents for relevance to the query. The explanation will help you decide whether or not something is relevant.

1. gardening wet soil conditions
   What special considerations must be made when planting a garden in very wet soil conditions? Are there plants that will not work and/or that will work particularly well? Are there any special techniques that will help reduce the amount of moisture? Only pages that deal with gardening are relevant.
   
2. oil vs. propane furnace
   Looking for pages that list the tradeoffs for using a propane furnace rather than a fuel oil furnace. The pages should provide comparison and are only relevant if they talk about both. They can talk about other types of heating (e.g., electric), provided they talk about oil and propane. Propane is also called LP or LPG (liquid propane [gas]).

When judging pages for relevance, please note the following:

• A page is relevant if it is on the appropriate topic. It is not relevant only because it has a link to a page that has relevant content.
• Google indents some of the items on its ranked list to indicate that they come from the same domain (roughly) as the preceding entry. Include those entries. There should always be 10 entries per page.
• If a page is not found when you follow it, use the cached version. If there is no cached version, make your judgment from the title and snippet. If you cannot make a call with that information, mark it as non-relevant.

To decide which 10 documents to judge, use the last digit of your student ID number:

• 0 to 5 – do the first page (results 1-10)
• 6 to 9 – do the second page (results 11-20)

Email your results in the following format to whomever we chose on Tuesday [Ben Perry <pianoman@udel.edu>]:

G-or-T   query-number  doc’s-rank  R-or-N    doc’s-title

where G-or-T indicates whether you ran this on Google or Teoma, query-number is 1 or 2 from above, doc’s-rank is the number from 1 to 20 of this document’s rank, R-or-N is “R” if the page is relevant and “N” otherwise, and doc’s-title is the title of the document (to help us verify that results make sense).

As an example, evaluating the first document retrieved by Google on the query "gardening wet soil conditions" might result in the following (fictitious) judgment:

G 1 1 N Gardening for dummies

Since you are judging 10 pages from each of two queries on each of two search engines, the file you email should have 40 non-blank lines.

Whomever we choose on Tuesday [Ben Perry <pianoman@udel.edu>] should email the pooled results back to everyone in the class by the end of the day SEPTEMBER 9!

Class mailing list is: <CISC887-010-05F@udel.edu>

Problem 2 (20 points) [Continuing Problem 1, after Sep 9]

In the second part of this assignment, for one of the topics [YOUR CHOICE--make this clear], you will

1. analyze agreement on the relevance judgments
2. adjudicate the judgments
3. use the adjudicated set to evaluate both Teoma and Google
   

2.1. Agreement on Relevance Judgments

There should be judgments from N different people for each document (Web page). The first question you'll answer is: How often do judges agree on relevance? There are four possibilities:

• all judges agree that the document is relevant.
• all judges agree that the document is not relevant.
• majority agree that the document is relevant.
• majority agree that the document is not relevant.
• equal split on relavant/not relevant.

For your chosen topic, figure out how often each case happens (both in terms of counts and in terms of percentage). Turn this information in. Pick two cases where judgments about a particular topic are not uniform, and briefly speculate why this may be so. Turn this in.

2.2 Adjudication

Adjudication is simply the process of reconciling inconsistent judgments. Do this by simple majority voting. If you have an equal split for a particular document then simply pick one randomly. The result should be something like this:
G N Gardening for dummies
G R How to deal with wet soil conditions
...

Turn your adjudicated relevance judgments in.

2.3. Evaluation of Teoma and Google

Now, evaluate Teoma and Google using the adjudicated relevance judgments you just created (for the topic you chose). Issue the query to Google and Teoma again, and examine the top 20 hits. Turn in the following information for both search system:

• Precision at 10, 20
• Recall at 10, 20
• Precision at each of the 11 standard recall levels (interpolated)
• draw the precision-recall graphs

In addition, answer the following questions:

• Which system is better? Why?
• What is the assumption we're making with respect to the completeness of the judgment set? What are the implications on our evaluation?
  

Problem 3 (15 points)

3.A. The following list of R’s and N’s represents relevant (R) and non-relevant (N) documents in a ranked list of 50 documents. The “top” of the ranked list is on the left of the list, so that represents the most highly weighted document, the one that the system believes is most likely to be relevant. The list runs across the page to the right and then starts on the next line. This list shows 8 relevant documents. Assume that there are an additional two (2) relevant documents that were not retrieved by this system.

R R N N N R N N N N        R N N N R N N N N R       R N N N N N N N R N
N N N N N N N N N N        N N N N N N N N N N

Based on that list, calculate the following measures:

1. Average precision
2. Precision at 50% recall
3. Precision at 33% recall (interpolated)
4. Assuming the ordering is simple, search length for n=4
5. Assuming a weak ordering with five values for ranking (resulting in the five groups of ten shown), what is the expected search length for n=4. That is, for this question, assume that documents in the first group all have the same score, the second group has another score, and so on.
6. What is the largest and smallest possible error in Mean Average Precision for the above system?
   

3.B. Now, Imagine another system retrieves the following ranked list for the same query.

R R R R N N N N N N        N N N N N N N N N N         N N N N N N N N N N N
N N N N N N N N N N        N N N N N N N N N R

Repeat parts (3.A.1), (3.A.2) and (3.A.3) for the above ranked list. Compare the two ranked lists on the basis of these 3 metrics that you have computed—i.e., if you were given only these 3 numbers (Mean Average Precision, Precision at 50% recall and Precision at 33% recall) what can you determine about the relative performance of the two systems in general?

3.C. Plot a recall/precision graph for the above two systems. Generate both an uninterpolated and an interpolated graph (probably as two graphs to make the four plots easier to see). What do the graphs tell you about the system in A and the one in B?