CISC889 - Project 2 results

This is just a collection of the final reports and results of the user evaluation. I'll try to write up a "conclusions" article over on my blog when I have some time. The findings were pretty interesting and you can get a good sense of things by reading the reports, then inspecting the clouds and seeing how the methods led to each success or failure.

The system labeled as "Manuel" uses human-generated tag clouds. Praveen and I went through each URL and generated a set of candidates until we were satisfied or bored, then associated scores with them. We picked a weight of 20 for the most important term then scored in the range 1-20 for the rest of them.

The system labeled as "Trnka reference" is the reference implementation I posted here. I developed it on my thesis using detex. The included results are nearly identical but added some code to use BoilerPipe to load the text and didn't lowercase the terms, which resulted in abysmal performance on the "Academic search committee code of conduct" article. If I get the chance, maybe I'll update it. The results and reports really showed me that I needed to merge terms more aggressively and needed to handle casing better. They also showed me that some degree of formatting-handling can drastically improve results. Oh and simple NP chunking works wonders.

Final reports

Team Java - Chris Boston, Philip Saponaro, Dan Waegel
report
Dongqing Zhu
report
Tim Walsh
report
Nicole Sparks and Charlie Greenbacker
report
Tim Armstrong and Zhou (Ivanka) Li
report

Per-URL Evaluation

The evaluation is the number of positive votes (preferred or equally good) divided by the number of times the URL was evaluated.

Piercing the Shroud (Starcraft 2 mission) (webpage, clouds)

  1. Manuel (24.4%)
  2. Dongqing (15.5%)
  3. Team Java (14.4%)
  4. Trnka reference (13.2%)
  5. Tim Walsh (12.1%)
  6. Tim Armstrong/Ivanka (9.5%)
  7. Nicole/Charlie (5.7%)

Higher Education Bubble (webpage, clouds)

  1. Manuel (23.6%)
  2. Team Java (18.8%)
  3. Tim Armstrong/Ivanka (16.6%)
  4. Trnka reference (14.4%)
  5. Nicole/Charlie (13.3%)
  6. Dongqing (9.6%)
  7. Tim Walsh (4.4%)

Jonathan Swift's A Modest Proposal (webpage, clouds)

  1. Manuel (25.6%)
  2. Team Java (14.9%)
  3. Tim Walsh (14.1%)
  4. Dongqing (13.8%)
  5. Nicole/Charlie (5.6%)
  6. Trnka reference (3.7%)
  7. Tim Armstrong/Ivanka (3.3%)

Academic search committee code of conduct (webpage, clouds)

  1. Manuel (24.5%)
  2. Team Java (13.9%)
  3. Tim Armstrong/Ivanka (12.1%)
  4. Dongqing (10.6%)
  5. Nicole/Charlie (10.3%)
  6. Tim Walsh (7.0%)
  7. Trnka reference (1.1%)

Han shot first (Star Wars) (webpage, clouds)

  1. Manuel (22.9%)
  2. Nicole/Charlie (17.3%)
  3. Tim Armstrong/Ivanka (15.1%)
  4. Trnka reference (13.7%)
  5. Tim Walsh (10.7%)
  6. Team Java (8.1%)
  7. Dongqing (7.7%)

Sale of Pyrex hurt the crack-cocaine industry (webpage, clouds)

  1. Manuel (24.3%)
  2. Trnka reference (21.7%)
  3. Tim Armstrong/Ivanka (21.0%)
  4. Dongqing (18.0%)
  5. Nicole/Charlie (11.1%)
  6. Tim Walsh (7.5%)
  7. Team Java (4.9%)

Improving sales by improving product review quality with Mechanical Turk (webpage, clouds)

  1. Manuel (26.3%)
  2. Dongqing (16.2%)
  3. Team Java (16.2%)
  4. Trnka reference (11.3%)
  5. Tim Armstrong/Ivanka (9.0%)
  6. Nicole/Charlie (8.3%)
  7. Tim Walsh (7.5%)

Fitts' Law (HCI metric) (webpage, clouds)

  1. Manuel (26.0%)
  2. Tim Armstrong/Ivanka (17.6%)
  3. Dongqing (12.8%)
  4. Trnka reference (12.5%)
  5. Nicole/Charlie (9.2%)
  6. Team Java (8.1%)
  7. Tim Walsh (4.0%)

LA food trucks (webpage, clouds)

  1. Manuel (22.4%)
  2. Trnka reference (17.5%)
  3. Dongqing (15.2%)
  4. Tim Armstrong/Ivanka (14.8%)
  5. Team Java (14.4%)
  6. Nicole/Charlie (10.3%)
  7. Tim Walsh (3.8%)

Tag cloud article (webpage, clouds)

  1. Manuel (25.0%)
  2. Team Java (17.8%)
  3. Trnka reference (17.4%)
  4. Dongqing (13.6%)
  5. Nicole/Charlie (10.2%)
  6. Tim Walsh (4.2%)
  7. Tim Armstrong/Ivanka (2.3%)

Aggregate results (average)

  1. Manuel (24.5%)
  2. Dongqing (13.3%)
  3. Team Java (13.1%)
  4. Trnka reference (12.6%)
  5. Tim Armstrong/Ivanka (12.1%)
  6. Nicole/Charlie (10.1%)
  7. Tim Walsh (7.5%)