"An Empirical Comparison of Seven Programming Languages"
A Paper Critique
CISC 470/670 - Programming Languages
Fall 2000
DUE DATE: Tuesday, November 21, 2000
NOTE: You may discuss the article and the questions below, but
each person should write up their own critique without collaboration.
Overview and Purpose:
This may be the first time you ever read and really probed into an article
that reports on an experiment conducted with the goal of comparing
several approaches to a computer science problem. In this case, the
author has compared the use of 7 different programming languages
as the different approaches
to writing a program for a single programming problem.
As a programmer, you should not just quickly choose
or recommend a particular language by reading the findings, without carefully
examining the methods for comparison and the collected data yourself.
This project has several different learning objectives with the overall
goal of helping you learn how to technically critique such an article. A
student should gain experience in:
- reading to understand a fairly technical computer science article
- examining the application of the experimental method in computer science research
- analyzing the parameters used in a computer science experiment for
validity and implications
- analyzing the data and comparing your own observations and conclusions
with the claims of the computer scientist
- formulating your own critique of a computer science experiment and article
as a succinct paper review
- gaining some insight into the issues and problems involved in comparing
different programming languages
Instructions:
You are taking on the role of a technical reviewer of an article that
has just been published in a journal.
The deliverable in this project is a paper critique that expresses your
views on the article, with particular attention to the following set of
questions. Your review should follow the formatting instructions given below
for readability by the instructor (who is over 40, and needs space in
case she needs to turn
black-and-white paper into full living color!).
In each question below, you need to give solid justifications for your
answers.
The questions to ponder:
- Summary of setup used: In one paragraph summarize the setup
and goals of the experiment. What was
actually used as the approaches, parameters, metrics measured, ...?
- Defining the experiment: You can imagine a very
large experiment if you were really interested
in comparing programming languages with the goal of uncovering
the strengths, weaknesses, and peculiarities of each clearly and completely.
So, what choices/assumptions/limitations did the author make in order to conduct
an experiment in a reasonable timeframe to get a paper published in our
lifetime? Do you agree with all these limitations, or would you suggest others
in their place knowing that the study has to be limited in some way?
- The representative problem used: What are your views on the use of the
phonecode problem for this kind of study? What do you think is
learned or missed by selecting this kind of problem? What implications
can be made from its results and what kinds of implications do you think
can't be made in terms of comparing programming languages?
- Data collection and presentation:
Another choice an experimental computer scientist needs to make is
how to collect and present their data. Do you think that this author has fairly
represented the data in a thorough manner with sufficient quantitative
methods?
Is it overdone? Is it not adequate? Are the charts readable? Are the
textual explanations understandable? How could
the data collection/presentation be improved?
- The conclusions: Do you agree with the conclusions based on analyzing
the charts yourself? Do you believe the conclusions are justified based
on the data? Which ones do you not agree with completely? Which ones do
you believe need further experimentation?
The author does state several caveats in his work. Do you believe they
are adequate, and do you believe that the "significant findings" section
is an accurate portrayal of the data given the caveats mentioned?
- What additional experiments would you suggest to continue the comparison
study?
(additional parameters,...)
- What have you learned about these programming languages that you did
not realize prior to reading this article? Is there anything that
strikes you as counterintuitive about the significant findings, or
are the results the ones you would have expected,
and thus this is an experiment
confirming preconceived beliefs about these languages?
What to Hand in:
You should physically hand to the instructor
a hardcopy of your essay (produced by
a word processor; no handwriting please!) at the beginning of
class on the due date.
Formatting Instructions:
Please format your essay in double (or 1 1/2) spacing, with 1 inch margins all around.
Please use 10 or 11 point font.
Please limit your essay to less than 5 pages, if possible.
Last Change: November 6, 2000 /
pollock@cis.udel.edu