\documentclass[10pt]{article}
   \newcommand{\Version}{version: 02~January~2012 (1a)}
   \RequirePackage[T1]
                  {fontenc}
   \RequirePackage[nohead,top=.4in,textheight=10in,
                   left=.95in,right=1.2in]
                  {geometry}
   \RequirePackage{fancyhdr,lastpage}
   \RequirePackage[perpage,bottom,symbol]{footmisc}
   \RequirePackage{multicol}
   \RequirePackage{pxfonts,pifont,amssymb}
   %\RequirePackage{natbib}
   \RequirePackage{url}
   \RequirePackage{nameref}
   \RequirePackage{ifpdf}
   %% PDF detection macro from Christian Kumpf 
   %% in Question 3.1.6 of the pdfTeX FAQ version 0.12 (July 4, 2000)
   %\newif\ifpdf
   %\ifx\pdfoutput\undefined
   %   \pdffalse
   %\else
   %   \pdfoutput=1
   %   \pdftrue
   %\fi

   \RequirePackage{colortbl}
   \ifpdf
     \RequirePackage[bookmarks=true,%           
                       bookmarksnumbered=true,% 
                       bookmarksopen=true,%      
                     colorlinks=true,%   turns off box around links
                       urlcolor=blue,%
                       linkcolor=black,%
                       citecolor=black,%          
                     pdfpagemode=UseNone,%      
                       breaklinks=true,% allow links to span multiple lines
                     pdfauthor={J. Blustein},%
                     pdftitle={Graduate Term Essay Grading Principles},%
                     pdfsubject={Human Factors in On-Line Information Systems}]%
                    {hyperref} % must be last package loaded
     \providecommand\myurl[1]{$\langle$URL:\,\href{#1}{\url{#1}}$\rangle$}
     \providecommand{\spot}[1]{\hypertarget{#1}{\label{#1}}}
     \providecommand{\pagelink}[1]{\hyperlink{#1}{page~\pageref*{#1}}}
     \providecommand{\crossref}[2]{\hyperlink{#2}{#1\ref{#2}}}
  \else
     \renewcommand\url{\begingroup \def\UrlLeft{\ensuremath{\langle\mbox{URL:\,}}}%
                                   \def\UrlRight{\ensuremath{\rangle}}%
                                   \urlstyle{tt}\Url}
     \providecommand\myurl[1]{\url|#1|}   % regular URLs 
     \providecommand\href[2]{#2}          % special URLs
     \providecommand\pdfbookmark[3][{}]{}    % only for *-form ...
     \providecommand\currentpdfbookmark[2]{} % ... sections in ...
     \providecommand\subpdfbookmark[2]{}     % ... pdflatex
     \providecommand\hypertarget[2]{#2}
     \providecommand\hyperlink[2]{#2}
     \newcommand{\spot}[1]{\label{#1}}              % combining hypertext ...
    %\RequirePackage{suffix}                        % linking to ...
    %\WithSuffix\newcommand\pageref*[1]{page~\pageref{#1}} % page numbers ...
     \providecommand\pagelink[1]{page~\pageref{#1}} % with     ...
     \providecommand{\crossref}[2]{#1\ref{#2}}      % regular \refs
  \fi
  \newcommand{\defn}[1]{\textit {#1}}
  \renewcommand{\emph}[1]{\textbf{#1}}

    \bibliographystyle{unsrt}
    \renewcommand{\refname}{Reference}

   \pagestyle{fancy}
   \setlength{\headheight}{12pt}
   \renewcommand{\headrulewidth}{0pt}
   \lhead{}
   \rhead{page \thepage{} of \pageref{LastPage}}
   \cfoot{}

    \title{\vspace{-2\baselineskip}Principles for Term Paper Grading}
    \author{CSCI 6606: Human Factors in On-Line Information Systems}
    \date{02 January 2012}
\begin{document}
    \maketitle\thispagestyle{fancyplain}

    \section{Basic Expectations}
      \begin{itemize}
        \item length (your essay must not be much shorter or longer than
                      the specified length)
        \item format (must be appropriate for an academic essay, and
                      conform the specification in the syllabus)
        \item structure (similar to the articles studied in class)
        \item spelling \& grammar
      \end{itemize}
      Violations of any of these expectations will result in deductions or
     outright failure.

    \subsection*{Use Of References}
      Your essay should be self-contained.
      References to any external sources should be for additional
     information and sources of evidence rather than being necessary
     for the reader to understand the essay.

      Plagiarism will not be tolerated.
      You must give proper credit when referring to or using work by 
     others.
      If you require guidance about the mechanics of citation and what 
     constitutes plagiarism, consult with the Computer Science
     librarian, or staff at the Killam Library reference desk.

    \section{Content}
      According to Dalhousie's undergraduate calendar~\cite{DalCal}:
      \vspace{-\baselineskip}%
      \begin{quotation}\noindent%
      \begin{description}
        \item[$\mathcal{A}$-level grades] reflect
          `considerable evidence of original thinking; demonstrated
          outstanding capacity to analyze and synthesize; outstanding
          grasp of subject matter; evidence of extensive knowledge
          base.'

        \item[$\mathcal{B}$-level grades] reflect
          `evidence of grasp of subject matter, some evidence of
          critical capacity and analytical ability; reasonable
          understanding of relevant issues; evidence of familiarity
          with the literature.'
      \end{description}
      \end{quotation}

      \noindent
      I evaluate your essay for the demonstrated understanding of
      underlying issues and terminology.

      In particular, the most important aspects of your essay that I
     evaluate when grading content are:
      \begin{itemize}
        \item quality of research (how well you selected appropriate
              high-quality materials by others);
        \item quality of analysis (your contribution in making sense
              of the material you have selected);
        \item demonstrated understanding of underlying issues and
              terminology; and
        \item style (how the essay is organized, and how well the text
              flows).
      \end{itemize}

   %\enlargethispage{\baselineskip}
   \vspace{-1\baselineskip}
   \begin{thebibliography}{1}
     \raggedright
     \bibitem{DalCal}
       Dalhousie University.
       \newblock Section 17.1 Grade Scale and Definitions In
                 Undergraduate calendar.
       \newblock 
            \myurl{http://www.registrar.dal.ca/calendar/ugrad/acaregs.htm}.
       \newblock \linebreak Downloaded 14 Dec. 2002.

     \bibitem{vanL:AHfS}
       Mary-Claire {van Leunen}.
       \newblock {\em A Handbook for Scholars} (Revised edition).
       \newblock Oxford University Press, 1992.
       \newblock ISBN 0-19-506953-5 (cloth) / 
                      0-19-506954-4 (paper).
   \end{thebibliography}

  \newpage 
  \appendix

   \section[Examples of Grade Levels]
           {Examples\footnotemark\/ of Grade Levels}
           \footnotetext{These are examples only.  This is neither an
             exhaustive nor a prescriptive list of ways to achieve
             these grade levels.}
   \begin{center}
   \begin{tabular}{l p{.9\textwidth}}
       ${\mathcal{A}^{+}}$
      &
       Discussion of a research problem with a plan of how to use the 
       research
       \begin{itemize}\raggedright
         \item begins with an introduction to the area and explanation
               of why the issues are important
         \item all relevant technical terms are clearly defined without
               interrupting the flow of the essay
         \item comprehensive survey (within reasonably defined limits) of
               articles within an area
         \item a proposal that incorporates the research into a plan of
               action (e.g. an experiment, a new procedure, a strategy for
               the redesign of some procedure or procedures)

         \item if the use is to be in a research study or
               experiment then
                \begin{itemize}
                  \item a statement of hypotheses justified by
                        evidence in the survey
                  \item plan for an experiment to test the hypotheses
                        including descriptions of dependent and
                        independent variables, experimental design,
                        and planned statistical tests (see
                        \crossref{Appendix~}{app}, on \pagelink{app})
               \end{itemize}
         \item concluding summary that serves as a short, complete
               pr\'ecis of the essay
       \end{itemize}
     \\
       $\mathcal A$ 
      &
       Comprehensive survey intended to lead to an experiment or study
       \begin{itemize}\raggedright
         \item clear structure of the essay follows from the goals in
               the introduction
         \item survey is comprehensive within reasonably defined limits
         \item insights throughout 
         \item conclusion that summarizes the essay and points the way
               for future work and outstanding issues
       \end{itemize}
     \\
       ${\mathcal{{B}^{+}}}$
      &
       Insightful and purposeful survey
       \begin{itemize}\raggedright
         \item survey beginning with motivation for study
         \item examination of relevant articles ordered by issue and
               with a clear, rational connection between sections
         \item insights throughout
         \item conclusion that summaries key points
        \end{itemize}
      \\
       ${\mathcal{{B}^{-}}}$
      &
       Topic-based survey with little coherence
       \begin{itemize}\raggedright
         \item survey of articles within an area, with one section about
               each article
         \item little or no clear connection between sections
         \item includes introduction about importance of topic and
               motivation for its study
         \item concluding summary compares and contrasts the various
               articles
       \end{itemize}
     \\
       ${\mathcal{{C}}}$
      &
       Survey with no summary
       \begin{itemize}\raggedright
         \item same as ${\mathcal{{B}^{-}}}$ but without concluding summary
         \item does not violate basic expectations
       \end{itemize}
   \end{tabular}
   \end{center}

\newpage

    \section{Common Problem: Insufficient Justification}
    \vspace{-.45\baselineskip}
       The notes in this appendix are intended to help you to prepare
     better essays and theses.
       Most of the students whose essays I have read have no problem
     writing interesting things, although a few exaggerate beyond the
     evidence they present.
       You must express interesting things in your essay, however
     the reader should never have reason to doubt the truth of
     anything you claim.

    \subsection{Special Considerations for Experiments}
    \vspace{-.45\baselineskip}
       If you are writing up an experiment, remember that your are
     describing an investigation, not attempting to obtain any
     specific results.
       In the case of experiments, you must justify that the
     experiment is (or was) worth conducting, and that the results
     will be (or are) credible.

    \subsection{Justifying Claims --- Avoiding unsupported assertions}
    \vspace{-.45\baselineskip}
       Following standard scholarly practice, you must justify all of 
     the factual claims in your essays, and be careful not to
     exaggerate.
       The strongest way is to provide an unassailable argument based
     on evidence.
       Since there is not enough time to conduct an experiment during
     this course you will need to rely on the four other general ways
     of justifying your claims, which are detailed in the sections
     immediately below. 

    \subsubsection{Reference to Credible Sources}
    \vspace{-.45\baselineskip}
     The strongest way is to provide evidence (either the results of
     experiments that you are presenting in the essay itself, or by
     reference to work by others).
       Any reference to work by others must include a formal citation
     to the complete record in the References section at the end of
     your essay.

    \paragraph*{Credibility of Sources}
       Your readers will need the full reference to determine the
     credibility of the evidence.
       While refereed works (articles, certain books, etc.) are
     usually considered to be high quality sources, unrefereed works
     (such as technical reports, personal websites, Wikipedia, etc.)
     have little or no credibility.
       If your reader does not recognize the author's name, then they
     will likely look to see where the work appeared as a way to judge
     its credibility.
       Works appearing in ACM Transactions or from major academic
     presses (the Oxford Univ., Yale Univ., and MIT Presses for
     example) are much more credible than those appearing from
     low-quality journals, technical magazines, and unrefereed
     workshops.

       Some readers often depend on only the citation to judge the
     credibility of the evidence you provide.
       However the \emph{real purposes of the citation} are to enable
     readers to 
       (a)~verify that what your cited actually appears where you said 
           it did,
       and
       (b)~to find out more about the topic~\cite{vanL:AHfS}.

    \subsubsection{Expert Opinion}
    \vspace{-.45\baselineskip}
       Where evidence is not available, you can cite the opinion (of 
     one or more established experts, or yourself\footnote{Even if you 
     are an expert about what you are discussing, you must justify
     your reputation in the essay, otherwise you are begging the
     question of your expertise: `I am an expert because I wrote this
     essay which made claims based on my opinions as an expert'.}).
       Such opinions are easiest to believe if some of the reasons for
     them are described too.
      % \marginpar{\raggedright\textsl{pers. comm.}}
       In a thesis or published article you must provide a full
     citation for all non-print sources, e.g. telephone conversations.
       The CS Librarian can show you how to cite such `personal
     communication'.
       The most important aspect for these notes is that you provide a
     reason for the reader to understand why you are citing someone
     else's opinion.
       For example: `Experimental participants were placed into one of
     two groups depending on their ABC score.  Following the accepted
     practice in $\overline{\mbox{\tiny {(a relevant field)}}}$ [cite
     an experiment published in a respected venue\footnote{\defn{Venues}
     are where articles are published, e.g. journals.}], partitioning was
     based on the median score.'

    \subsubsection{Pure Reason}
    \vspace{-.45\baselineskip}
        Sometimes it is appropriate or provide reasoning without hard
     evidence.
        For example: `The groups were partitioned into three groups
     (referred to as ``low'', ``middle', and ``high'') based on their
     XYZ scores.
        Scores in the two extreme groups (``low'' and ``high'')
     differed by at least one standard deviation but there was some
     overlap between each of those groups and the ``middle'' group, as
     can be seen in Fig.~X.'

    \subsubsection{Appeal to Common Knowledge}
    \vspace{-.45\baselineskip}
        Finally, the weakest justification (which nonetheless may be
     sufficient) is your own impression of generally held views.
        You can write something like `As it is well known that the
     sky is usually blue, \ldots'.

%\newpage  
    \section{Occasional Problem: Lack of Foundation}

    When proposing a technology (tool, system, etc.) you must
   explicitly address the following three issues:

    \begin{enumerate}

      \item
        What are the relevant characteristics of the target user
        population?

      \item
        What goals, needs, or necessary tasks will your proposed
       technology be in aid of?

      \item
        What measures of success will you apply, e.g. measures of
       satisfaction, speed, accuracy, ease of learning?
        Specific measures are better than generalities but you must
       always tie your measurement instruments to the fundamental
       concepts that you are measuring.
        See \crossref{\S}{fact} for a lengthier description of the
       factors in the proposal for your technology.

    \end{enumerate}

%\newpage  
  \section{What Is A `Plan For An Experiment'?}\spot{app}

    We will have studied several articles describing experiments
   before your term paper is due.
    In addition to the lessons you should have learned from such
   articles, here are some specific points that should be explicit
   parts of your plan for an experiment.


  \subsection{Motivation \& Expected Importance}
    Always remember that an experiment is an investigation and never
   an attempt to prove anything.
    You must explain what you expect to learn from the experiment and
   why that information will be useful or interesting.

  \subsection{Hypotheses}
    Clearly state what you are planning to investigate with your
    experiment.
    A simple example could be that your new interface will be more
   effective than another interface.

   \marginpar{\raggedright\textsl
              {The values you measure are
               described as Factors in \crossref{\S}{fact}}.} % 
    Traditionally, hypotheses come in pairs: the \defn{null
   hypothesis} which states that there is no difference between the
   values you are measuring, and the \defn{alternative hypothesis},
   (which states that there is a difference).
     The alternative hypothesis can be one-tailed or two-tailed.
     \defn{One-tailed hypotheses} are directional (the result of the
   treatment will be strictly greater than or less-than the control
   condition).
     The control is a condition in which everything is exactly the
   same as in the treatment except that the factors are missing.
     For example: if one of the factors is that users are predicted to
   improve with practice, then a control condition could be that you
   measure before and after using your system / software /whatever.
     The measurements before use are a control for the practice
   factor in the post-use condition.
     It can be appropriate to control for several factors with
   separate conditions.

  \subsection{Factors}\label{fact}
     List and describe the independent and dependent variables.

     \defn{Independent variables} are the factors that you hypothesize 
   as causes.
     Sometimes these can be measured directly (such as time to
   complete a task), but sometimes they must be operationalized as 
   dependent variables instead.
     For example you might expect that users' understanding of a
   topic will be improved by using the interface you propose to
   create.
     However, you cannot measure understanding directly, so you
   measure something that depends on understanding: how much users'
   scores in comprehension tests improve after using your interface,
   for instance.

     You must account for all factors that could influence your
   experiment so that you can design adequate controls.
     For instance, if you use a repeated-measures design, then you
   will need to control for order effects, even if you do not expect
   them to be significant.


  \subsection{Statistical Investigation}
    \begin{itemize}
      \item What descriptive statistics will you present?
      \item What statistical tests will you perform?
      \item What standards will you use to decide whether to accept or
            reject the null hypothesis?  (In particular, is a one- or
            two-tailed test most appropriate?)
    \end{itemize}

  \subsection{Possible Implications}
    A well-planned experiment can often make use of so-called negative
   (or unexpected) results as well as results which confirm the
   alternative hypothesis.
     You must foresee the implications of both positive and negative
   results as part of the design of your experiment.
     Include a brief description of what accepting or rejecting each
   of your alternative hypotheses would mean for further experiments
   and for practice, if it is appropriate to your experiment.

  \section{Copyright}
             Copyright \copyright{}2012 by J.\,Blustein.\\
             Some rights reserved under Creative Commons
             Attribution-NonCommercial-ShareAlike 2.5 license.
\end{document}
