ACM SIGMOD/PODS Conference: Vancouver, 2008 SIGMOD: Guidelines for Research Papers Experimental Repeatability Requirements To help published papers achieve an impact and stand as reliable reference-able works for future research, the SIGMOD 2008 reviewing process includes an assessment of the extent to which the presented experiments are repeatable by someone with access to all the required hardware, software, and test data. Thus, we attempt to establish that the code developed by the authors exists, runs correctly on well-defined inputs, and performs in a manner compatible with that presented in the paper. Papers that are accepted and are verified this way will be eligible to include the following text in the proceedings: "The results in this paper were verified by the SIGMOD repeatability committee" . If a subset of the results cannot be verified by the SIGMOD repeatability committee (e.g., for IP reasons), then the phrasing will change to: "The results in Figures and Tables in this paper were verified by the SIGMOD repeatability committee." If verified code and data is also made available for archiving, the following phrase may be added: "And the code and data are available at ." If we test your code for repeatability, we will give feedback about any problems we encounter. This will not influence whether your paper is selected or not for publication. Submission of code and data is optional for SIGMOD 2008. that is, submitting code and data will have no influence on whether your paper is accepted for SIGMOD 2008. If you choose not to submit, however, then you are still required to submit, on December 16, 2007, a note stating: (1) the reasons you feel you could not submit; (2) how much time, if at all feasible, it would take you to satisfy these requirements; and (3) any suggestions you have about how to achieve the goal of scientific repeatability in our field. If you choose to submit code and data for at least some of your reported experiments, please follow the requirements below. Authors are required to upload to a special website (to be announced on this page), at the latest on Dec. 16, 2007 (one month after the SIGMOD paper submission deadline): 1. The code needed to run the experiments quoted in the paper. Executable code (strongly preferred) or source code are both acceptable. If source code is provided, any other software (libraries etc.) required in order to run the software should also be included. For third-party software that can be downloaded for free, only download URIs are required. Executable code should run on one of the following platforms: * Windows XP * Linux Fedora Core 6, Linux Redhat 9, or Linux Mandrake 10.1 * Mac OS X 10.4 To facilitate the repeatability assessment, authors submitting executable code are strongly encouraged to produce an image with no or very few dynamically bound libraries. Software used for processing raw results should also be described and, if it is developed by the authors, provided. If intellectual property or privacy issues prevent sharing the code, a description and explanation of these issues should be provided. If some code is proprietary but other code is not, then the non-proprietary code should be provided. 2. The data sets used in the experiments. If the data sets come from standard data sources unaffiliated with the authors, only a download URI is required. If the data sets are produced by some generator, the generator itself together with the configuration used by the authors are preferable to the actual data. If intellectual property or privacy issues prevent sharing the data sets, a reduced sample or similar synthetic data set may be provided. If some data is proprietary but other data is not, then the non-proprietary data should be provided. 3. A plain-text file named INSTALL, describing: * the hardware and software platform required in order to run the experiment * the sequence of installation steps It is the authors' responsibility to make sure the provided software installs correctly on one of the platforms specified above. 4. A plain-text file named HOWTO, describing for each table or figure in the paper for which code and data were supplied: * the formats of the expected input and the interpretation of the produced output; * which data sets to use; * how to run the experiment. We encourage authors to provide simple, command-line-style interfaces for their code. An example is provided here: %% Sample experiment tarball. %% The data, code and instructions available %% here are meant to allow reproducing some measures from the paper %% "Towards micro-benchmarking XQuery". We consider the experiments %% from Section 3.5 of the paper (Figure 15, part of Figure 16, Figures %% 17 and 18). Figure 16 involves a document that is not provided in %% the tarball. Repeatability Expectations The experimental evaluation committee has reasonable expectations regarding the similarity to be observed between the experiment results as observed by the authors, and as observed by the committee. For instance, there is no way to ensure that the hardware used by the authors is available to the experimental evaluation team. Therefore, we do not expect measured execution times to match those reported in the paper, but rather roughly similar curve tendencies (unless the hardware used by the authors and unavailable to us enables specific optimizations, such as e.g. parallel processing on a multi-processor machine). When measuring other things than running time, such as e.g. result sizes in experiments with no randomized component, we do expect to obtain the results presented in the paper.