PARSEC Benchmark Suite 3.0
==========================


1.) Overview

The Princeton Application Repository for Shared-Memory Computers (PARSEC) is a
collection of parallel programs which can be used for performance studies of
multiprocessor machines.

The PARSEC distribution is composed of packages and the PARSEC framework.
Packages correspond to benchmark programs, libraries and other essential
components. Each package can be compiled in a number of ways as determined by
a build configuration. Build configurations contain information such as which
features of the package should be enabled, which compilers to use and how the
package should be optimized. PARSEC ships with predefined inputs that can be
used to run the benchmarks. The inputs for each program exhibit different
characteristics such as execution time and working set size.


2.) Licenses

Before you start using, modifying or distributing PARSEC, its programs or the
supplied inputs in any way, make sure you understand all licenses involved.
The PARSEC framework itself is available under a liberal open source license,
as explained in the file LICENSE which is in the same directory as this README
file. Each program uses its own license, which is different in some cases. Some
of the inputs have their own license, too. Licenses for source code can be found
in the 'src/' directory of each package. Licenses for inputs can be found in the
'inputs/' directory of the package.

We distribute the programs and their workloads bundled with PARSEC merely to
allow PARSEC users a convenient access to them and because the license terms
allow us do so in each case. You have to take appropriate steps yourself to make
sure you don't violate any license terms.


3.) Requirements

PARSEC requires at least 8 GB, but we recommend 12 GB or more. The disk usage
can be broken down as follows:

PARSEC occupies about 7 GB with a raw installation. Additional 500 MB are
needed for each set of binaries. To build packages and run them extra space is
required for temporary files, up to several GB if the whole suite is to be
built and executed in one run without cleaning up intermittently.

The PARSEC benchmarks have been parallelized with pthreads, OpenMP, TBB and
atomic instructions. Many workloads support more than one parallelization. Each
parallelization has its own requirements that have to be fulfilled in order to
be able to build and run workloads that use it. By default only support for
pthreads and OpenMP are needed. Gcc supports OpenMP since version 4.2.0.

PARSEC has been successfully tested on the following systems:

	- Linux/i386
	- Linux/x86_64
	- Linux/Itanium
	- Solaris/Sparc

Limited support exists for the following platforms, but not all benchmark
programs might be available:

	- Darwin/PowerPC


3.) Usage

PARSEC ships with several tools which are installed in the 'bin/' directory. You
can use them to customize and manage your installation of the benchmark suite.
'parsecmgmt' is the main tool. Its purpose is to build and run packages as well
as perform other management operations. 'bldconfadd' and 'bldconfdel' can be
used to create and delete your own build configurations.

When you build and run PARSEC with 'parsecmgmt', it will create a log file which
will contain all output in the 'log/' directory. You can get a help summary for
each tool by invoking it with option '-h'.

A full set of man pages documenting PARSEC, its tools and the most important
parts of the source code is given in the `man/' directory. If you add the
`bin/' directory to the PATH environment variable and the `man/' directory
to the MANPATH variable then all tools and man pages are accessible at the
command line without having to specify the full path every time. A bash shell
script `env.sh' is provided in the root directory of the PARSEC distribution
which modifies the environment in that way. If you work with the bash shell you
can make use of it before you start working with PARSEC by executing
`source env.sh'. You can then start browsing the documentation by running
`man parsec'. Support for other shells is currently not available.

The following examples assume that the 'bin/' directory of PARSEC is in your
path.

3.1.) How to Build PARSEC

Before compiling the PARSEC benchmarks, please change some variables in the
file "config/gcc.bldconf" such that the PARSEC command  can locate the 
compiler path correctly.
 
To compile all programs of the benchmark suite with the default configuration,
simply run:

	parsecmgmt -a build

Building the whole benchmark suite takes a lot of time, usually 30-60 min
depending on your system It is possible to selectively build packages
(option '-p') and to choose different build configurations (option '-c'), read
Section 4 for a more comprehensive explanation of build configurations and
Section 6 for more complex usage examples.

3.2.) How to Run PARSEC

After you have built the suite, you can use the following command to run all
benchmarks with the minimal test input:

	parsecmgmt -a run

The test should finish within 5 seconds. Its purpose is to quickly verify that
all benchmarks have been built successfully and are executable. Do not use it
for performance measurements. You can also choose a different input (option
'-i') and alter the number of threads (option '-n'). Section 5 explains the
different inputs and Section 6 gives more detailed examples.

3.3.) How to Get More Information

To query PARSEC about all available packages and features, you can use the
following command:

	parsecmgmt -a info

If you would like to see information about dynamically created files such as
the available builds, you can run:

	parsecmgmt -a status


4.) Build Configurations

Besides the regular configuration which can be altered to customize the
benchmark suite, PARSEC also uses build configurations. A build configuration
is a specific way to compile a program. It determines, for example, what
compiler to use and which features of the package to enable. Build
configurations should be your first approach to alter how a benchmark program
is created. The only limitation is that the interface and output of a benchmark
must remain the same, otherwise the program will become incompatible with the
PARSEC framework. However, you can adapt the run configuration and reference
outputs to eliminate this problem.

PARSEC ships with the following preinstalled build configurations:

	- 'gcc'		  Build parallel version of suite with gcc
	- 'gcc-serial'	  Build serial version of suite with gcc
	- 'gcc-hooks'	  Build parallel version of suite with PARSEC hooks
			  enabled with gcc
	- 'icc'		  Build parallel version of suite with Intel compiler
	- 'gcc-pthreads'  Build with pthreads parallelization (if supported)
	- 'gcc-openmp'    Build with OpenMP parallelization (if supported)
	- 'gcc-tbb'       Build with TBB parallelization (if supported)

For example, to build PARSEC with enabled hooks, you can use:

	parsecmgmt -a build -c gcc-hooks

The three build configurations 'gcc-pthreads', 'gcc-openmp' and 'gcc-tbb' can
be used to compile a workload with one of these three parallelization models if
it is supported by the program. The build configuration 'gcc' implicitly
defines a standard parallelization for each workload by falling back to one of
these three configurations for each benchmark program.

We also defined an alias for each parallelization model that will be resolved
resolved to a complete list of all workloads that support this parallelization
model. They have the same name as the parallelization. For example, to build
all workloads which support OpenMP with exactly that parallelization, you
can use:

	parsecmgmt -a build -p openmp -c gcc-openmp

Additional build configurations can be created with the tool 'bldconfadd'. For
example, to add a new configuration 'myconfig' that is based on gcc-serial,
you can use the following command:

	bldconfadd -n myconfig -s gcc-serial

To remove this configuration, you can use 'bldconfdel' as follows:

	bldconfdel -n myconfig


5.) Performance Measurement & Research

For each benchmark, we define a Region-Of-Interest (ROI) which includes the
computationally intensive, parallelized phase of the benchmark, but not the
initialization or shutdown phase. Instead of measuring the total program
runtime, you can use and report the execution time of the ROI for any kind of
analysis and comparisons. We provide six inputs for each benchmark program:

	- 'test'	Minimal input to verify that programs are executable.
	- 'simdev'	Very small input which causes code execution comparable
			to a typical input for this program. Intended for
			microarchitectural simulator development.
	- 'simsmall'	Small input for performance measurements with
			microarchitectural simulators
	- 'simmedium'	Medium-sized input for performance measurements with
			microarchitectural simulators
	- 'simlarge'	Large-sized input for performance measurements with
			microarchitectural simulators
	- 'native'	Very large input intended for large-scale experiments
			on real machines

All inputs except 'test' and 'simdev' can be used for performance analysis. As
a rough guideline, on a Pentium 4 processor with 3.0 GHz you can expect
approximately the following execution times:

	- 'test'	almost instantaneous
	- 'simdev'	almost instantaneous
	- 'simsmall'	~1s
	- 'simsmall'	~3-5s
	- 'simlarge'	~12-20s
	- 'native'	~10-30min

The exact runtime depends on the program and its inputs and deviates in some
cases from the described rule of thumb. Different build configurations and
system parameters may also result in a deviation.

5.1.) PARSEC Hooks

All benchmark programs of the benchmark suite support PARSEC hooks, a library
which allows rapid instrumentation of all benchmark. If the benchmarks are
compiled with hooks enabled, at various locations in the code the programs will
call the corresponding hook function. Among other things, the ROI has been
instrumented in such a way, and by default the PARSEC hooks library measures the
time spent in the ROI.

Documentation and code of the PARSEC hooks is available in the 'hooks' package
in group 'libs'. PARSEC hooks are enabled in the build configuration 'gcc-hooks'
and disabled in all other ones.


6.) More Examples

6.1.) Compatible to PARSEC 2.1

Build 'x264' and 'blackscholes' benchmarks with Intel compilers:

	parsecmgmt -a build -p x264 blackscholes -c icc

Build serial version of all kernels:

	parsecmgmt -a build -p kernels -c gcc-serial

Run a self-test with all applications:

	parsecmgmt -a run -p apps -i test

Use version of 'vips' and all kernels with enabled hooks to measure the
performance on a real machine with 32 threads:

	parsecmgmt -a run -p vips kernels -c gcc-hooks -n 32

Use SimpleScalar simulator to run a simulation with the 'gcc-hooks'
build configuration of the 'freqmine' application with small simulator inputs
and 4 threads:

	parsecmgmt -a run -c gcc-hooks -s sim-outorder -p freqmine \
		-i simsmall -n 4

Use an executable 'qsub' to submit runs with all kernels with 16 threads and
large simulator inputs to a batch system (this requires that a program 'qsub'
is installed and in the path which handles the job submission):

	parsecmgmt -a run -s qsub -p kernels -i simlarge -n 16

Clean up after a build or benchmarking run:

	parsecmgmt -a fullclean -p all

Uninstall 'gcc-serial' build of package 'gsl' and build a new version:

	parsecmgmt -a uninstall -p gsl -c gcc-serial
	parsecmgmt -a build -p gsl -c gcc-serial


6.2.) Network benchmarks

Check the status of all components involved in network benchmarks:

        parsecmgmt -a status -p netapps

Build network benchmark 'netstreamcluster':

        parsecmgmt -a build -p raytrace

Build all network benchmark:

        parsecmgmt -a build -p netapps

 Run network benchmark 'netdedup' w/ input 'native' and 2 server threads:

        parsecmgmt -a run -p netdedup -i native -n 2

Run network benchmark 'netferret' w/ input 'simlarge', 4 server threads 
and 2 client connections:

        parsecmgmt -a run -p netferret -i simlarge -n 4 -t 2

For simulation, run 'netdedup' server on a simulator w/ 4 threads and 
run client on a real machine:

        parsecmgmt -a run -p netdedup -i simlarge -n 4 -m server
        parsecmgmt -a run -p netdedup -i simlarge -m client

Do a full cleanup for network benchmarks:

        parsecmgmt -a fullclean -p netapps


6.3.) SPLASH-2 Suite and SPLASH-2x Suite

Check the status of SPLASH-2 suite and SPLASH-2x suite:

        parsecmgmt -a status -p splash2
        parsecmgmt -a status -p splash2x

Build benchmark 'raytrace' from SPLASH-2x suite other than PARSEC:
        parsecmgmt -a build -p splash2x.raytrace
        parsecmgmt -a build -p raytrace   ## defaultly from PARSEC (for comparison)

Build benchmark 'fft' from SPLASH-2 suite with 'gcc-serial' build configuration:

        parsecmgmt -a build -c gcc-serial -p splash2.fft

Build all benchmarks from SPLASH-2 suite and SPLASH-2x suite:

        parsecmgmt -a build -p splash2
        parsecmgmt -a build -p splash2x

Run benchmark 'fft' from SPLASH-2x w/ input 'simsmall' and 4 threads:

        parsecmgmt -a run -p splash2x.fft -i simsmall -n 4

Do a full cleanup for SPLASH-2 suite

        parsecmgmt -a fullclean -p splash2


7.) Structure

The PARSEC suite is composed of the software packages, which are the benchmark
programs and their required libraries and tools, and the framework, which is
everything else. Software packages are located in the 'pkgs/' directory. Each
software package is part of exactly one package group, which has its own
subdirectory. For example, a package named 'foo' which belongs to group 'bar' 
would be located in the directory 'pkgs/bar/foo/'.

PARSEC has the following structure:

	bin/		directory with PARSEC tools
	config/		global configuration
	log/		log files of builds and runs (dynamically created)
	man/		man pages of the PARSEC distribution
	pkgs/		package groups which contain the packages
	version		file with version number of PARSEC distribution

A package has the following directory structure:

	inputs/		directory with input archives (optional)
	inst/		installation directory (dynamically created)
	obj/		build directory (dynamically created)
	outputs/	directory with reference outputs (optional)
	parsec/		PARSEC configuration files
	run/		directory for program execution (dynamically created)
	src/		source code of the package
	version		file with package version

Some of these directories will be auto-generated by 'parsecmgmt' on the fly.

Each package can have multiple builds and installations, and 'parsecmgmt' will
use separate subdirectories for them. PARSEC uses the name of the build
configuration and the platform for which the program is compiled to form a key
that is used to distinguish different builds and installations of a package.

7.1.) Configuration Files

PARSEC distinguishes between global and local configuration. Global
configuration files are located in the 'config/' directory in the root of the
benchmark suite. Local configuration files are only valid for a single software
package and are stored in the 'parsec/' directory of the package they belong to.

The following types of configuration files exist: 'parsec.conf' is a global
file which defines the general structure of the benchmark suite, such as
aliases and which software packages exist. Files named '*.sysconf' define
the basic programs which 'parsecmgmt' is to use on each operating system
platform. To port 'parsecmgmt' to a new operating system, a corresponding file
has to be created. '*.runconf' are configuration files which determine how
benchmark programs are to be executed. Finally, files named '*.bldconf'
describe the build configuration of a package. Run and build configurations
are composed of both global and local configuration files.


8.) Manual Usage

It is possible to build and run PARSEC benchmark programs manually. To do so,
the correct build configuration respectively run configuration should be used
as defined in the PARSEC configuration files.

The following steps are executed by 'parsecmgmt' to build a package:

	- Set variable PARSECDIR to the installation root of PARSEC
	- Set variable PARSECPLAT to the build key used to identify the platform
	- Source system configuration
	- Source global build configuration
	- Source local build configuration
	- Create build directory and change to it
	- If it exists, modify build environment and call configure script
	- Modify build environment and execute 'make'
	- Modify build environment and execute 'make install'

The following steps are executed by 'parsecmgmt' to run a benchmark:

        - Set variable PARSECDIR to the installation root of PARSEC
        - Set variable PARSECPLAT to the build key used to identify the platform
	- Source system configuration
	- Source global run configuration
        - Source local run configuration
	- Create run directory and cd to it
	- Unpack desired input
	- Execute benchmark with parameters from run configuration


9.) Contact

You can reach the PARSEC team as follows:

	http://parsec.cs.princeton.edu/
	parsec@lists.cs.princeton.edu