src/gpu/halo-finder/src/README - public/gem5-resources - Git at Google


 Parallel Halo Finder Operation
 ------------------------------

 The test driver for the parallel halo finder is HaloTestP.cxx.  The basic
 operation of the parallel halo finder is to take input data of particle
 location, velocity, mass and tag identifier and distribute the particles on
 processors such that the particles are completely partitioned.  These are
 called the alive particles belonging to a processor.  Each processor will
 share information so that particles which are alive on a neighbor processor
 are added to a processor becoming dead particles there.  By definition the
 belt of dead particles will be all particles which can comprise a halo.

 Once each processor has both alive and dead particles, the serial halo finder
 is called and will return all halos found on that processor.  A halo will
 contain only alive particles, in which case it is owned by the processor,
 or only dead particles and it will be discarded because it will belong to
 one of the neighbor processors.  A halo might contain a combination of
 alive and dead particles and this mixed halo will be shared by one or
 more neighbor processors.

 The parallel halo finder must merge these mixed halos before giving the
 results.  When a particle is assigned to a processor it has a status which
 can be ALIVE, or DEAD.  The dead particles are actually given a status of
 0 through 26 to indicate which neighbor processor contains that particle
 as alive.  When the serial halo finder completes and halos are looked at,
 a mixed halo will also know what neighbors share it.  A simple rule using
 the planes of the neighbor allows most mixed halos to be categorized
 immediately.  If the neigbor sharing the halo is in the "upper" planes
 of the physical space (right, top and back planes of x,y,z) then this
 processor will keep the halo as its own.  Conversely if the neighbor sharing
 the halo is in the "lower" planes (left, bottom, front) then it will delete
 that halo, knowing the the neighbor will claim it.  Any other halos which
 cross planes or are contain in more that one other neighbor are gathered
 up and sent to the master processor who arbitrates which processor will
 claim the halo and which will give it up.


 Input Files
 -----------

 The halo finder takes two styles of input.  The first is the .cosmo format
 which is called RECORD format.  Each particle has seven 4 byte floats
 (x location, x velocity, y location, y velocity, z location, z velocity,
 mass (which is 1.0)) and one 4 byte integer which is the unique particle
 identifier which starts with 1.

 The second format is the Gadget style which is called BLOCK.  In this file
 there is a header of 256 bytes and which is described in Definition.h.
 It is followed by three blocks of data.  The first has 4 byte floats for
 each particle's x,y,z location.  Note that particle 0 is written but since
 particles start with identifier 1, these first three floats are ignored.
 Next is a block with x,y,z velocity and again the first three floats
 must be ignored.  Finally a block of 4 byte integer tag identifiers with
 the first tag ignored.


 Methods of Input
 ----------------

 Data files can either contain only the alive particles for a processor or
 a mixture of particles such that each processor must read each file to
 obtain its alive particles.  In the first case the processor only has
 to read one file, and then can do an exchange of dead particles with
 immediate neighbor processors which is done in rotation.  All processors
 send right face neighbor particles to the right, and receive from the left,
 and then the reverse is done.  With 26 of these rotations all neighbors will
 get dead particles and every processor has something to do on each step
 of the rotation so that the MPI Send/Receive works.

 In the second case since every file has to be read and then MPI messages
 containing the particles are created and passed round robin so that
 every processor has a chance to choose its own alive particles.  After
 this the same particle exchange is done to populate the dead particles.


 Output Files
 ------------

 The .cosmo files are written again as output, but with the mass field
 replaced by the mass * the number of particles in the halo.  Only the
 first particle in a halo contains that accumulated mass.  Every other
 particle in the halo reports the tag identifier of that first particle
 so that you can tell what particles make up a halo.  If the mass is -1
 that means the particle is not in any halo.  An additional file per
 processor is written with the identifier and size of every halo on that
 processor and is used by the BinHalos.cxx program to create a histogram.


 C++ Classes
 -----------

 Five classes are available for use.

 "Partition" is a static class which does the MPI initialize and finalize and
 which creates the Cartesian topology of the problem and identifies this
 processor within the topology and all of its neighbors in the topology.

 "ParticleDistribute" does the reading of files for either ROUND_ROBIN or
 ONE_TO_ONE and in both cases the final result has only alive particles.

 The round robin read is complicated because of the MPI message passing.
 Since there can be fewer files than processors, if they can be divided
 nicely as in 8 files across 32 processors, then four processors read the
 8 files and distribute in four ROUND_ROBIN loops.  If there are fewer
 processors than files, then many loops of read will happen although the
 final loop may have some processors participating in the exchange which
 did not initially read any data and will just be passing buffers of 0 particles.

 The one to one read assumes that files have only alive particles and have
 been laid out in the same topology that MPI Cartesian topology will use.
 This is a preamble to simply passing alive particles in memory.

 "ParticleExchange" does the exchange with nearest neighbors using either
 the vectors from ParticleDistribute or from some other initializer which
 contain only alive particles.  When it exits the vectors have both alive
 and dead particles in it.

 "CosmoHaloFinderP" takes the information from either ParticleDistribute or
 ParticleExchange, normalizes the locations, and calls the serial
 halo finder.  It collects the results, merges the mixed halos and writes
 halo output.

 "CosmoHaloFinder" is the serial halo finder and can operate on its own using
 HaloTest.cxx as a driver.


 Command Line Arguments
 ----------------------

 An input file format was devised and is read by the class HaloFinderInput.
 Examples for BLOCK and COSMO formats are in sb256_gadget2.in and
 sb256_cosmo.in.

 ################################################################################
 # Header version information
 ################################################################################
 HALOFINDER_HEADER_VERSION 1.0.0

 ################################################################################
 # Input base name ending in '.' if followed by processor id
 ################################################################################
 INPUT_BASE_NAME /Users/pkf/Cosmo.files/gadget_sb256.cosmo.

 ################################################################################
 # Input data style (RECORD = .cosmo)  (BLOCK = .gadget2)
 ################################################################################
 INPUT_TYPE RECORD

 ################################################################################
 # Particle distribution style
 #  ROUND_ROBIN indicates particles must be looked at by all processors
 #  ONE_TO_ONE  indicates that particles physically reside on matchin processor
 ################################################################################
 DISTRIBUTE_TYPE ROUND_ROBIN

 ################################################################################
 # Output base name
 ################################################################################
 OUTPUT_BASE_NAME sb256_c

 ################################################################################
 # Box size (rL)
 ################################################################################
 BOX_SIZE 64.0

 ################################################################################
 # Overload zone size (dead zone)
 ################################################################################
 OVERLOAD_SIZE 5.0

 ################################################################################
 # Number of particles in all files (np^3)
 ################################################################################
 NUMBER_OF_PARTICLES 256

 ################################################################################
 # Minimum distance between particles in a halo (bb)
 ################################################################################
 MINIMUM_PARTICLE_DISTANCE 0.20

 ################################################################################
 # Minimum number of particles in a halo (pmin)
 ################################################################################
 MINIMUM_PARTICLES_PER_HALO 10

 ################################################################################
 # Omega dm
 ################################################################################
 OMEGADM 1.0

 ################################################################################
 # Hubble constant
 ################################################################################
 HUBBLE_CONSTANT 0.5

 ################################################################################
 # Deut
 ################################################################################
 DEUT 0.0

 ################################################################################
 # Output all particle data with mass field replaced by halo tag
 ################################################################################
 OUTPUT_PARTICLES 0

 ################################################################################
 # Output the halo catalog of one entry per halo (.cosmo and ascii format)
 ################################################################################
 OUTPUT_HALO_CATALOG 1

 ################################################################################
 # Output FOF halo properties report (ascii)
 ################################################################################
 OUTPUT_FOF_PROPERTIES 1


 Sample Command Lines
 --------------------

 mpirun -np 8 HaloFinder sb256_cosmo.in
 mpirun -np 8 HaloFinder sb256_gadget2.in


 Compile, Load and Run
 ---------------------

 Modules to load must include a c++ compile and MPI.
 % make HaloFinder will compile.

	Parallel Halo Finder Operation
	------------------------------

	The test driver for the parallel halo finder is HaloTestP.cxx. The basic
	operation of the parallel halo finder is to take input data of particle
	location, velocity, mass and tag identifier and distribute the particles on
	processors such that the particles are completely partitioned. These are
	called the alive particles belonging to a processor. Each processor will
	share information so that particles which are alive on a neighbor processor
	are added to a processor becoming dead particles there. By definition the
	belt of dead particles will be all particles which can comprise a halo.

	Once each processor has both alive and dead particles, the serial halo finder
	is called and will return all halos found on that processor. A halo will
	contain only alive particles, in which case it is owned by the processor,
	or only dead particles and it will be discarded because it will belong to
	one of the neighbor processors. A halo might contain a combination of
	alive and dead particles and this mixed halo will be shared by one or
	more neighbor processors.

	The parallel halo finder must merge these mixed halos before giving the
	results. When a particle is assigned to a processor it has a status which
	can be ALIVE, or DEAD. The dead particles are actually given a status of
	0 through 26 to indicate which neighbor processor contains that particle
	as alive. When the serial halo finder completes and halos are looked at,
	a mixed halo will also know what neighbors share it. A simple rule using
	the planes of the neighbor allows most mixed halos to be categorized
	immediately. If the neigbor sharing the halo is in the "upper" planes
	of the physical space (right, top and back planes of x,y,z) then this
	processor will keep the halo as its own. Conversely if the neighbor sharing
	the halo is in the "lower" planes (left, bottom, front) then it will delete
	that halo, knowing the the neighbor will claim it. Any other halos which
	cross planes or are contain in more that one other neighbor are gathered
	up and sent to the master processor who arbitrates which processor will
	claim the halo and which will give it up.


	Input Files
	-----------

	The halo finder takes two styles of input. The first is the .cosmo format
	which is called RECORD format. Each particle has seven 4 byte floats
	(x location, x velocity, y location, y velocity, z location, z velocity,
	mass (which is 1.0)) and one 4 byte integer which is the unique particle
	identifier which starts with 1.

	The second format is the Gadget style which is called BLOCK. In this file
	there is a header of 256 bytes and which is described in Definition.h.
	It is followed by three blocks of data. The first has 4 byte floats for
	each particle's x,y,z location. Note that particle 0 is written but since
	particles start with identifier 1, these first three floats are ignored.
	Next is a block with x,y,z velocity and again the first three floats
	must be ignored. Finally a block of 4 byte integer tag identifiers with
	the first tag ignored.


	Methods of Input
	----------------

	Data files can either contain only the alive particles for a processor or
	a mixture of particles such that each processor must read each file to
	obtain its alive particles. In the first case the processor only has
	to read one file, and then can do an exchange of dead particles with
	immediate neighbor processors which is done in rotation. All processors
	send right face neighbor particles to the right, and receive from the left,
	and then the reverse is done. With 26 of these rotations all neighbors will
	get dead particles and every processor has something to do on each step
	of the rotation so that the MPI Send/Receive works.

	In the second case since every file has to be read and then MPI messages
	containing the particles are created and passed round robin so that
	every processor has a chance to choose its own alive particles. After
	this the same particle exchange is done to populate the dead particles.


	Output Files
	------------

	The .cosmo files are written again as output, but with the mass field
	replaced by the mass * the number of particles in the halo. Only the
	first particle in a halo contains that accumulated mass. Every other
	particle in the halo reports the tag identifier of that first particle
	so that you can tell what particles make up a halo. If the mass is -1
	that means the particle is not in any halo. An additional file per
	processor is written with the identifier and size of every halo on that
	processor and is used by the BinHalos.cxx program to create a histogram.


	C++ Classes
	-----------

	Five classes are available for use.

	"Partition" is a static class which does the MPI initialize and finalize and
	which creates the Cartesian topology of the problem and identifies this
	processor within the topology and all of its neighbors in the topology.

	"ParticleDistribute" does the reading of files for either ROUND_ROBIN or
	ONE_TO_ONE and in both cases the final result has only alive particles.

	The round robin read is complicated because of the MPI message passing.
	Since there can be fewer files than processors, if they can be divided
	nicely as in 8 files across 32 processors, then four processors read the
	8 files and distribute in four ROUND_ROBIN loops. If there are fewer
	processors than files, then many loops of read will happen although the
	final loop may have some processors participating in the exchange which
	did not initially read any data and will just be passing buffers of 0 particles.

	The one to one read assumes that files have only alive particles and have
	been laid out in the same topology that MPI Cartesian topology will use.
	This is a preamble to simply passing alive particles in memory.

	"ParticleExchange" does the exchange with nearest neighbors using either
	the vectors from ParticleDistribute or from some other initializer which
	contain only alive particles. When it exits the vectors have both alive
	and dead particles in it.

	"CosmoHaloFinderP" takes the information from either ParticleDistribute or
	ParticleExchange, normalizes the locations, and calls the serial
	halo finder. It collects the results, merges the mixed halos and writes
	halo output.

	"CosmoHaloFinder" is the serial halo finder and can operate on its own using
	HaloTest.cxx as a driver.


	Command Line Arguments
	----------------------

	An input file format was devised and is read by the class HaloFinderInput.
	Examples for BLOCK and COSMO formats are in sb256_gadget2.in and
	sb256_cosmo.in.

	################################################################################
	# Header version information
	################################################################################
	HALOFINDER_HEADER_VERSION 1.0.0

	################################################################################
	# Input base name ending in '.' if followed by processor id
	################################################################################
	INPUT_BASE_NAME /Users/pkf/Cosmo.files/gadget_sb256.cosmo.

	################################################################################
	# Input data style (RECORD = .cosmo) (BLOCK = .gadget2)
	################################################################################
	INPUT_TYPE RECORD

	################################################################################
	# Particle distribution style
	# ROUND_ROBIN indicates particles must be looked at by all processors
	# ONE_TO_ONE indicates that particles physically reside on matchin processor
	################################################################################
	DISTRIBUTE_TYPE ROUND_ROBIN

	################################################################################
	# Output base name
	################################################################################
	OUTPUT_BASE_NAME sb256_c

	################################################################################
	# Box size (rL)
	################################################################################
	BOX_SIZE 64.0

	################################################################################
	# Overload zone size (dead zone)
	################################################################################
	OVERLOAD_SIZE 5.0

	################################################################################
	# Number of particles in all files (np^3)
	################################################################################
	NUMBER_OF_PARTICLES 256

	################################################################################
	# Minimum distance between particles in a halo (bb)
	################################################################################
	MINIMUM_PARTICLE_DISTANCE 0.20

	################################################################################
	# Minimum number of particles in a halo (pmin)
	################################################################################
	MINIMUM_PARTICLES_PER_HALO 10

	################################################################################
	# Omega dm
	################################################################################
	OMEGADM 1.0

	################################################################################
	# Hubble constant
	################################################################################
	HUBBLE_CONSTANT 0.5

	################################################################################
	# Deut
	################################################################################
	DEUT 0.0

	################################################################################
	# Output all particle data with mass field replaced by halo tag
	################################################################################
	OUTPUT_PARTICLES 0

	################################################################################
	# Output the halo catalog of one entry per halo (.cosmo and ascii format)
	################################################################################
	OUTPUT_HALO_CATALOG 1

	################################################################################
	# Output FOF halo properties report (ascii)
	################################################################################
	OUTPUT_FOF_PROPERTIES 1


	Sample Command Lines
	--------------------

	mpirun -np 8 HaloFinder sb256_cosmo.in
	mpirun -np 8 HaloFinder sb256_gadget2.in


	Compile, Load and Run
	---------------------

	Modules to load must include a c++ compile and MPI.
	% make HaloFinder will compile.