title: ROCm 4.2 tags: - x86 - fullsystem layout: default permalink: resources/rocm42 shortdoc: > Resources to build a disk image with AMD ROCm. author: [“Matthew Poremba”] license: BSD-3-Clause

This document includes instructions on how to create an Ubuntu 18.04 disk-image with ROCm 4.2 installed. The disk-image will be compatible with the gem5 simulator. It also demonstrates how to simulate the same using an example gem5 script with a pre-configured system.

## Building the disk image

In order to build the disk-image for ROCm 4.2 with gem5, build the m5 utility in `src/gpu-fs/` using the following:

```sh
git clone https://gem5.googlesource.com/public/gem5
cd gem5/util/m5
scons build/x86/out/m5
cp build/x86/out/m5 /path/to/gem5-resources/src/gpu-fs/

We use packer to create our disk-image. The instructions on how to install packer is shown below:

cd disk-image
./build.sh          # the script downloading packer binary and building the disk image

You can find the disk-image in disk-image/rocm42/rocm42-image/rocm42.

Simulating GPU full system using an example script

An example script with a pre-configured system is available in the following directory within the gem5 repository:

gem5/configs/example/gpufs/vega10_kvm.py

The example script specifies a system with the following parameters:

• A single ‘KVM’ CPU with the MOESI_AMD_Base protocol. The CPU and CPU cache configurations are largely irrelevant for GPU simulation.
• 2 Level GPU_VIPER cache with 32 kB L1I (SQC), 16 kB per-CU L1D (TCP), and 256 kB L2 (TCC).
• The system has 3 GB of --mem-type memory for CPU and 16 GB of --mem-type memory for GPU.

The example script must be run with the VEGA_X86 binary. To build:

git clone https://gem5.googlesource.com/public/gem5
cd gem5
scons build/VEGA_X86/gem5.opt -j<proc>

Once compiled, you may use one of the example config scripts to run a GPU application on the simulated machine:

gem5/configs/example/gpufs/hip_samples.py
gem5/configs/example/gpufs/hip_cookbook.py
gem5/configs/example/gpufs/hip_rodinia.py

These scripts can be run as follows pointing to the disk image created above and the provided kernel and GPU trace in gem5-resources. For example:

build/VEGA_X86/gem5.opt configs/example/gpufs/hip_samples.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app PrefixSum
build/VEGA_X86/gem5.opt configs/example/gpufs/hip_cookbook.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app 4_shfl
build/VEGA_X86/gem5.opt configs/example/gpufs/hip_rodinia.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app nn

You can obtain the vmlinux-5.4.0-105-generic kernel using the following path from gem5-resources: wget --no-check-certificate https://dist.gem5.org/dist/v22-1/kernels/x86/static/vmlinux-5.4.0-105-generic

It is sometimes useful to build your own application and run in gem5. A docker is provided to allow users to build applications without needing to install ROCm locally. A pre-built docker image is available on gcr.io. This image can be pulled then used to build as follows:

docker pull gcr.io/gem5-test/gpu-fs:latest
cd /path/to/gem5-resources/src/gpu/square
docker run --rm -v ${PWD}:${PWD} -w ${PWD} gcr.io/gem5-test/gpu-fs:latest bash -c 'make clean; HCC_AMDGPU_TARGET=gfx900 make'

Currently only Vega (gfx900) is supported for full system GPU simulation in gem5. It is therefore required to tell the compiler to build for this ISA using the HCC_AMDGPU_TARGET environment variable. Otherwise, the command to build the application is the same as if you were building locally.

The build docker can also be built from the gem5 directory:

cd gem5/util/dockerfiles/gpu-fs/
docker build -t rocm42-build .
cd /path/to/gem5-resources/src/gpu/square
docker run --rm -v ${PWD}:${PWD} -w ${PWD} rocm42-build bash -c 'make clean; HCC_AMDGPU_TARGET=gfx900 make'

The application can then be run using the vega10_kvm.py example script. There are two arguments available in the example script:

• --app, which copies the pre-built application from the host into the simulated gem5 environment and runs the command with the options given by --opts.
• --opts, which passes options to the application being run

Below is an example using the square application which was built in above using the docker image:

build/VEGA_X86/gem5.opt configs/example/gpufs/vega10_kvm.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app /path/to/gem5-resources/src/gpu/square/bin/square

Working Status

The known working ROCm 4.2 applications for gem5-22 is below for each of the example config scripts and other gem5-resources. Missing applications either do not work or have not been fully tested:

• hip_samples.py: BinomialOption, BitonicSort, FastWalshTransform, FloydWarshall, Histogram, PrefixSum, RecursiveGaussian, SimpleConvolution, dct, dwtHaar1D
• hip_cookbook.py: 0_MatrixTranspose, 3_shared_memory, 4_shfl, 5_2dshfl, 6_dynamic_shared, 7_streams, 9_unroll, 10_inline_asm, 11_texture_driver, 13_occupancy, 14_gpu_arch, 15_static_library
• hip_rodinia.py: bfs, nn
• gem5-resources: heterosync (lfTreeBarrUniq 10 16 4), pagerank, fw

The following features are known not to work:

• Dynamic scratch space allocation (gem5 will fatal)
• HIP events (simulation will hang/never finish).

Troubleshooting

• perf_event_paranoid error when running a FS simulation:

This error may be caused by a too restrictive setting   in the file
'/proc/sys/kernel/perf_event_paranoid'  The default value was changed to 2
in kernel 4.6  A value greater than 1 prevents gem5 from making  the
syscall to perf_event_open.

You need to run something like the following (as root).

# echo -1 > /proc/sys/kernel/perf_event_paranoid

• If you encounter Qemu stderr: qemu-system-x86_64: failed to initialize KVM: Permission denied, the issue is likely related to permission on /dev/kvm