This document includes instructions on how to create an Ubuntu 18.04 disk-image with ROCm 4.2 installed. The disk-image will be compatible with the gem5 simulator. It also demonstrates how to simulate the same using an example gem5 script with a pre-configured system.
## Building the disk image In order to build the disk-image for ROCm 4.2 with gem5, build the m5 utility in `src/gpu-fs/` using the following: ```sh git clone https://gem5.googlesource.com/public/gem5 cd gem5/util/m5 scons build/x86/out/m5 cp build/x86/out/m5 /path/to/gem5-resources/src/gpu-fs/
We use packer to create our disk-image. The instructions on how to install packer is shown below:
cd disk-image ./build.sh # the script downloading packer binary and building the disk image
You can find the disk-image in disk-image/rocm42/rocm42-image/rocm42.
An example script with a pre-configured system is available in the following directory within the gem5 repository:
gem5/configs/example/gpufs/vega10_kvm.py
The example script specifies a system with the following parameters:
MOESI_AMD_Base protocol. The CPU and CPU cache configurations are largely irrelevant for GPU simulation.GPU_VIPER cache with 32 kB L1I (SQC), 16 kB per-CU L1D (TCP), and 256 kB L2 (TCC).The example script must be run with the VEGA_X86 binary. To build:
git clone https://gem5.googlesource.com/public/gem5 cd gem5 scons build/VEGA_X86/gem5.opt -j<proc>
Once compiled, you may use one of the example config scripts to run a GPU application on the simulated machine:
gem5/configs/example/gpufs/hip_samples.py gem5/configs/example/gpufs/hip_cookbook.py gem5/configs/example/gpufs/hip_rodinia.py
These scripts can be run as follows pointing to the disk image created above and the provided kernel and GPU trace in gem5-resources. For example:
build/VEGA_X86/gem5.opt configs/example/gpufs/hip_samples.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app PrefixSum build/VEGA_X86/gem5.opt configs/example/gpufs/hip_cookbook.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app 4_shfl build/VEGA_X86/gem5.opt configs/example/gpufs/hip_rodinia.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app nn
You can obtain the vmlinux-5.4.0-105-generic kernel using the following path from gem5-resources: wget --no-check-certificate https://dist.gem5.org/dist/v22-1/kernels/x86/static/vmlinux-5.4.0-105-generic
It is sometimes useful to build your own application and run in gem5. A docker is provided to allow users to build applications without needing to install ROCm locally. A pre-built docker image is available on gcr.io. This image can be pulled then used to build as follows:
docker pull gcr.io/gem5-test/gpu-fs:latest cd /path/to/gem5-resources/src/gpu/square docker run --rm -v ${PWD}:${PWD} -w ${PWD} gcr.io/gem5-test/gpu-fs:latest bash -c 'make clean; HCC_AMDGPU_TARGET=gfx900 make'
Currently only Vega (gfx900) is supported for full system GPU simulation in gem5. It is therefore required to tell the compiler to build for this ISA using the HCC_AMDGPU_TARGET environment variable. Otherwise, the command to build the application is the same as if you were building locally.
The build docker can also be built from the gem5 directory:
cd gem5/util/dockerfiles/gpu-fs/ docker build -t rocm42-build . cd /path/to/gem5-resources/src/gpu/square docker run --rm -v ${PWD}:${PWD} -w ${PWD} rocm42-build bash -c 'make clean; HCC_AMDGPU_TARGET=gfx900 make'
The application can then be run using the vega10_kvm.py example script. There are two arguments available in the example script:
Below is an example using the square application which was built in above using the docker image:
build/VEGA_X86/gem5.opt configs/example/gpufs/vega10_kvm.py --disk-image /path/to/gem5-resources/src/gpu-fs/disk-image/rocm42/rocm42-image/rocm42 --kernel /path/to/gem5-resources/src/gpu-fs/vmlinux-5.4.0-105-generic --gpu-mmio-trace /path/to/gem5-resources/src/gpu-fs/vega_mmio.log --app /path/to/gem5-resources/src/gpu/square/bin/square
The known working ROCm 4.2 applications for gem5-22 is below for each of the example config scripts and other gem5-resources. Missing applications either do not work or have not been fully tested:
The following features are known not to work:
perf_event_paranoid error when running a FS simulation:This error may be caused by a too restrictive setting in the file '/proc/sys/kernel/perf_event_paranoid' The default value was changed to 2 in kernel 4.6 A value greater than 1 prevents gem5 from making the syscall to perf_event_open. You need to run something like the following (as root). # echo -1 > /proc/sys/kernel/perf_event_paranoid
Qemu stderr: qemu-system-x86_64: failed to initialize KVM: Permission denied, the issue is likely related to permission on /dev/kvm