website: Add ARM power and DVFS documentation, update FS ARM.
This is a squash of 4 commits. This commit:
- Adds a section to Learning gem5 part1_6 on how to set up and run an
ARM FS simulation.
- Adds documentation and example code for adding power modelling to an
ARM FS simulation, describing the different components required.
This is placed in Learning gem5 part2_7.
- Adds documentation and example code for adding DVFS support to an ARM
FS simulation, describing and explaining Clock Domains and Voltage
Domains in the process.
This is placed in Learning gem5 part2_8.
- Updates some broken or deprecated links and references in the
documentation on building a Linux kernel for FS ARM simulations.
The new files relate to:
Jira: https://gem5.atlassian.net/browse/GEM5-801
Jira: https://gem5.atlassian.net/browse/GEM5-783
(and somewhat also related:
Jira: https://gem5.atlassian.net/browse/GEM5-463
Jira: https://gem5.atlassian.net/browse/GEM5-769
)
Change-Id: Ia0375c94b2ae6441cfd3e6f114625934f1df5b37
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5-website/+/36555
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: Bobby R. Bruce <bbruce@ucdavis.edu>
diff --git a/_data/documentation.yml b/_data/documentation.yml
index 53e0147..b999970 100755
--- a/_data/documentation.yml
+++ b/_data/documentation.yml
@@ -229,6 +229,10 @@
url: /documentation/learning_gem5/part2/memoryobject
- page: Creating a simple cache object
url: /documentation/learning_gem5/part2/simplecache
+ - page: ARM Power Modelling
+ url: /documentation/learning_gem5/part2/arm_power_modelling
+ - page: ARM DVFS Support
+ url: /documentation/learning_gem5/part2/arm_dvfs_support
- title: Modeling Cache Coherence with Ruby
id: part3
diff --git a/_pages/documentation/general_docs/fullsystem/building_arm_kernel.md b/_pages/documentation/general_docs/fullsystem/building_arm_kernel.md
index 3c3dd70..72a1c96 100644
--- a/_pages/documentation/general_docs/fullsystem/building_arm_kernel.md
+++ b/_pages/documentation/general_docs/fullsystem/building_arm_kernel.md
@@ -10,7 +10,8 @@
This page contains instructions for building up-to-date kernels for gem5 running on ARM.
-If you don't want to build the Kernel on your own you could still [download a prebuilt version](./guest_binaries/)
+If you don't want to build the Kernel on your own you could still [download a
+prebuilt version](./guest_binaries).
## Prerequisites
These instructions are for running headless systems. That is a more "server" style system where there is no frame-buffer. The description has been created using the latest known-working tag in the repositories linked below, however the tables in each section list previous tags that are known to work. To built the kernels on an x86 host you'll need ARM cross compilers and the device tree compiler. If you're running a reasonably new version of Ubuntu or Debian you can get required software through apt:
@@ -19,11 +20,18 @@
apt-get install gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu device-tree-compiler
```
-If you can't use these pre-made compilers the next easiest way to obtain the required compilers from [Linaro](http://releases.linaro.org/latest/components/toolchain/binaries/).
+If you can't use these pre-made compilers the next easiest way to obtain the
+required compilers from ARM:
+- [Cortex A cross-compilers](https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads)
+- [Cortex RM cross-compilers](https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads)
+
+Download (one of) these and make sure the binaries are on your `PATH`.
Depending on the exact source of your cross compilers, the compiler names used below will required small changes.
-To actually run the kernel, you'll need to download or compile gem5's bootloader. See the (bootloaders)(#bootloaders) section in this documents for details.
+To actually run the kernel, you'll need to download or compile gem5's
+bootloader. See the [bootloaders](#bootloaders) section in this documents for
+details.
## Linux 4.x
Newer gem5 kernels for ARM (v4.x and later) are based on the vanilla Linux kernel and typically have a small number of patches to make them work better with gem5. The patches are optional and you should be able to use a vanilla kernel as well. However, this requires you to configure the kernel yourself. Newer kernels all use the VExpress\_GEM5\_V1 gem5 platform for both AArch32 and AArch64. The required DTB files to describe the hardware to the OS ship with gem5. To build them, execute this command:
@@ -143,4 +151,5 @@
make -C system/arm/bootloader/arm64
```
-Once you have compiled the binaries, put them in the binaries directory in your M5\_PATH.
+Once you have compiled the binaries, put them in the binaries directory in your
+`M5_PATH`.
diff --git a/_pages/documentation/learning_gem5/part1/part1_6_extending_configs.md b/_pages/documentation/learning_gem5/part1/part1_6_extending_configs.md
index 73223a2..ae4117a 100644
--- a/_pages/documentation/learning_gem5/part1/part1_6_extending_configs.md
+++ b/_pages/documentation/learning_gem5/part1/part1_6_extending_configs.md
@@ -1,17 +1,17 @@
---
layout: documentation
-title: Extending gem5 to run ARM binaries
+title: Extending gem5 to run ARM binaries
doc: Learning gem5
parent: part1
permalink: /documentation/learning_gem5/part1/extending_configs
-author: Julian T. Angeles
+author: Julian T. Angeles, Thomas E. Hansen
---
Extending gem5 for ARM
======================
This chapter assumes you've already built a basic x86 system with
-gem5 and created a simple configuration script.
+gem5 and created a simple configuration script.
Downloading ARM Binaries
------------------------
@@ -108,3 +108,81 @@
-50000
Exiting @ tick 258647411000 because exiting with last active thread context
```
+
+ARM Full System Simulation
+--------------------------
+To run an ARM FS Simulation, there are some changes required to the setup.
+
+If you haven't already, from the gem5 repository's root directory, `cd` into
+the directory `util/term/` by running
+
+```bash
+$ cd util/term/
+```
+
+and then compile the `m5term` binary by running
+
+```bash
+$ make
+```
+
+The gem5 repository comes with example system setups and configurations. These
+can be found in the `configs/example/arm/` directory.
+
+A collection of full system Linux image files are available
+[here](https://www.gem5.org/documentation/general_docs/fullsystem/guest_binaries).
+Save these in a directory and remember the path to it. For example, you could
+store them in
+
+```
+/path/to/user/gem5/fs_images/
+```
+
+The `fs_images` directory will be assumed to contain the extracted FS images
+for the rest of this example.
+
+With the image(s) downloaded, execute the following command in your terminal:
+
+```bash
+$ export IMG_ROOT=/absolute/path/to/fs_images/<image-directory-name>
+```
+
+replacing "\<image-directory-name\>" with the name of the directory extracted
+from the downloaded image file, without the angle-brackets.
+
+We are now ready to run a FS ARM simulation. From the root of the gem5
+repository, run:
+
+```bash
+$ ./build/ARM/gem5.opt configs/example/arm/fs_bigLITTLE.py \
+ --caches \
+ --bootloader="$IMG_ROOT/binaries/<bootloader-name>" \
+ --kernel="$IMG_ROOT/binaries/<kernel-name>" \
+ --disk="$IMG_ROOT/disks/<disk-image-name>" \
+ --bootscript=path/to/bootscript.rcS
+```
+
+replacing anything in angle-brackets with the name of the directory or file,
+without the angle-brackets.
+
+You can then attach to the simulation by, in a different terminal window,
+running:
+
+```bash
+$ ./util/term/m5term 3456
+```
+
+The full details of what the `fs_bigLITTLE.py` script supports can be gotten by
+running:
+
+```bash
+$ ./build/ARM/gem5.opt configs/example/arm/fs_bigLITTLE.py --help
+```
+
+> **An aside on FS simulations:**
+>
+> Note that FS simulations take a long time; like "1 hour to load the kernel"
+> long time! There are ways to "fast-forward" a simulation and then resume the
+> detailed simulation at the interesting point, but these are beyond the scope
+> of this chapter.
+
diff --git a/_pages/documentation/learning_gem5/part2/part2_7_arm_power_modelling.md b/_pages/documentation/learning_gem5/part2/part2_7_arm_power_modelling.md
new file mode 100644
index 0000000..58e31de
--- /dev/null
+++ b/_pages/documentation/learning_gem5/part2/part2_7_arm_power_modelling.md
@@ -0,0 +1,296 @@
+---
+layout: documentation
+title: ARM Power Modelling
+doc: Learning gem5
+parent: part2
+permalink: /documentation/learning_gem5/part2/arm_power_modelling/
+author: Thomas E. Hansen
+---
+
+
+ARM Power Modelling
+===================
+
+It is possible to model and monitor the energy and power usage of a gem5
+simulation. This is done by using various stats already recorded by gem5 in a
+`MathExprPowerModel`; a way to model power usage through mathematical
+equations. This chapter of the tutorial details what the various components
+required for power modelling are and explains how to add them to an existing
+ARM simulation.
+
+This chapter draws on the `fs_power.py` configuration script, provided in the
+`configs/example/arm` directory, and also provides instructions for how to
+extend this script or other scripts.
+
+Note that power models can only be applied when using the more detailed
+"timing" CPUs.
+
+An overview of how power modelling is built into gem5 and which other parts of
+the simulator they interact with can be found in [Sascha Bischoff's
+presentation](https://youtu.be/3gWyUWHxVj4) from the 2017 ARM Research Summit.
+
+Dynamic Power States
+--------------------
+
+Power Models consist of two functions which describe how to calculate the power
+consumption in different power states. The power states are (from
+`src/sim/PowerState.py`):
+
+- `UNDEFINED`: Invalid state, no power state derived information is available.
+ This state is the default.
+- `ON`: The logic block is actively running and consuming dynamic and leakage
+ energy depending on the amount of processing required.
+- `CLK_GATED`: The clock circuity within the block is gated to save dynamic
+ energy, the power supply to the block is still on and leakage energy is
+ being consumed by the block.
+- `SRAM_RETENTION`: The SRAMs within the logic blocks are pulled into retention
+ state to reduce leakage energy further.
+- `OFF`: The logic block is power gated and is not consuming any energy.
+
+A Power Model is assigned to each of the states, apart from `UNDEFINED`, using
+the `PowerModel` class's `pm` field. It is a list containing 4 Power Models,
+one for each state, in the following order:
+
+0. `ON`
+1. `CLK_GATED`
+2. `SRAM_RETENTION`
+3. `OFF`
+
+Note that although there are 4 different entries, these do not have to be
+different Power Models. The provided `fs_power.py` file uses one Power Model
+for the `ON` state and then the same Power Model for the remaining states.
+
+Power Usage Types
+-----------------
+
+The gem5 simulator models 2 types of power usage:
+
+- **static**: The power used by the simulated system regardless of activity.
+- **dynamic**: The power used by the system due to various types of activity.
+
+A Power Model must contain an equation for modelling both of these (although
+that equation can be as simple as `st = "0"` if, for example, static power is
+not desired or irrelevant in that Power Model).
+
+MathExprPowerModels
+-------------------
+
+The provided Power Models in `fs_power.py` extend the `MathExprPowerModel`
+class. `MathExprPowerModels` are specified as strings containing mathematical
+expressions for how to calculate the power used by the system. They typically
+contain a mix of stats and automatic variables, e.g. temperature, for example:
+
+```python
+class CpuPowerOn(MathExprPowerModel):
+ def __init__(self, cpu_path, **kwargs):
+ super(CpuPowerOn, self).__init__(**kwargs)
+ # 2A per IPC, 3pA per cache miss
+ # and then convert to Watt
+ self.dyn = "voltage * (2 * {}.ipc + 3 * 0.000000001 * " \
+ "{}.dcache.overall_misses / sim_seconds)".format(cpu_path,
+ cpu_path)
+ self.st = "4 * temp"
+```
+
+(The above power model is taken from the provided `fs_power.py` file.)
+
+We can see that the automatic variables (`voltage` and `temp`) do not require
+a path, whereas component-specific stats (the CPU's Instructions Per Cycle
+`ipc`) do. Further down in the file, in the `main` function, we can see that
+the CPU object has a `path()` function which returns the component's "path" in
+the system, e.g. `system.bigCluster.cpus0`. The `path` function is provided by
+`SimObject` and so can be used by any object in the system which extends this,
+for example the l2 cache object uses it a couple of lines further down from
+where the CPU object uses it.
+
+(Note the division of `dcache.overall_misses` by `sim_seconds` to convert to
+Watts. This is a _power_ model, i.e. energy over time, and not an energy model.
+It is good to be cautious when using these terms as they are often used
+interchangeably, but mean very specific things when it comes to power and
+energy simulation/modelling.)
+
+Extending an existing simulation
+--------------------------------
+
+The provided `fs_power.py` script extends the existing `fs_bigLITTLE.py` script
+by importing it and then modifying the values. As part of this, several loops
+are used to iterate through the descendants of the SimObjects to apply the
+Power Models to. So to extend an existing simulation to support power models,
+it can be helpful to define a helper function which does this:
+
+```python
+def _apply_pm(simobj, power_model, so_class=None):
+ for desc in simobj.descendants():
+ if so_class is not None and not isinstance(desc, so_class):
+ continue
+
+ desc.power_state.default_state = "ON"
+ desc.power_model = power_model(desc.path())
+```
+
+The function above takes a SimObject, a Power Model, and optionally a class
+that the SimObject's descendant have to instantiate in order for the PM to be
+applied. If no class is specified, the PM is applied to all the descendants.
+
+Whether you decide to use the helper function or not, you now need to define
+some Power Models. This can be done by following the pattern seen in
+`fs_power.py`:
+
+0. Define a class for each of the power states you are interested in. These
+ classes should extend `MathExprPowerModel`, and contain a `dyn` and an `st`
+ field. Each of these fields should contain a string describing how to
+ calculate the respective type of power in this state. Their constructors
+ should take a path to be used through `format` in the strings describing the
+ power calculation equation, and a number of kwargs to be passed to the
+ super-constructor.
+1. Define a class to hold all the Power Models defined in the previous step.
+ This class should extend `PowerModel` and contain a single field `pm` which
+ contains a list of 4 elements: `pm[0]` should be an instance of the Power
+ Model for the "ON" power state; `pm[1]` should be an instance of the Power
+ Model for the "CLK_GATED" power state; etc. This class's constructor should
+ take the path to pass on to the individual Power Models, and a number of
+ kwargs which are passed to the super-constructor.
+2. With the helper function and the above classes defined, you can then extend
+ the `build` function to take these into account and optionally add a
+ command-line flag in the `addOptions` function if you want to be able to
+ toggle the use of the models.
+
+> **Example implementation:**
+>
+> ```python
+> class CpuPowerOn(MathExprPowerModel):
+> def __init__(self, cpu_path, **kwargs):
+> super(CpuPowerOn, self).__init__(**kwargs)
+> self.dyn = "voltage * 2 * {}.ipc".format(cpu_path)
+> self.st = "4 * temp"
+>
+>
+> class CpuPowerClkGated(MathExprPowerModel):
+> def __init__(self, cpu_path, **kwargs):
+> super(CpuPowerOn, self).__init__(**kwargs)
+> self.dyn = "voltage / sim_seconds"
+> self.st = "4 * temp"
+>
+>
+> class CpuPowerOff(MathExprPowerModel):
+> dyn = "0"
+> st = "0"
+>
+>
+> class CpuPowerModel(PowerModel):
+> def __init__(self, cpu_path, **kwargs):
+> super(CpuPowerModel, self).__init__(**kwargs)
+> self.pm = [
+> CpuPowerOn(cpu_path), # ON
+> CpuPowerClkGated(cpu_path), # CLK_GATED
+> CpuPowerOff(), # SRAM_RETENTION
+> CpuPowerOff(), # OFF
+> ]
+>
+> [...]
+>
+> def addOptions(parser):
+> [...]
+> parser.add_argument("--power-models", action="store_true",
+> help="Add power models to the simulated system. "
+> "Requires using the 'timing' CPU."
+> return parser
+>
+>
+> def build(options):
+> root = Root(full_system=True)
+> [...]
+> if options.power_models:
+> if options.cpu_type != "timing":
+> m5.fatal("The power models require the 'timing' CPUs.")
+>
+> _apply_pm(root.system.bigCluster.cpus, CpuPowerModel
+> so_class=m5.objects.BaseCpu)
+> _apply_pm(root.system.littleCluster.cpus, CpuPowerModel)
+>
+> return root
+>
+> [...]
+> ```
+
+Stat Names
+----------
+
+The stat names are usually the same as can be seen in the `stats.txt` file
+produced in the `m5out` directory after a simulation. However, there are some
+exceptions:
+
+- The CPU clock is referred to as `clk_domain.clock` in `stats.txt` but is
+ accessed in power models using `clock_period` and _not_ `clock`.
+
+Stat dump frequency
+-------------------
+
+By default, gem5 dumps simulation stats to the `stats.txt` file every simulated
+second. This can be controlled through the `m5.stats.periodicStatDump`
+function, which takes the desired frequency for dumping stats measured in
+simulated ticks, not seconds. Fortunately, `m5.ticks` provides a `fromSeconds`
+function for ease of usability.
+
+Below is an example of how stat dumping frequency affects result resolution,
+taken from [Sascha Bischoff's presentation](https://youtu.be/3gWyUWHxVj4) slide
+16:
+
+
+
+How frequently stats are dumped directly affects the resolution of the graphs
+that can be produced based on the `stats.txt` file. However, it also affects
+the size of the output file. Dumping stats every simulated second vs. every
+simulated millisecond increases the file size by a factor of several hundreds.
+Therefore, it makes sense to want to control the stat dump frequency.
+
+Using the provided `fs_power.py` script, this can be done as follows:
+
+```python
+[...]
+
+def addOptions(parser):
+ [...]
+ parser.add_argument("--stat-freq", type=float, default=1.0,
+ help="Frequency (in seconds) to dump stats to the "
+ "'stats.txt' file. Supports scientific notation, "
+ "e.g. '1.0E-3' for milliseconds.")
+ return parser
+
+[...]
+
+def main():
+ [...]
+ m5.stats.periodicStatDump(m5.ticks.fromSeconds(options.stat_freq))
+ bL.run()
+
+[...]
+```
+
+The stat dump frequency could then be specified using
+```
+--stat-freq <val>
+```
+when invoking the simulation.
+
+Common Problems
+---------------
+
+- gem5 crashes when using the provided `fs_power.py`, with the message `fatal:
+ statistic '' (160) was not properly initialized by a regStats() function`
+- gem5 crashes when using the provided `fs_power.py`, with the message `fatal:
+ Failed to evaluate power expressions: [...]`
+
+These are due to gem5's stats framework recently having been refactored.
+Getting the latest version of the gem5 source code and re-building should fix
+the problem. If this is not desirable, the following two sets of patches are
+required:
+
+1. [https://gem5-review.googlesource.com/c/public/gem5/+/26643](https://gem5-review.googlesource.com/c/public/gem5/+/26643)
+2. [https://gem5-review.googlesource.com/c/public/gem5/+/26785](https://gem5-review.googlesource.com/c/public/gem5/+/26785)
+
+These can be checked out and applied by following the download instructions at
+their respective links.
+
diff --git a/_pages/documentation/learning_gem5/part2/part2_8_arm_dvfs_support.md b/_pages/documentation/learning_gem5/part2/part2_8_arm_dvfs_support.md
new file mode 100644
index 0000000..c1b2401
--- /dev/null
+++ b/_pages/documentation/learning_gem5/part2/part2_8_arm_dvfs_support.md
@@ -0,0 +1,316 @@
+---
+layout: documentation
+title: ARM DVFS Support
+doc: Learning gem5
+parent: part2
+permalink: /documentation/learning_gem5/part2/arm_dvfs_support/
+author: Thomas E. Hansen
+---
+
+ARM DVFS modelling
+==================
+
+Like most modern CPUs, ARM CPUs support DVFS. It is possible to model this and,
+for example, monitor the resulting power usage in gem5. DVFS modelling is done
+through the use of two components of Clocked Objects: Voltage Domains and Clock
+Domains. This chapter details the different components and shows different ways
+to add them to an existing simulation.
+
+Voltage Domains
+---------------
+
+Voltage Domains dictate the voltage values the CPUs can use. If no VD is
+specified when running a Full System simulation in gem5, a default value of
+1.0 Volts is used. This is to avoid forcing users to consider voltage when they
+are not interested in simulating this.
+
+Voltage Domains can be constructed from either a single value or a list of
+values, passed to the `VoltageDomain` constructor using the `voltage` kwarg. If
+a single value and multiple frequencies are specified, the voltage is used for
+all the frequencies in the Clock Domain. If a list of voltage values is
+specified, its number of entries must match the number of entries in the
+corresponding Clock Domain and the entries must be arranged in _descending_
+order. As with real hardware, a Voltage Domain applies to the entire processor
+socket. This means that if you want to have different VDs for the different
+processors (e.g. for a big.LITTLE setup) you need to make sure the big and the
+LITTLE cluster are on different sockets (check the `socket_id` value associated
+with the clusters).
+
+There are 2 ways to add a VD to an existing CPU/simulation, one is more
+flexible, the other is more straightforward. The first method adds command-line
+flags to the provided `configs/example/arm/fs_bigLITTLE.py` file, while the
+second method adds custom classes.
+
+1. The most flexible way to add Voltage Domains to a simulation is to use
+ command-line flags. To add a command-line flag, find the `addOptions`
+ function in the file and add the flag there, optionally with some help
+ text.
+ An example supporting both a single and multiple voltages:
+
+ ```python
+ def addOptions(parser):
+ [...]
+ parser.add_argument("--big-cpu-voltage", nargs="+", default="1.0V",
+ help="Big CPU voltage(s).")
+ return parser
+ ```
+
+ The voltage domain value(s) could then be specified with
+
+ ```
+ --big-cpu-voltage <val1>V [<val2>V [<val3>V [...]]]
+ ```
+
+ This would then be accessed in the `build` function using
+ `options.big_cpu_voltage`. The `nargs="+"` ensures that at least one
+ argument is required.
+ Example usage in `build`:
+
+ ```python
+ def build(options):
+ [...]
+ # big cluster
+ if options.big_cpus > 0:
+ system.bigCluster = big_model(system, options.big_cpus,
+ options.big_cpu_clock,
+ options.big_cpu_voltage)
+ [...]
+ ```
+
+ A similar flag and additions to the `build` function could be added to
+ support specifying voltage values for the LITTLE CPU. This approach allows
+ for very easy specification and modification of the voltages. The only
+ downside to this method is that the multiple command line arguments, some
+ being in list form, could clutter up the command used to invoke the
+ simulator.
+
+2. The less flexible way to specify Voltage Domains is by creating sub-classes
+ of the `CpuCluster`. Similar to the existing `BigCluster` and
+ `LittleCluster` sub-classes, these will extend the `CpuCluster` class.
+ In the constructor of the subclass, in addition to specifying a CPU-type, we
+ also define a lists of values for the Voltage Domain and pass this to the
+ call to the `super` constructor using the kwarg `cpu_voltage`.
+ Here is an example, for adding voltage to a `BigCluster`:
+
+ ```python
+ class VDBigCluster(devices.CpuCluster):
+ def __init__(self, system, num_cpus, cpu_clock=None, cpu_voltage=None):
+ # use the same CPU as the stock BigCluster
+ abstract_cpu = ObjectList.cpu_list.get("O3_ARM_v7a_3")
+ # voltage value(s)
+ my_voltages = [ '1.0V', '0.75V', '0.51V']
+
+ super(VDBigCluster, self).__init__(
+ cpu_voltage=my_voltages,
+ system=system,
+ num_cpus=num_cpus,
+ cpu_type=abstract_cpu,
+ l1i_type=devices.L1I,
+ l1d_type=devices.L1D,
+ wcache_type=devices.WalkCache,
+ l2_type=devices.L2
+ )
+ ```
+
+ Adding voltages to the `LittleCluster` could then be done by defining a
+ similar `VDLittleCluster` class.
+
+ With the subclass(es) defined, we still need to add an entry to the
+ `cpu_types` dictionary in the file, specifying a string name as the key and
+ a pair of classes as the value, e.g:
+
+ ```python
+ cpu_types = {
+ [...]
+ "vd-timing" : (VDBigCluster, VDLittleCluster)
+ }
+ ```
+
+ The CPUs with VDs could then be used by passing
+
+ ```
+ --cpu-type vd-timing
+ ```
+
+ to the command invoking the simulation.
+
+ Since any modifications to the voltage values have to be done by finding the
+ right subclass and modifying its code, or adding more subclasses and
+ `cpu_types` entries, this approach is a lot less flexible than the
+ flag-based approach.
+
+Clock Domains
+-------------
+
+Voltage Domains are used in conjunction with Clock Domains. As previously
+mentioned, if no custom voltage values have been specified, a default value of
+1.0V is used for all values in the Clock Domain.
+
+Types of Clock Domain
+In contrast to Voltage Domains, there are 3 types of Clock Domains (from
+`src/sim/clock_domain.hh`):
+
+- `ClockDomain` -- provides a clock to a group of Clocked Objects bundled under
+ the same Clock Domain. The CDs are in turn grouped into Voltage Domains. The
+ CDs provide support for a hierarchical structure with "Source" and "Derived"
+ Clock Domains.
+- `SrcClockDomain` -- provides the notion of a CD that is connected to a
+ tunable clock source. It maintains the clock period and provides the methods
+ for setting/getting the clock, as well as the configuration parameters for
+ the CD that a handler is going to manage. This includes frequency values at
+ various performance levels, a Domain ID, and the current performance level.
+ Note that a performance level as requested by the software corresponds to one
+ of the frequency operation points the CD can operate at.
+- `DerivedClockDomain` -- provides the notion of a CD that is connected to a
+ parent CD which can either be a `SrcClockDomain` or a `DerivedClockDomain`.
+ It maintains the clock divider and provides methods for getting the clock.
+
+Adding Clock Domains to an existing simulation
+----------------------------------------------
+
+This example will use the same provided files as the VD examples, i.e.
+`configs/example/arm/fs_bigLITTLE.py` and `configs/example/arm/devices.py`.
+
+Like VDs, CDs can be a single value or a list of values. If a list of clock
+speeds is given, the same rules apply as for a list of voltages given to a VD,
+i.e. the number of values in the CD must match the number of values in the VD;
+and the clock speeds must be given in _descending_ order. The provided files
+come with support for specifying the clock as a single value (through the
+`--{big,little}-cpu-clock` flags), but not as a list of values.
+Extending/Modifying the behaviour of the provided flags is the simplest and
+most flexible way to add support for multi-value CDs, but it is also possible
+to do it by adding subclasses.
+
+1. To add multi-value support to the existing `--{big,little}-cpu-clock` flags,
+ locate the `addOptions` function in the
+ `configs/example/arm/fs_bigLITTLE.py` file. Amongst the various
+ `parser.add_argument` calls, find the ones that add the CPU-clock flags and
+ replace the kwarg `type=str` with `nargs="+"`:
+ ```python
+ def addOptions(parser):
+ [...]
+ parser.add_argument("--big-cpu-clock", nargs="+", default="2GHz",
+ help="Big CPU clock frequency.")
+ parser.add_argument("--little-cpu-clock", nargs="+", default="1GHz",
+ help="Little CPU clock frequency.")
+ [...]
+ ```
+ With this, multiple frequencies can be specified similarly to the flag used
+ for VDs:
+ ```
+ --{big,little}-cpu-clock <val1>GHz [<val2>MHz [<val3>MHz [...]]]
+ ```
+
+ Since this modifies existing flags, the flags' values are already wired up
+ to the relevant constructors and kwargs in the `build` function, so there is
+ nothing to be modified there.
+
+2. To add CDs in a subclass, the process is very similar to the process of
+ adding VDs as a subclass. The difference is that instead of specifying
+ voltages and using the `cpu_voltage` kwarg, we specify clock values and use
+ the `cpu_clock` kwarg in the `super` call:
+ ```python
+ class CDBigCluster(devices.CpuCluster):
+ def __init__(self, system, num_cpus, cpu_clock=None, cpu_voltage=None):
+ # use the same CPU as the stock BigCluster
+ abstract_cpu = ObjectList.cpu_list.get("O3_ARM_v7a_3")
+ # clock value(s)
+ my_freqs = [ '1510MHz', '1000MHz', '667MHz']
+
+ super(VDBigCluster, self).__init__(
+ cpu_clock=my_freqs,
+ system=system,
+ num_cpus=num_cpus,
+ cpu_type=abstract_cpu,
+ l1i_type=devices.L1I,
+ l1d_type=devices.L1D,
+ wcache_type=devices.WalkCache,
+ l2_type=devices.L2
+ )
+ ```
+ This could be combined with the VD example so as to specify both VDs and CDs
+ for the cluster.
+
+ As with adding VDs using this approach, you would need to define a class for
+ each of the CPU-types you wanted to use and specify their name-cpuPair value
+ in the `cpu_types` dictionary. This method also has the same limitations and
+ is a lot less flexible than the flag-based approach.
+
+Making sure CDs have a valid DomainID
+-------------------------------------
+
+Regardless of which of the previous methods are used, there are some additional
+modifications required. These concern the provided
+`configs/example/arm/devices.py` file.
+
+In the file, locate the `CpuClusters` class and find the place where
+`self.clk_domain` is initialised to a `SrcClockDomain`. As noted in the comment
+concerning `SrcClockDomain` above, these have a Domain ID. If this is not set,
+as is the case in the provided setup, then the default ID of `-1` will be used.
+Instead of this, change the code to make sure the Domain ID is set:
+
+```python
+[...]
+self.clk_domain = SrcClockDomain(clock=cpu_clock,
+ voltage_domain=self.voltage_domain,
+ domain_id=system.numCpuClusters())
+[...]
+```
+
+The `system.numCpuClusters()` is used here since the CD applies to the entire
+cluster, i.e. it will be 0 for the first cluster, 1 for the second cluster,
+etc.
+
+If you don't set the Domain ID, you will get the following error when trying to
+run a DVFS-capable simulation as some internal checks catch the default Domain
+ID:
+
+```
+fatal: fatal condition domain_id == SrcClockDomain::emptyDomainID occurred:
+DVFS: Controlled domain system.bigCluster.clk_domain needs to have a properly
+assigned ID.
+```
+
+The DVFS Handler
+----------------
+
+If you specify VDs and CDs and then try to run your simulation, it will most
+likely run, but you might notice the following warning in the output:
+
+```
+warn: Existing EnergyCtrl, but no enabled DVFSHandler found.
+```
+
+The VDs and CDs have been added, but there is no `DVFSHandler` which the system
+can interface with to adjust the values. The simplest way to fix this is to add
+another command-line flag, in the `configs/example/arm/fs_bigLITTLE.py` file.
+
+As in the VD and CD examples, locate the `addOptions` function and append the
+following code to it:
+
+```python
+def addOptions(parser):
+ [...]
+ parser.add_argument("--dvfs", action="store_true",
+ help="Enable the DVFS Handler.")
+ return parser
+```
+
+Then, locate the `build` function and append this code to it:
+
+```python
+def build(options):
+ [...]
+ if options.dvfs:
+ system.dvfs_handler.domains = [system.bigCluster.clk_domain,
+ system.littleCluster.clk_domain]
+ system.dvfs_handler.enable = options.dvfs
+
+ return root
+```
+
+With this in place, you should now be able to run a DVFS-capable simulation by
+using the `--dvfs` flag when invoking the simulation, with the option to
+specify the voltage and frequency operating points of both the big and the
+LITTLE cluster as necessary.
+
diff --git a/_pages/static/figures/empowering_the_masses_slide16.png b/_pages/static/figures/empowering_the_masses_slide16.png
new file mode 100644
index 0000000..75d45a9
--- /dev/null
+++ b/_pages/static/figures/empowering_the_masses_slide16.png
Binary files differ
