| ==================================================== |
| IN-KERNEL CACHE OBJECT REPRESENTATION AND MANAGEMENT |
| ==================================================== |
| |
| By: David Howells <dhowells@redhat.com> |
| |
| Contents: |
| |
| (*) Representation |
| |
| (*) Object management state machine. |
| |
| - Provision of cpu time. |
| - Locking simplification. |
| |
| (*) The set of states. |
| |
| (*) The set of events. |
| |
| |
| ============== |
| REPRESENTATION |
| ============== |
| |
| FS-Cache maintains an in-kernel representation of each object that a netfs is |
| currently interested in. Such objects are represented by the fscache_cookie |
| struct and are referred to as cookies. |
| |
| FS-Cache also maintains a separate in-kernel representation of the objects that |
| a cache backend is currently actively caching. Such objects are represented by |
| the fscache_object struct. The cache backends allocate these upon request, and |
| are expected to embed them in their own representations. These are referred to |
| as objects. |
| |
| There is a 1:N relationship between cookies and objects. A cookie may be |
| represented by multiple objects - an index may exist in more than one cache - |
| or even by no objects (it may not be cached). |
| |
| Furthermore, both cookies and objects are hierarchical. The two hierarchies |
| correspond, but the cookies tree is a superset of the union of the object trees |
| of multiple caches: |
| |
| NETFS INDEX TREE : CACHE 1 : CACHE 2 |
| : : |
| : +-----------+ : |
| +----------->| IObject | : |
| +-----------+ | : +-----------+ : |
| | ICookie |-------+ : | : |
| +-----------+ | : | : +-----------+ |
| | +------------------------------>| IObject | |
| | : | : +-----------+ |
| | : V : | |
| | : +-----------+ : | |
| V +----------->| IObject | : | |
| +-----------+ | : +-----------+ : | |
| | ICookie |-------+ : | : V |
| +-----------+ | : | : +-----------+ |
| | +------------------------------>| IObject | |
| +-----+-----+ : | : +-----------+ |
| | | : | : | |
| V | : V : | |
| +-----------+ | : +-----------+ : | |
| | ICookie |------------------------->| IObject | : | |
| +-----------+ | : +-----------+ : | |
| | V : | : V |
| | +-----------+ : | : +-----------+ |
| | | ICookie |-------------------------------->| IObject | |
| | +-----------+ : | : +-----------+ |
| V | : V : | |
| +-----------+ | : +-----------+ : | |
| | DCookie |------------------------->| DObject | : | |
| +-----------+ | : +-----------+ : | |
| | : : | |
| +-------+-------+ : : | |
| | | : : | |
| V V : : V |
| +-----------+ +-----------+ : : +-----------+ |
| | DCookie | | DCookie |------------------------>| DObject | |
| +-----------+ +-----------+ : : +-----------+ |
| : : |
| |
| In the above illustration, ICookie and IObject represent indices and DCookie |
| and DObject represent data storage objects. Indices may have representation in |
| multiple caches, but currently, non-index objects may not. Objects of any type |
| may also be entirely unrepresented. |
| |
| As far as the netfs API goes, the netfs is only actually permitted to see |
| pointers to the cookies. The cookies themselves and any objects attached to |
| those cookies are hidden from it. |
| |
| |
| =============================== |
| OBJECT MANAGEMENT STATE MACHINE |
| =============================== |
| |
| Within FS-Cache, each active object is managed by its own individual state |
| machine. The state for an object is kept in the fscache_object struct, in |
| object->state. A cookie may point to a set of objects that are in different |
| states. |
| |
| Each state has an action associated with it that is invoked when the machine |
| wakes up in that state. There are four logical sets of states: |
| |
| (1) Preparation: states that wait for the parent objects to become ready. The |
| representations are hierarchical, and it is expected that an object must |
| be created or accessed with respect to its parent object. |
| |
| (2) Initialisation: states that perform lookups in the cache and validate |
| what's found and that create on disk any missing metadata. |
| |
| (3) Normal running: states that allow netfs operations on objects to proceed |
| and that update the state of objects. |
| |
| (4) Termination: states that detach objects from their netfs cookies, that |
| delete objects from disk, that handle disk and system errors and that free |
| up in-memory resources. |
| |
| |
| In most cases, transitioning between states is in response to signalled events. |
| When a state has finished processing, it will usually set the mask of events in |
| which it is interested (object->event_mask) and relinquish the worker thread. |
| Then when an event is raised (by calling fscache_raise_event()), if the event |
| is not masked, the object will be queued for processing (by calling |
| fscache_enqueue_object()). |
| |
| |
| PROVISION OF CPU TIME |
| --------------------- |
| |
| The work to be done by the various states was given CPU time by the threads of |
| the slow work facility. This was used in preference to the workqueue facility |
| because: |
| |
| (1) Threads may be completely occupied for very long periods of time by a |
| particular work item. These state actions may be doing sequences of |
| synchronous, journalled disk accesses (lookup, mkdir, create, setxattr, |
| getxattr, truncate, unlink, rmdir, rename). |
| |
| (2) Threads may do little actual work, but may rather spend a lot of time |
| sleeping on I/O. This means that single-threaded and 1-per-CPU-threaded |
| workqueues don't necessarily have the right numbers of threads. |
| |
| |
| LOCKING SIMPLIFICATION |
| ---------------------- |
| |
| Because only one worker thread may be operating on any particular object's |
| state machine at once, this simplifies the locking, particularly with respect |
| to disconnecting the netfs's representation of a cache object (fscache_cookie) |
| from the cache backend's representation (fscache_object) - which may be |
| requested from either end. |
| |
| |
| ================= |
| THE SET OF STATES |
| ================= |
| |
| The object state machine has a set of states that it can be in. There are |
| preparation states in which the object sets itself up and waits for its parent |
| object to transit to a state that allows access to its children: |
| |
| (1) State FSCACHE_OBJECT_INIT. |
| |
| Initialise the object and wait for the parent object to become active. In |
| the cache, it is expected that it will not be possible to look an object |
| up from the parent object, until that parent object itself has been looked |
| up. |
| |
| There are initialisation states in which the object sets itself up and accesses |
| disk for the object metadata: |
| |
| (2) State FSCACHE_OBJECT_LOOKING_UP. |
| |
| Look up the object on disk, using the parent as a starting point. |
| FS-Cache expects the cache backend to probe the cache to see whether this |
| object is represented there, and if it is, to see if it's valid (coherency |
| management). |
| |
| The cache should call fscache_object_lookup_negative() to indicate lookup |
| failure for whatever reason, and should call fscache_obtained_object() to |
| indicate success. |
| |
| At the completion of lookup, FS-Cache will let the netfs go ahead with |
| read operations, no matter whether the file is yet cached. If not yet |
| cached, read operations will be immediately rejected with ENODATA until |
| the first known page is uncached - as to that point there can be no data |
| to be read out of the cache for that file that isn't currently also held |
| in the pagecache. |
| |
| (3) State FSCACHE_OBJECT_CREATING. |
| |
| Create an object on disk, using the parent as a starting point. This |
| happens if the lookup failed to find the object, or if the object's |
| coherency data indicated what's on disk is out of date. In this state, |
| FS-Cache expects the cache to create |
| |
| The cache should call fscache_obtained_object() if creation completes |
| successfully, fscache_object_lookup_negative() otherwise. |
| |
| At the completion of creation, FS-Cache will start processing write |
| operations the netfs has queued for an object. If creation failed, the |
| write ops will be transparently discarded, and nothing recorded in the |
| cache. |
| |
| There are some normal running states in which the object spends its time |
| servicing netfs requests: |
| |
| (4) State FSCACHE_OBJECT_AVAILABLE. |
| |
| A transient state in which pending operations are started, child objects |
| are permitted to advance from FSCACHE_OBJECT_INIT state, and temporary |
| lookup data is freed. |
| |
| (5) State FSCACHE_OBJECT_ACTIVE. |
| |
| The normal running state. In this state, requests the netfs makes will be |
| passed on to the cache. |
| |
| (6) State FSCACHE_OBJECT_INVALIDATING. |
| |
| The object is undergoing invalidation. When the state comes here, it |
| discards all pending read, write and attribute change operations as it is |
| going to clear out the cache entirely and reinitialise it. It will then |
| continue to the FSCACHE_OBJECT_UPDATING state. |
| |
| (7) State FSCACHE_OBJECT_UPDATING. |
| |
| The state machine comes here to update the object in the cache from the |
| netfs's records. This involves updating the auxiliary data that is used |
| to maintain coherency. |
| |
| And there are terminal states in which an object cleans itself up, deallocates |
| memory and potentially deletes stuff from disk: |
| |
| (8) State FSCACHE_OBJECT_LC_DYING. |
| |
| The object comes here if it is dying because of a lookup or creation |
| error. This would be due to a disk error or system error of some sort. |
| Temporary data is cleaned up, and the parent is released. |
| |
| (9) State FSCACHE_OBJECT_DYING. |
| |
| The object comes here if it is dying due to an error, because its parent |
| cookie has been relinquished by the netfs or because the cache is being |
| withdrawn. |
| |
| Any child objects waiting on this one are given CPU time so that they too |
| can destroy themselves. This object waits for all its children to go away |
| before advancing to the next state. |
| |
| (10) State FSCACHE_OBJECT_ABORT_INIT. |
| |
| The object comes to this state if it was waiting on its parent in |
| FSCACHE_OBJECT_INIT, but its parent died. The object will destroy itself |
| so that the parent may proceed from the FSCACHE_OBJECT_DYING state. |
| |
| (11) State FSCACHE_OBJECT_RELEASING. |
| (12) State FSCACHE_OBJECT_RECYCLING. |
| |
| The object comes to one of these two states when dying once it is rid of |
| all its children, if it is dying because the netfs relinquished its |
| cookie. In the first state, the cached data is expected to persist, and |
| in the second it will be deleted. |
| |
| (13) State FSCACHE_OBJECT_WITHDRAWING. |
| |
| The object transits to this state if the cache decides it wants to |
| withdraw the object from service, perhaps to make space, but also due to |
| error or just because the whole cache is being withdrawn. |
| |
| (14) State FSCACHE_OBJECT_DEAD. |
| |
| The object transits to this state when the in-memory object record is |
| ready to be deleted. The object processor shouldn't ever see an object in |
| this state. |
| |
| |
| THE SET OF EVENTS |
| ----------------- |
| |
| There are a number of events that can be raised to an object state machine: |
| |
| (*) FSCACHE_OBJECT_EV_UPDATE |
| |
| The netfs requested that an object be updated. The state machine will ask |
| the cache backend to update the object, and the cache backend will ask the |
| netfs for details of the change through its cookie definition ops. |
| |
| (*) FSCACHE_OBJECT_EV_CLEARED |
| |
| This is signalled in two circumstances: |
| |
| (a) when an object's last child object is dropped and |
| |
| (b) when the last operation outstanding on an object is completed. |
| |
| This is used to proceed from the dying state. |
| |
| (*) FSCACHE_OBJECT_EV_ERROR |
| |
| This is signalled when an I/O error occurs during the processing of some |
| object. |
| |
| (*) FSCACHE_OBJECT_EV_RELEASE |
| (*) FSCACHE_OBJECT_EV_RETIRE |
| |
| These are signalled when the netfs relinquishes a cookie it was using. |
| The event selected depends on whether the netfs asks for the backing |
| object to be retired (deleted) or retained. |
| |
| (*) FSCACHE_OBJECT_EV_WITHDRAW |
| |
| This is signalled when the cache backend wants to withdraw an object. |
| This means that the object will have to be detached from the netfs's |
| cookie. |
| |
| Because the withdrawing releasing/retiring events are all handled by the object |
| state machine, it doesn't matter if there's a collision with both ends trying |
| to sever the connection at the same time. The state machine can just pick |
| which one it wants to honour, and that effects the other. |
