Implementation of a caching mechanism for files. More...

#include <FileCache.h>

Collaboration diagram for FileCache:

Classes
class	Item
	Manage the state of an open file descriptor for a cached item. More...

class	PutItem

Public Member Functions
bool	clear () const
	Remove all files from the cache. Zero the cache info file.

bool	del (const std::string &key, int lock_type=LOCK_EX\|LOCK_NB)
	Remove the item at the given key Remove the key/item. Updates the size recorded in cache_info. Returns false if a number of bad things happen OR if the item could not be locked (i.e., there was an error or the item is in use).

	FileCache (const FileCache &)=delete

bool	get (const std::string &key, Item &item, int lock_type=LOCK_SH\|LOCK_NB)
	Get a locked (shared) item for a file in the cache.

virtual bool	initialize (const std::string &cache_dir, long long size, long long purge_size)
	Initialize the cache.

FileCache &	operator= (const FileCache &rhs)=delete

bool	purge ()
	Purge the lest recently used items from the cache. The purge() method for FileCache is public. It is the user's job to call purge. The idea behind this is that user code can decide if purge needs to be called on every put, every Nth put or not at all. Note that purge() (often) does nothing more than compare the size recorded in the cache_info file (updated on every put()) with the configured max cache size.

bool	put (const std::string &key, const std::string &file_name)
	Put an item in the cache Put the contents of a file in the cache, referenced by the given key. The item is unlocked when this method returns and cache_info file is updated.

bool	put (const std::string &key, PutItem &item)
	Put an item in the cache Put an item in the cache by returning an open and lock file descriptor to the item. The called can write directly to the item, rewind the descriptor and read from it and then close the file descriptor to release the Exclusive lock.

bool	put_data (const std::string &key, const std::string &data)

Static Public Member Functions
static std::string	hash_key (const std::string &key, bool log_it=false)
	Return a SHA256 hash of the given key.

Friends
class	FileCacheTest

Detailed Description

Implementation of a caching mechanism for files.

This cache uses simple advisory locking found on most modern unix file systems. It was originally designed to hold the decompressed versions of compressed files.

Compressed files are uncompressed and stored in a cache where they can be used over and over until removed from the cache. Several processes can share the cache with each reading from files. At the same time, new files can be added and the cache can be purged, without disrupting the existing read operations.

How it works: Before a file is added to the cache, the cache is locked - no other processes can add, read or remove files. Once a file has been added, the cache size is updated cache is unlocked. Unlike other caches, this implementation does not automatically purge entries when the size becomes too large. It is up to the client code to call the purge() method when the size is at the maximum size.

When a process tries to get a file that is already in the cache, the entire cache is locked. If the file is present, a shared read lock on the cached file is obtained and the cache is unlocked.

For the put(), get(), and del() methods, the client code must manage the mapping between the things in the cache and the keys.

This is the Nth rewrite of the original BES 'uncompress cache' and it now supplies a much simpler interface: initialize(), put(), get(), del(), and purge(). The constructor for FileCache no longer initializes the cache, use the named method for that. The put(key, source file) method locks the cache and copies the contents of 'source file' into a new file that is named 'key.' The get(key, item) method opens the file named 'key' and returns an instance of FileCache::Item that holds the locked (shared, using flock(2)) file. Use close(item.get_fd()) to release the lock and close the file. The del(key) method deletes the file if it can get an exclusive lock on the file. The del() method is the only method that uses a non-blocking lock on a file. If can be forced to use a blocking lock using an optional second argument. There is a second put(key, PutItem) method that returns a PutItem instance that holds an open, locked, file descriptor. The PutItem dtor updates the cache_info file. This method exists to allow the caller to write directly to the file and then close the file descriptor to release the lock.

Note: The locking mechanism uses Unix flock(2) and so is per file. Using flock(2) instead of fcntl(2) means that the locking is thread-safe. On older linux kernels (< 2.6.12) flock(2) does not work with NFSv4. Based on stackoverflow0, it should work with an AWS EFS volume and newer NFS implementations. (YMMV).

Definition at line 122 of file FileCache.h.

Constructor & Destructor Documentation

◆ ~FileCache()

virtual FileCache::~FileCache ( )

inlinevirtual

Definition at line 399 of file FileCache.h.

Member Function Documentation

◆ clear()

bool FileCache::clear ( ) const

inline

Remove all files from the cache. Zero the cache info file.

Returns: false if the cache directory could not be opened or a file in the cache could not be removed, true otherwise.

Definition at line 646 of file FileCache.h.

◆ del()

bool FileCache::del	(	const std::string &	key,
		int	lock_type = LOCK_EX \| LOCK_NB )

inline

Remove the item at the given key Remove the key/item. Updates the size recorded in cache_info. Returns false if a number of bad things happen OR if the item could not be locked (i.e., there was an error or the item is in use).

Note: This is not used by purge()

Parameters

key	The item key
lock_type	The kind of lock to use; Exclusive Non-blocking by default.

Returns: True if the key/item is deleted, false if not.

Definition at line 611 of file FileCache.h.

◆ get()

bool FileCache::get	(	const std::string &	key,
		Item &	item,
		int	lock_type = LOCK_SH \| LOCK_NB )

inline

Get a locked (shared) item for a file in the cache.

Parameters

key	The key to the cached item
item	A reference to an Item - a value-result parameter. Release the lock by closing the file.
lock_type	By default, get a shared non-blocking lock.

Returns: True if the item was found and locked, false otherwise

Definition at line 574 of file FileCache.h.

◆ hash_key()

static std::string FileCache::hash_key	(	const std::string &	key,
		bool	log_it = false )

inlinestatic

Return a SHA256 hash of the given key.

Parameters

key	The key to has
log_it	If true, write an info message to the bes log so it's easier to track the key --> hash mapping. False by default.

Returns: The SHA256 hash of the key.

Definition at line 314 of file FileCache.h.

◆ initialize()

virtual bool FileCache::initialize	(	const std::string &	cache_dir,
		long long	size,
		long long	purge_size )

inlinevirtual

Initialize the cache.

Parameters

cache_dir	Directory on some filesystem where the cache will be stored. This directory is made if it does not exist.
size	Allow this many bytes in the cache
purge_size	When purging, remove items until this many bytes remain.

Returns: False if the cache object could not be initialized, true otherwise.

Definition at line 413 of file FileCache.h.

◆ purge()

bool FileCache::purge ( )

inline

Purge the lest recently used items from the cache. The purge() method for FileCache is public. It is the user's job to call purge. The idea behind this is that user code can decide if purge needs to be called on every put, every Nth put or not at all. Note that purge() (often) does nothing more than compare the size recorded in the cache_info file (updated on every put()) with the configured max cache size.

Returns: True if the purge operation encountered no errors, false if failures were found. Note that if an entry cannot be removed, that is not an error because the item might be locked since it's in use.

Definition at line 678 of file FileCache.h.

◆ put() [1/2]

bool FileCache::put	(	const std::string &	key,
		const std::string &	file_name )

inline

Put an item in the cache Put the contents of a file in the cache, referenced by the given key. The item is unlocked when this method returns and cache_info file is updated.

Note: This method copies the file contents to cache it.

Parameters

key	The key used to access the file
file_name	The name of the file that will be cached.

Returns: True if the data are cached, false otherwise.

Definition at line 450 of file FileCache.h.

◆ put() [2/2]

bool FileCache::put	(	const std::string &	key,
		PutItem &	item )

inline

Put an item in the cache Put an item in the cache by returning an open and lock file descriptor to the item. The called can write directly to the item, rewind the descriptor and read from it and then close the file descriptor to release the Exclusive lock.

Note: When the PutItem goes out of scope, the cache_info file is updated. Make sure no operation that locks the cache gets called before the PutItem lock is removed, otherwise there will be a deadlock.

Parameters

key	The key that can be used to access the file
item	A value-result parameter than is a reference to a PutItem instance.

Returns: True if the PutItem holds an open, locked, file descriptor, otherwise false.

Definition at line 544 of file FileCache.h.

◆ put_data()

bool FileCache::put_data	(	const std::string &	key,
		const std::string &	data )

inline

Definition at line 496 of file FileCache.h.

Friends And Related Symbol Documentation

◆ FileCacheTest

friend class FileCacheTest

friend

Definition at line 304 of file FileCache.h.

The documentation for this class was generated from the following file:

FileCache.h

Classes

Public Member Functions

Static Public Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

◆ ~FileCache()

Member Function Documentation

◆ clear()

◆ del()

◆ get()

◆ hash_key()

◆ initialize()

◆ purge()

◆ put() [1/2]

◆ put() [2/2]

◆ put_data()

Friends And Related Symbol Documentation

◆ FileCacheTest