Implementation of a caching mechanism for compressed data. This cache uses simple advisory locking found on most modern unix file systems. Compressed files are decompressed and stored in a cache where they can be used over and over until removed from the cache. Several processes can share the cache with each reading from files. At the same time, new files can be added and the cache can be purged, without disrupting the existing read operations. More...

#include <DAPCache3.h>

Inheritance diagram for libdap::DAPCache3:

Collaboration diagram for libdap::DAPCache3:

Public Member Functions
virtual bool	cache_too_big (unsigned long long current_size) const
	look at the cache size; is it too large? Look at the cache size and see if it is too big.

virtual bool	create_and_lock (const string &target, int &fd)
	Create a file in the cache and lock it for write access. If the file does not exist, make it, open it for read-write access and get an exclusive lock on it. The locking operation blocks, although that should never happen.

virtual void	dump (ostream &strm) const
	dumps information about this object

virtual void	dump (std::ostream &strm) const =0
	dump the contents of this object to the specified ostream

virtual void	exclusive_to_shared_lock (int fd)
	Transfer from an exclusive lock to a shared lock. If the file has an exclusive write lock on it, change that to a shared read lock. This is an atomic operation. If the call to fcntl(2) is protected by locking the cache, a dead lock will result given typical use of this class. This method exists to help with the situation where one process has the cache locked and is blocking on a shared read lock for a file that a second process has locked exclusively (for writing). By changing the exclusive lock to a shared lock, the first process can get its shared lock and then release the cache.

string	get_cache_file_name (const string &src, bool mangle=true)

virtual unsigned long long	get_cache_size ()
	Get the cache size. Read the size information from the cache info file and return it. This methods locks the cache.

virtual bool	get_read_lock (const string &target, int &fd)
	Get a read-only lock on the file if it exists.

virtual void	lock_cache_read ()

virtual void	lock_cache_write ()

virtual void	purge_file (const string &file)
	Purge a single file from the cache.

virtual void	unlock_and_close (const string &target)

virtual void	unlock_and_close (int fd)

virtual void	unlock_cache ()

virtual void	update_and_purge (const string &new_file)
	Purge files from the cache.

virtual unsigned long long	update_cache_info (const string &target)
	Update the cache info file to include 'target'.

Static Public Member Functions
static DAPCache3 *	get_instance ()

static DAPCache3 *	get_instance (const string &cache_dir, const string &prefix, unsigned long long size)

Detailed Description

Implementation of a caching mechanism for compressed data. This cache uses simple advisory locking found on most modern unix file systems. Compressed files are decompressed and stored in a cache where they can be used over and over until removed from the cache. Several processes can share the cache with each reading from files. At the same time, new files can be added and the cache can be purged, without disrupting the existing read operations.

How it works. When a file is added to the cache, the cache is locked - no other processes can add, read or remove files. Once a file has been added, the cache size is examined and, if needed, the cache is purged so that its size is 80% of the maximum size. Then the cache is unlocked. When a process looks to see if a file is already in the cache, the entire cache is locked. If the file is present, a shared read lock is obtained and the cache is unlocked.

Methods: create_and_lock() and get_read_lock() open and lock files; the former creates the file and locks it exclusively iff it does not exist, while the latter obtains a shared lock iff the file already exists. The unlock() methods unlock a file. The lock_cache_info() and unlock_cache_info() are used to control access to the whole cache - with the open + lock and close + unlock operations performed atomically. Other methods that operate on the cache info file must only be called when the lock has been obtained.

Definition at line 82 of file DAPCache3.h.

Member Function Documentation

◆ cache_too_big()

bool libdap::DAPCache3::cache_too_big ( unsigned long long current_size ) const

virtual

look at the cache size; is it too large? Look at the cache size and see if it is too big.

Returns: True if the size is too big, false otherwise.

Definition at line 735 of file DAPCache3.cc.

◆ create_and_lock()

bool libdap::DAPCache3::create_and_lock	(	const string &	target,
		int &	fd )

virtual

Create a file in the cache and lock it for write access. If the file does not exist, make it, open it for read-write access and get an exclusive lock on it. The locking operation blocks, although that should never happen.

Parameters

target	The name of the file to make/open/lock
fd	Value-result param that holds the file descriptor of the opened file

Returns: True if the operation was successful, false otherwise. This method will return false if the file already existed (the file won't be locked and the descriptor reference is undefined - but likely -1).

Exceptions

BESInternalError if any error except EEXIST is returned by open(2) or if fcntl(2) returns an error.

Definition at line 573 of file DAPCache3.cc.

◆ dump() [1/2]

void libdap::DAPCache3::dump ( ostream & strm ) const

virtual

dumps information about this object

Displays the pointer value of this instance along with information about this cache.

Parameters

strm	C++ i/o stream to dump the information to

Definition at line 958 of file DAPCache3.cc.

◆ dump() [2/2]

virtual void libdap::DapObj::dump ( std::ostream & strm ) const

pure virtualinherited

dump the contents of this object to the specified ostream

This method is implemented by all derived classes to dump their contents, in other words, any state they might have, private variables, etc...

Parameters

strm	C++ i/o stream to dump the object to

Implemented in libdap::Array, libdap::AttrTable, libdap::BaseType, libdap::Byte, libdap::Constructor, libdap::D4Attribute, libdap::D4Attributes, libdap::D4Enum, libdap::D4Opaque, libdap::D4Sequence, libdap::D4StreamMarshaller, libdap::D4StreamUnMarshaller, libdap::DAS, libdap::DataDDS, libdap::DDS, libdap::DMR, libdap::Float32, libdap::Float64, libdap::Grid, libdap::Int16, libdap::Int32, libdap::Int64, libdap::Int8, libdap::Marshaller, libdap::Sequence, libdap::Str, libdap::Structure, libdap::UInt16, libdap::UInt32, libdap::UInt64, libdap::UnMarshaller, libdap::Vector, libdap::XDRFileMarshaller, libdap::XDRFileUnMarshaller, libdap::XDRStreamMarshaller, and libdap::XDRStreamUnMarshaller.

◆ exclusive_to_shared_lock()

void libdap::DAPCache3::exclusive_to_shared_lock ( int fd )

virtual

Transfer from an exclusive lock to a shared lock. If the file has an exclusive write lock on it, change that to a shared read lock. This is an atomic operation. If the call to fcntl(2) is protected by locking the cache, a dead lock will result given typical use of this class. This method exists to help with the situation where one process has the cache locked and is blocking on a shared read lock for a file that a second process has locked exclusively (for writing). By changing the exclusive lock to a shared lock, the first process can get its shared lock and then release the cache.

Parameters

fd	The file descriptor that is exclusively locked and which, on exit, will have a shared lock.

Definition at line 601 of file DAPCache3.cc.

◆ get_cache_file_name()

string libdap::DAPCache3::get_cache_file_name	(	const string &	src,
		bool	mangle = true )

Build the name of file that will holds the uncompressed data from 'src' in the cache.

Note: How names are mangled: 'src' is the full name of the file to be cached.Tthe file name passed has an extension on the end that will be stripped once the file is cached. For example, if the full path to the file name is /usr/lib/data/fnoc1.nc.gz then the resulting file name will be #<prefix>#usr#lib#data#fnoc1.nc.

Parameters

src	The source name to cache
mangle	if True, assume the name is a file pathname and mangle it. If false, do not mangle the name (assume the caller has sent a suitable string) but do turn the string into a pathname located in the cache directory with the cache prefix. the 'mangle' param is true by default.

Definition at line 506 of file DAPCache3.cc.

◆ get_cache_size()

unsigned long long libdap::DAPCache3::get_cache_size ( )

virtual

Get the cache size. Read the size information from the cache info file and return it. This methods locks the cache.

Returns: The size of the cache.

Definition at line 746 of file DAPCache3.cc.

◆ get_instance() [1/2]

DAPCache3 * libdap::DAPCache3::get_instance ( )

static

Get an instance of the DAPCache3 object. This version is used when there's no question that the cache has been instantiated.

Definition at line 148 of file DAPCache3.cc.

◆ get_instance() [2/2]

DAPCache3 * libdap::DAPCache3::get_instance	(	const string &	cache_dir,
		const string &	prefix,
		unsigned long long	size )

static

Get an instance of the DAPCache3 object. This class is a singleton, so the first call to any of three 'get_instance()' methods makes an instance and subsequent calls return a pointer to that instance.

Parameters

cache_dir_key	Key to use to get the value of the cache directory
prefix_key	Key for the item/file prefix. Each file added to the cache uses this as a prefix so cached items can be easily identified when /tmp is used for the cache.
size_key	How big should the cache be, in megabytes

Returns: A pointer to a DAPCache3 object

Definition at line 135 of file DAPCache3.cc.

◆ get_read_lock()

bool libdap::DAPCache3::get_read_lock	(	const string &	target,
		int &	fd )

virtual

Get a read-only lock on the file if it exists.

Try to get a read-only lock on the file, blocking until we can get it. If the file does not exist, return false.

Note: If this code returns false, that means the file did not exist in the cache at the time of the test. by the time the caller gets the result, the file may have been added to the cache by another process.

Parameters

src	src file that will be cached eventually
target	a value-result parameter set to the resulting cached file

Returns: true if the file is in the cache and has been locked, false if the file is/was not in the cache.

Exceptions

Error if the attempt to get the (shared) lock failed for any reason other than that the file does/did not exist.

Definition at line 546 of file DAPCache3.cc.

◆ lock_cache_read()

void libdap::DAPCache3::lock_cache_read ( )

virtual

Get a shared lock on the 'cache info' file.

Definition at line 633 of file DAPCache3.cc.

◆ lock_cache_write()

void libdap::DAPCache3::lock_cache_write ( )

virtual

Get an exclusive lock on the 'cache info' file. The 'cache info' file is used to control certain cache actions, ensuring that they are atomic. These include making sure that the create_and_lock() and read_and_lock() operations are atomic as well as the purge and related operations.

Note: This is intended to be used internally only but might be useful in some settings.

Definition at line 622 of file DAPCache3.cc.

◆ purge_file()

void libdap::DAPCache3::purge_file ( const string & file )

virtual

Purge a single file from the cache.

Purge a single file from the cache. The file might be old, etc., and need to be removed. Don't use this to shrink the cache when it gets too big, use update_and_purge() instead since that file optimizes accesses to the cache control file for several changes in a row.

Todo: This is a new feature; add to BESCache3

Parameters

file	The name of the file to purge.

Definition at line 911 of file DAPCache3.cc.

◆ unlock_and_close() [1/2]

virtual void libdap::DAPCache3::unlock_and_close ( const string & target )

virtual

◆ unlock_and_close() [2/2]

void libdap::DAPCache3::unlock_and_close ( int fd )

virtual

Unlock the file. This does not do any name mangling; it just unlocks whatever is named (or throws BESInternalError if the file cannot be closed).

Parameters

fd	The descriptor of the file to unlock.

Exceptions

BESInternalError

Definition at line 677 of file DAPCache3.cc.

◆ unlock_cache()

void libdap::DAPCache3::unlock_cache ( )

virtual

Unlock the cache info file.

Note: This is intended to be used internally only bt might be useful in some settings.

Definition at line 646 of file DAPCache3.cc.

◆ update_and_purge()

void libdap::DAPCache3::update_and_purge ( const string & new_file )

virtual

Purge files from the cache.

Purge files, oldest to newest, if the current size of the cache exceeds the size of the cache specified in the constructor. This method uses an exclusive lock on the cache for the duration of the purge process.

Parameters

new_file The name of a file this process just added to the cache. Using fcntl(2) locking there is no way this process can detect its own lock, so the shared read lock on the new file won't keep this process from deleting it (but will keep other processes from deleting it).

Definition at line 821 of file DAPCache3.cc.

◆ update_cache_info()

unsigned long long libdap::DAPCache3::update_cache_info ( const string & target )

virtual

Update the cache info file to include 'target'.

Add the size of the named file to the total cache size recorded in the cache info file. The cache info file is exclusively locked by this method for its duration. This updates the cache info file and returns the new size.

Parameters

target The name of the file

Returns: The new size of the cache

Definition at line 695 of file DAPCache3.cc.

The documentation for this class was generated from the following files:

Public Member Functions

Static Public Member Functions

Detailed Description

Member Function Documentation

◆ cache_too_big()

◆ create_and_lock()

◆ dump() [1/2]

◆ dump() [2/2]

◆ exclusive_to_shared_lock()

◆ get_cache_file_name()

◆ get_cache_size()

◆ get_instance() [1/2]

◆ get_instance() [2/2]

◆ get_read_lock()

◆ lock_cache_read()

◆ lock_cache_write()

◆ purge_file()

◆ unlock_and_close() [1/2]

◆ unlock_and_close() [2/2]

◆ unlock_cache()

◆ update_and_purge()

◆ update_cache_info()