llvm::cas::ondisk::OnDiskGraphDB Class Reference

On-disk CAS nodes database, independent of a particular hashing algorithm. More...

#include "llvm/CAS/OnDiskGraphDB.h"

Classes
struct	IndexProxy
	Proxy for an on-disk index record. More...

Public Types
enum class	FaultInPolicy { SingleNode , FullTree }
	How to fault-in nodes if an upstream database is used. More...
using	HashingFuncT
	Hashing function type for validation.

Public Member Functions
Error	store (ObjectID ID, ArrayRef< ObjectID > Refs, ArrayRef< char > Data)
	Associate data & references with a particular object ID.
Expected< std::optional< ObjectHandle > >	load (ObjectID Ref)
ArrayRef< uint8_t >	getDigest (ObjectID Ref) const
Expected< ObjectID >	getReference (ArrayRef< uint8_t > Hash)
	Form a reference for the provided hash.
std::optional< ObjectID >	getExistingReference (ArrayRef< uint8_t > Digest)
	Get an existing reference to the object Digest.
Expected< bool >	isMaterialized (ObjectID Ref)
	Check whether the object associated with Ref is stored in the CAS.
bool	containsObject (ObjectID Ref) const
	Check whether the object associated with Ref is stored in the CAS.
ArrayRef< char >	getObjectData (ObjectHandle Node) const
object_refs_range	getObjectRefs (ObjectHandle Node) const
size_t	getStorageSize () const
unsigned	getHardStorageLimitUtilization () const
void	print (raw_ostream &OS) const
Error	validate (bool Deep, HashingFuncT Hasher) const
	Validate the OnDiskGraphDB.
	~OnDiskGraphDB ()

Static Public Member Functions
static Expected< std::unique_ptr< OnDiskGraphDB > >	open (StringRef Path, StringRef HashName, unsigned HashByteSize, std::unique_ptr< OnDiskGraphDB > UpstreamDB=nullptr, FaultInPolicy Policy=FaultInPolicy::FullTree)
	Open the on-disk store from a directory.

Detailed Description

On-disk CAS nodes database, independent of a particular hashing algorithm.

Definition at line 256 of file OnDiskGraphDB.h.

Member Typedef Documentation

HashingFuncT

using llvm::cas::ondisk::OnDiskGraphDB::HashingFuncT

Initial value:

 function_ref<void(
      ArrayRef<ArrayRef<uint8_t>>, ArrayRef<char>, SmallVectorImpl<uint8_t> &)>

Hashing function type for validation.

Definition at line 315 of file OnDiskGraphDB.h.

Member Enumeration Documentation

FaultInPolicy

enum class llvm::cas::ondisk::OnDiskGraphDB::FaultInPolicy

strong

How to fault-in nodes if an upstream database is used.

Enumerator
SingleNode	Copy only the requested node.
FullTree	Copy the the entire graph of a node.

Definition at line 327 of file OnDiskGraphDB.h.

Constructor & Destructor Documentation

~OnDiskGraphDB()

OnDiskGraphDB::~OnDiskGraphDB

(

)

Definition at line 1631 of file OnDiskGraphDB.cpp.

Member Function Documentation

containsObject()

bool llvm::cas::ondisk::OnDiskGraphDB::containsObject ( ObjectID Ref ) const

inline

Check whether the object associated with Ref is stored in the CAS.

Note that this function does not fault-in.

Definition at line 288 of file OnDiskGraphDB.h.

References containsObject(), and llvm::Ref.

Referenced by containsObject().

getDigest()

ArrayRef< uint8_t > llvm::cas::ondisk::OnDiskGraphDB::getDigest ( ObjectID Ref ) const

inline

Returns

the hash bytes digest for the object reference.

Definition at line 268 of file OnDiskGraphDB.h.

References llvm::cantFail(), getDigest(), and llvm::Ref.

Referenced by getDigest(), load(), and store().

getExistingReference()

std::optional< ObjectID > OnDiskGraphDB::getExistingReference

(

ArrayRef< uint8_t >

Digest

)

Get an existing reference to the object Digest.

Returns nullopt if the object is not stored in this CAS.

Definition at line 1120 of file OnDiskGraphDB.cpp.

References llvm::expectedToOptional(), I, LLVM_UNLIKELY, P, and llvm::Ref.

getHardStorageLimitUtilization()

unsigned OnDiskGraphDB::getHardStorageLimitUtilization

(

)

const

Returns

The precentage of space utilization of hard space limits.

Return value is an integer between 0 and 100 for percentage.

Definition at line 1546 of file OnDiskGraphDB.cpp.

getObjectData()

ArrayRef< char > OnDiskGraphDB::getObjectData

(

ObjectHandle

Node

)

const

Returns

the data part of the provided object handle.

Definition at line 1182 of file OnDiskGraphDB.cpp.

References assert(), and getContentFromHandle().

getObjectRefs()

object_refs_range llvm::cas::ondisk::OnDiskGraphDB::getObjectRefs ( ObjectHandle Node ) const

inline

Returns

the object referenced by the provided object handle.

Definition at line 296 of file OnDiskGraphDB.h.

References llvm::cas::ondisk::InternalRefArrayRef::begin(), llvm::cas::ondisk::InternalRefArrayRef::end(), and llvm::make_range().

getReference()

Expected< ObjectID > OnDiskGraphDB::getReference

(

ArrayRef< uint8_t >

Hash

)

Form a reference for the provided hash.

The reference can be used as part of a CAS object even if it's not associated with an object yet.

Definition at line 1108 of file OnDiskGraphDB.cpp.

References I, and LLVM_UNLIKELY.

getStorageSize()

size_t OnDiskGraphDB::getStorageSize

(

)

const

Returns

Total size of stored objects.

NOTE: There's a possibility that the returned size is not including a large object if the process crashed right at the point of inserting it.

Definition at line 1542 of file OnDiskGraphDB.cpp.

isMaterialized()

Expected< bool > OnDiskGraphDB::isMaterialized

(

ObjectID

Ref

)

Check whether the object associated with Ref is stored in the CAS.

Note that this function will fault-in according to the policy.

Definition at line 1260 of file OnDiskGraphDB.cpp.

References llvm_unreachable, and llvm::Ref.

load()

Expected< std::optional< ObjectHandle > > OnDiskGraphDB::load

(

ObjectID

Ref

)

Returns

nullopt if the object associated with Ref does not exist.

Definition at line 1198 of file OnDiskGraphDB.cpp.

References llvm::sys::fs::closeFile(), createCorruptObjectError(), llvm::createFileError(), llvm::cas::ondisk::ObjectHandle::fromFileOffset(), llvm::cas::ondisk::ObjectHandle::fromMemory(), getDigest(), I, llvm_unreachable, llvm::make_scope_exit(), llvm::sys::fs::openNativeFileForRead(), llvm::sys::fs::mapped_file_region::readonly, llvm::Ref, and llvm::sys::fs::status().

open()

Expected< std::unique_ptr< OnDiskGraphDB > > OnDiskGraphDB::open ( StringRef Path, StringRef HashName, unsigned HashByteSize, std::unique_ptr< OnDiskGraphDB > UpstreamDB = nullptr, FaultInPolicy Policy = FaultInPolicy::FullTree )

static

Open the on-disk store from a directory.

Parameters

Path	directory for the on-disk store. The directory will be created if it doesn't exist.
HashName	Identifier name for the hashing algorithm that is going to be used.
HashByteSize	Size for the object digest hash bytes.
UpstreamDB	Optional on-disk store to be used for faulting-in nodes if they don't exist in the primary store. The upstream store is only used for reading nodes, new nodes are only written to the primary store.
Policy	If UpstreamDB is provided, controls how nodes are copied to primary store. This is recorded at creation time and subsequent opens need to pass the same policy otherwise the open will fail.

Definition at line 1552 of file OnDiskGraphDB.cpp.

References llvm::sys::path::append(), llvm::argument_out_of_domain, llvm::cas::ondisk::CASFormatVersion, llvm::cas::OnDiskDataAllocator::create(), llvm::cas::OnDiskTrieRawHashMap::create(), llvm::sys::fs::create_directories(), llvm::createFileError(), llvm::createStringError(), DataPoolFilePrefix, DataPoolTableName, llvm::cas::ondisk::getOverriddenMaxMappingSize(), IndexFilePrefix, IndexTableName, SingleNode, and llvm::cas::ondisk::useSmallMappingSize().

print()

void OnDiskGraphDB::print

(

raw_ostream &

OS

)

const

Definition at line 1025 of file OnDiskGraphDB.cpp.

References assert(), D(), llvm::Data, llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::isAligned(), llvm::Align::Of(), llvm::Offset, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::sort(), and llvm::toString().

store()

Error OnDiskGraphDB::store	(	ObjectID	ID,
		ArrayRef< ObjectID >	Refs,
		ArrayRef< char >	Data )

Associate data & references with a particular object ID.

If there is already a record for this object the operation is a no-op.

Parameters

ID	the object ID to associate the data & references with.
Refs	references
Data	data buffer.

Definition at line 1407 of file OnDiskGraphDB.cpp.

References llvm::Alloc, assert(), createCorruptObjectError(), createTempFile(), llvm::Data, llvm::dbgs(), llvm::ArrayRef< T >::empty(), llvm::cas::FileOffset::get(), getDigest(), llvm::handleErrors(), I, LLVM_DEBUG, LLVM_UNLIKELY, P, llvm::Ref, Size, and llvm::Error::success().

validate()

Error OnDiskGraphDB::validate	(	bool	Deep,
		HashingFuncT	Hasher ) const

Validate the OnDiskGraphDB.

Parameters

Deep	if true, rehash all the objects to ensure no data corruption in stored objects, otherwise just validate the structure of CAS database.
Hasher	is the hashing function used for objects inside CAS.

Definition at line 895 of file OnDiskGraphDB.cpp.

References llvm::arrayRefFromStringRef(), llvm::createStringError(), D(), llvm::ArrayRef< T >::drop_back(), llvm::sys::fs::exists(), llvm::MemoryBuffer::getFile(), llvm::cas::ondisk::InternalRef::getFromOffset(), I, llvm::illegal_byte_sequence, llvm::isAligned(), llvm_unreachable, llvm::Align::Of(), llvm::Offset, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, llvm::Error::success(), llvm::toHex(), llvm::toString(), and llvm::utohexstr().

---

The documentation for this class was generated from the following files:

• include/llvm/CAS/OnDiskGraphDB.h
• lib/CAS/OnDiskGraphDB.cpp