|
LLVM 24.0.0git
|
Content-addressable storage for objects. More...
#include "llvm/CAS/ObjectStore.h"
Public Member Functions | |
| virtual Expected< CASID > | parseID (StringRef ID)=0 |
Get a CASID from a ID, which should have been generated by CASID::print(). | |
| virtual Expected< ObjectRef > | store (ArrayRef< ObjectRef > Refs, ArrayRef< char > Data)=0 |
| Store object into ObjectStore. | |
| virtual CASID | getID (ObjectRef Ref) const =0 |
Get an ID for Ref. | |
| virtual Expected< ObjectRef > | storeFromFile (StringRef Path) |
| Stores the data of a file into ObjectStore. | |
| virtual Error | exportDataToFile (ObjectHandle Node, StringRef Path) const |
| Exports the data of an object to a file path. | |
| virtual std::optional< ObjectRef > | getReference (const CASID &ID) const =0 |
Get an existing reference to the object called ID. | |
| virtual Expected< bool > | isMaterialized (ObjectRef Ref) const =0 |
| virtual Error | validateObject (const CASID &ID)=0 |
| Validate the underlying object referred by CASID. | |
| virtual Error | validate (bool CheckHash) const =0 |
| Validate the entire ObjectStore. | |
| Expected< ObjectProxy > | createProxy (ArrayRef< ObjectRef > Refs, StringRef Data) |
| Helper functions to store object and returns a ObjectProxy. | |
| Expected< ObjectRef > | storeFromString (ArrayRef< ObjectRef > Refs, StringRef String) |
| Store object from StringRef. | |
| Expected< ObjectRef > | storeFromOpenFile (sys::fs::file_t FD, std::optional< sys::fs::file_status > Status=std::nullopt) |
Default implementation reads FD and calls storeNode(). | |
| Expected< ObjectProxy > | getProxy (const CASID &ID) |
| Create ObjectProxy from CASID. If the object doesn't exist, get an error. | |
| Expected< ObjectProxy > | getProxy (ObjectRef Ref) |
| Create ObjectProxy from ObjectRef. | |
| Expected< std::optional< ObjectProxy > > | getProxyIfExists (ObjectRef Ref) |
| uint64_t | readData (ObjectHandle Node, raw_ostream &OS, uint64_t Offset=0, uint64_t MaxBytes=-1ULL) const |
Read the data from Data into OS. | |
| virtual Error | setSizeLimit (std::optional< uint64_t > SizeLimit) |
| Set the size for limiting growth of on-disk storage. | |
| virtual Expected< std::optional< uint64_t > > | getStorageSize () const |
| virtual Error | pruneStorageData () |
| Prune local storage to reduce its size according to the desired size limit. | |
| Error | validateTree (ObjectRef Ref) |
| Validate the whole node tree. | |
| Expected< ObjectRef > | importObject (ObjectStore &Upstream, ObjectRef Other) |
| Import object from another CAS. | |
| virtual void | print (raw_ostream &) const |
| Print the ObjectStore internals for debugging purpose. | |
| void | dump () const |
| const CASContext & | getContext () const |
| Get CASContext. | |
| virtual | ~ObjectStore ()=default |
Static Public Member Functions | |
| static Error | createUnknownObjectError (const CASID &ID) |
Protected Member Functions | |
| virtual Expected< std::optional< ObjectHandle > > | loadIfExists (ObjectRef Ref)=0 |
Load the object referenced by Ref. | |
| Expected< ObjectHandle > | load (ObjectRef Ref) |
Like loadIfExists but returns an error if the object is missing. | |
| virtual uint64_t | getDataSize (ObjectHandle Node) const =0 |
| Get the size of some data. | |
| virtual Error | forEachRef (ObjectHandle Node, function_ref< Error(ObjectRef)> Callback) const =0 |
| Methods for handling objects. | |
| virtual ObjectRef | readRef (ObjectHandle Node, size_t I) const =0 |
| virtual size_t | getNumRefs (ObjectHandle Node) const =0 |
| virtual ArrayRef< char > | getData (ObjectHandle Node, bool RequiresNullTerminator=false) const =0 |
| virtual Expected< ObjectRef > | storeFromOpenFileImpl (sys::fs::file_t FD, std::optional< sys::fs::file_status > Status) |
| Get ObjectRef from open file. | |
| virtual std::unique_ptr< MemoryBuffer > | getStandaloneMemoryBufferImpl (ObjectHandle Node, StringRef Name, bool RequiresNullTerminator) |
| Customization point for getStandaloneMemoryBuffer(). | |
| StringRef | getDataString (ObjectHandle Node) |
Get a lifetime-extended StringRef pointing at Data. | |
| std::unique_ptr< MemoryBuffer > | getMemoryBuffer (ObjectHandle Node, StringRef Name="", bool RequiresNullTerminator=true) |
Get a MemoryBuffer pointing at Data. | |
| std::unique_ptr< MemoryBuffer > | getStandaloneMemoryBuffer (ObjectHandle Node, StringRef Name="", bool RequiresNullTerminator=true) |
Get a MemoryBuffer for Node that stays valid after this ObjectStore is destroyed. | |
| virtual void | readRefs (ObjectHandle Node, SmallVectorImpl< ObjectRef > &Refs) const |
| Read all the refs from object in a SmallVector. | |
| ObjectStore (const CASContext &Context) | |
Friends | |
| class | ObjectProxy |
Content-addressable storage for objects.
Conceptually, objects are stored in a "unique set".
The ObjectStore interface has a few ways of referencing objects:
Both ObjectRef and ObjectHandle are lightweight, wrapping a uint64_t and are only valid with the associated ObjectStore instance.
There are a few options for accessing content of objects, with different lifetime tradeoffs:
Definition at line 93 of file ObjectStore.h.
|
virtualdefault |
|
inlineprotected |
Definition at line 311 of file ObjectStore.h.
Referenced by llvm::cas::builtin::BuiltinCAS::BuiltinCAS(), and importObject().
| Expected< ObjectProxy > ObjectStore::createProxy | ( | ArrayRef< ObjectRef > | Refs, |
| StringRef | Data ) |
Helper functions to store object and returns a ObjectProxy.
Definition at line 130 of file ObjectStore.cpp.
References llvm::arrayRefFromStringRef(), llvm::Data, getProxy(), llvm::Ref, and store().
Referenced by llvm::cas::NamedValuesSchema::construct().
Definition at line 125 of file ObjectStore.cpp.
References llvm::createStringError().
Referenced by getProxy().
| LLVM_DUMP_METHOD void ObjectStore::dump | ( | ) | const |
Definition at line 27 of file ObjectStore.cpp.
References llvm::dbgs(), LLVM_DUMP_METHOD, and print().
|
virtual |
Exports the data of an object to a file path.
It does not include any references of the object.
An underlying implementation could perform optimizations that reduce I/O and disk space consumption.
| Node | the object to read data from. |
| FilePath | the path of the file data. |
Definition at line 161 of file ObjectStore.cpp.
References llvm::sys::path::append(), llvm::SmallVectorImpl< T >::clear(), llvm::createFileError(), llvm::sys::fs::createUniqueFile(), llvm::Data, llvm::SmallVectorTemplateCommon< T, typename >::empty(), getData(), llvm::sys::path::parent_path(), llvm::sys::fs::remove(), llvm::sys::fs::rename(), llvm::scope_exit(), llvm::sys::sandbox::scopedDisable(), and llvm::Error::success().
Referenced by getID().
|
protectedpure virtual |
Methods for handling objects.
CAS implementations need to override to provide functions to access stored CAS objects and references.
Referenced by readRefs(), and validateTree().
|
inline |
Get CASContext.
Definition at line 306 of file ObjectStore.h.
Referenced by llvm::cas::builtin::BuiltinCAS::parseID().
|
protectedpure virtual |
Implemented in llvm::cas::builtin::BuiltinCAS.
References getStandaloneMemoryBufferImpl(), and storeFromOpenFileImpl().
Referenced by exportDataToFile(), getDataString(), getMemoryBuffer(), getStandaloneMemoryBufferImpl(), importObject(), and readData().
|
protectedpure virtual |
Get the size of some data.
Implemented in llvm::cas::builtin::BuiltinCAS.
|
inlineprotected |
Get a lifetime-extended StringRef pointing at Data.
Depending on the CAS implementation, this may involve in-memory storage overhead.
Definition at line 190 of file ObjectStore.h.
References getData(), and toStringRef().
Get an ID for Ref.
References exportDataToFile(), llvm::Ref, and storeFromFile().
Referenced by llvm::cas::ObjectProxy::getReferenceID(), load(), and validateTree().
|
protected |
Get a MemoryBuffer pointing at Data.
The buffer may alias storage owned by this ObjectStore, in which case it is only valid for as long as the store is.
Definition at line 71 of file ObjectStore.cpp.
References getData(), llvm::MemoryBuffer::getMemBuffer(), and llvm::toStringRef().
|
protectedpure virtual |
Referenced by importObject().
| Expected< ObjectProxy > ObjectStore::getProxy | ( | const CASID & | ID | ) |
Create ObjectProxy from CASID. If the object doesn't exist, get an error.
Definition at line 99 of file ObjectStore.cpp.
References createUnknownObjectError(), getProxy(), getReference(), and llvm::Ref.
Referenced by createProxy(), getProxy(), and llvm::cas::NamedValuesSchema::load().
| Expected< ObjectProxy > ObjectStore::getProxy | ( | ObjectRef | Ref | ) |
Create ObjectProxy from ObjectRef.
If the object can't be loaded, get an error.
Definition at line 107 of file ObjectStore.cpp.
References H, llvm::cas::ObjectProxy::load(), load, and llvm::Ref.
| Expected< std::optional< ObjectProxy > > ObjectStore::getProxyIfExists | ( | ObjectRef | Ref | ) |
std::nullopt if the object is missing from the CAS. Definition at line 116 of file ObjectStore.cpp.
References H, llvm::cas::ObjectProxy::load(), loadIfExists(), and llvm::Ref.
|
pure virtual |
Get an existing reference to the object called ID.
Returns None if the object is not stored in this CAS.
Referenced by getProxy(), and llvm::cas::builtin::BuiltinCAS::validateObject().
|
protected |
Get a MemoryBuffer for Node that stays valid after this ObjectStore is destroyed.
May be more expensive than getMemoryBuffer(), which is free to alias storage the store already has mapped; prefer that one whenever the buffer cannot outlive the store. Never returns nullptr: copying the data always satisfies the lifetime requirement.
Definition at line 79 of file ObjectStore.cpp.
References getStandaloneMemoryBufferImpl().
|
protectedvirtual |
Customization point for getStandaloneMemoryBuffer().
The default implementation copies the data, which always satisfies the lifetime requirement; implementations that can hand out storage outliving themselves, e.g. a mapping of a file they do not keep open, should override this to avoid the copy. Must not return nullptr: fall back to ObjectStore::getStandaloneMemoryBufferImpl() where the cheaper path does not apply.
Definition at line 85 of file ObjectStore.cpp.
References getData(), llvm::MemoryBuffer::getMemBufferCopy(), and llvm::toStringRef().
Referenced by getData(), and getStandaloneMemoryBuffer().
|
inlinevirtual |
Implementations that don't have an implementation for this should return std::nullopt.
Definition at line 284 of file ObjectStore.h.
| Expected< ObjectRef > ObjectStore::importObject | ( | ObjectStore & | Upstream, |
| ObjectRef | Other ) |
Import object from another CAS.
This will import the full tree from the other CAS.
Keeps track of the state of visitation for current node and all of its parents. Upstream Cursor holds information only from upstream CAS.
PrimaryNodeStack holds the ObjectRef of the current CAS, with nodes either just stored in the CAS or nodes already exists in the current CAS.
A map from upstream ObjectRef to current ObjectRef.
Definition at line 216 of file ObjectStore.cpp.
References llvm::ArrayRef(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::SmallVectorTemplateCommon< T, typename >::front(), getData(), getNumRefs(), I, LLVM_UNLIKELY, load(), ObjectStore(), llvm::Other, llvm::SmallVectorTemplateBase< T, bool >::pop_back(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), readRef(), llvm::Ref, llvm::SmallVectorTemplateCommon< T, typename >::size(), store(), llvm::SmallVectorImpl< T >::truncate(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace().
References llvm::Ref.
|
protected |
Like loadIfExists but returns an error if the object is missing.
Definition at line 60 of file ObjectStore.cpp.
References llvm::createStringError(), getID(), llvm::invalid_argument, loadIfExists(), llvm::Ref, and llvm::toString().
Referenced by importObject().
|
protectedpure virtual |
Load the object referenced by Ref.
Errors if the object cannot be loaded.
std::nullopt if the object is missing from the CAS. References llvm::Ref.
Referenced by getProxyIfExists(), and load().
Get a CASID from a ID, which should have been generated by CASID::print().
This succeeds as long as validateID() would pass. The object may be unknown to this CAS instance.
TODO: Remove, and update callers to use validateID() or extractHashFromID().
Implemented in llvm::cas::builtin::BuiltinCAS.
|
inlinevirtual |
Print the ObjectStore internals for debugging purpose.
Definition at line 302 of file ObjectStore.h.
Referenced by dump().
|
inlinevirtual |
Prune local storage to reduce its size according to the desired size limit.
Pruning can happen concurrently with other operations.
Implementations may leave this unimplemented.
Definition at line 292 of file ObjectStore.h.
References llvm::Error::success().
|
inline |
Read the data from Data into OS.
Definition at line 263 of file ObjectStore.h.
References assert(), llvm::Data, getData(), llvm::Offset, and toStringRef().
|
protectedpure virtual |
References I.
Referenced by importObject().
|
protectedvirtual |
Read all the refs from object in a SmallVector.
Definition at line 91 of file ObjectStore.cpp.
References llvm::consumeError(), forEachRef(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, and llvm::Error::success().
|
inlinevirtual |
Set the size for limiting growth of on-disk storage.
This has an effect for when the instance is closed.
Implementations may leave this unimplemented.
Definition at line 276 of file ObjectStore.h.
References SizeLimit, and llvm::Error::success().
|
pure virtual |
Store object into ObjectStore.
Implemented in llvm::cas::builtin::BuiltinCAS.
References llvm::Data.
Referenced by createProxy(), importObject(), storeFromOpenFileImpl(), and storeFromString().
Stores the data of a file into ObjectStore.
An underlying implementation could perform optimizations that reduce I/O and disk space consumption.
If there are any concurrent modifications to the file, the contents in the CAS may be corrupt.
| FilePath | the path of the file data. |
Definition at line 151 of file ObjectStore.cpp.
References llvm::sys::fs::closeFile(), llvm::sys::fs::openNativeFileForRead(), llvm::scope_exit(), llvm::sys::sandbox::scopedDisable(), and storeFromOpenFile().
Referenced by getID().
|
inline |
Default implementation reads FD and calls storeNode().
Does not take ownership of FD; the caller is responsible for closing it.
If Status is sent in it is to be treated as a hint. Implementations must protect against the file size potentially growing after the status was taken (i.e., they cannot assume that an mmap will be null-terminated where Status implies).
Returns the CASID and the size of the file.
Definition at line 246 of file ObjectStore.h.
References storeFromOpenFileImpl().
Referenced by storeFromFile().
|
protectedvirtual |
Get ObjectRef from open file.
Definition at line 139 of file ObjectStore.cpp.
References llvm::arrayRefFromStringRef(), llvm::errorCodeToError(), llvm::MemoryBuffer::getOpenFile(), Size, and store().
Referenced by getData(), and storeFromOpenFile().
|
inline |
Store object from StringRef.
Definition at line 231 of file ObjectStore.h.
References llvm::arrayRefFromStringRef(), store(), and llvm::String.
Validate the entire ObjectStore.
Validate the underlying object referred by CASID.
Implemented in llvm::cas::builtin::BuiltinCAS.
Referenced by validateTree().
Validate the whole node tree.
Definition at line 192 of file ObjectStore.cpp.
References llvm::SmallVectorTemplateCommon< T, typename >::empty(), forEachRef(), getID(), I, llvm::detail::DenseSetImpl< ValueT, MapTy >::insert(), load, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, llvm::Error::success(), and validateObject().
|
friend |