clang
9.0.0
|
A list of tokens obtained by preprocessing a text buffer and operations to map between the expanded and spelled tokens, i.e. More...
#include "clang/Tooling/Syntax/Tokens.h"
Classes | |
struct | Expansion |
An expansion produced by the preprocessor, includes macro expansions and preprocessor directives. More... | |
Public Member Functions | |
TokenBuffer (const SourceManager &SourceMgr) | |
llvm::ArrayRef< syntax::Token > | expandedTokens () const |
All tokens produced by the preprocessor after all macro replacements, directives, etc. More... | |
llvm::Optional< llvm::ArrayRef< syntax::Token > > | spelledForExpanded (llvm::ArrayRef< syntax::Token > Expanded) const |
Find the subrange of spelled tokens that produced the corresponding Expanded tokens. More... | |
llvm::Optional< Expansion > | expansionStartingAt (const syntax::Token *Spelled) const |
If Spelled starts a mapping (e.g. More... | |
llvm::ArrayRef< syntax::Token > | spelledTokens (FileID FID) const |
Lexed tokens of a file before preprocessing. More... | |
const SourceManager & | sourceManager () const |
std::string | dumpForTests () const |
Friends | |
class | TokenCollector |
A list of tokens obtained by preprocessing a text buffer and operations to map between the expanded and spelled tokens, i.e.
TokenBuffer has information about two token streams:
Spelled tokens are {'#','define','FOO','10','int','a','=','FOO',';'}. Expanded tokens are {'int','a','=','10',';','eof'}.
Note that the expanded token stream has a tok::eof token at the end, the spelled tokens never store a 'eof' token.
The full list expanded tokens can be obtained with expandedTokens(). Spelled tokens for each of the files can be obtained via spelledTokens(FileID).
To map between the expanded and spelled tokens use findSpelledByExpanded().
To build a token buffer use the TokenCollector class. You can also compute the spelled tokens of a file using the tokenize() helper.
FIXME: allow to map from spelled to expanded tokens when use-case shows up. FIXME: allow mappings into macro arguments.
|
inline |
std::string TokenBuffer::dumpForTests | ( | ) | const |
Definition at line 554 of file Tokens.cpp.
References clang::frontend::DumpTokens, and clang::comments::tok::eof.
|
inline |
All tokens produced by the preprocessor after all macro replacements, directives, etc.
Source locations found in the clang AST will always point to one of these tokens. FIXME: figure out how to handle token splitting, e.g. '>>' can be split into two '>' tokens by the parser. However, TokenBuffer currently keeps it as a single '>>' token.
llvm::Optional< TokenBuffer::Expansion > TokenBuffer::expansionStartingAt | ( | const syntax::Token * | Spelled | ) | const |
If Spelled
starts a mapping (e.g.
if it's a macro name or '#' starting a preprocessor directive) return the subrange of expanded tokens that the macro expands to.
Definition at line 210 of file Tokens.cpp.
References clang::syntax::TokenBuffer::Expansion::Expanded, clang::SourceLocation::isFileID(), clang::syntax::Token::location(), clang::None, and clang::syntax::TokenBuffer::Expansion::Spelled.
|
inline |
llvm::Optional< llvm::ArrayRef< syntax::Token > > TokenBuffer::spelledForExpanded | ( | llvm::ArrayRef< syntax::Token > | Expanded | ) | const |
Find the subrange of spelled tokens that produced the corresponding Expanded
tokens.
EXPECTS: Expanded
is a subrange of expandedTokens().
Will fail if the expanded tokens do not correspond to a sequence of spelled tokens. E.g. for the following example:
#define FIRST f1 f2 f3 #define SECOND s1 s2 s3
a FIRST b SECOND c // expanded tokens are: a f1 f2 f3 b s1 s2 s3 c
the results would be:
a => a s1 s2 s3 => SECOND a f1 f2 f3 => a FIRST a f1 => can't map s1 s2 => can't map
If Expanded
is empty, the returned value is llvm::None. Complexity is logarithmic.
Definition at line 169 of file Tokens.cpp.
References clang::syntax::Token::location(), and clang::None.
llvm::ArrayRef< syntax::Token > TokenBuffer::spelledTokens | ( | FileID | FID | ) | const |
Lexed tokens of a file before preprocessing.
E.g. for the following input #define DECL(name) int name = 10 DECL(a); spelledTokens() returns {"#", "define", "DECL", "(", "name", ")", "eof"}. FIXME: we do not yet store tokens of directives, like #include, #define, #pragma, etc.
Definition at line 157 of file Tokens.cpp.
|
friend |