LLVM  9.0.0svn
MemorySanitizer.cpp
Go to the documentation of this file.
1 //===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2 //
3 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4 // See https://llvm.org/LICENSE.txt for license information.
5 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6 //
7 //===----------------------------------------------------------------------===//
8 //
9 /// \file
10 /// This file is a part of MemorySanitizer, a detector of uninitialized
11 /// reads.
12 ///
13 /// The algorithm of the tool is similar to Memcheck
14 /// (http://goo.gl/QKbem). We associate a few shadow bits with every
15 /// byte of the application memory, poison the shadow of the malloc-ed
16 /// or alloca-ed memory, load the shadow bits on every memory read,
17 /// propagate the shadow bits through some of the arithmetic
18 /// instruction (including MOV), store the shadow bits on every memory
19 /// write, report a bug on some other instructions (e.g. JMP) if the
20 /// associated shadow is poisoned.
21 ///
22 /// But there are differences too. The first and the major one:
23 /// compiler instrumentation instead of binary instrumentation. This
24 /// gives us much better register allocation, possible compiler
25 /// optimizations and a fast start-up. But this brings the major issue
26 /// as well: msan needs to see all program events, including system
27 /// calls and reads/writes in system libraries, so we either need to
28 /// compile *everything* with msan or use a binary translation
29 /// component (e.g. DynamoRIO) to instrument pre-built libraries.
30 /// Another difference from Memcheck is that we use 8 shadow bits per
31 /// byte of application memory and use a direct shadow mapping. This
32 /// greatly simplifies the instrumentation code and avoids races on
33 /// shadow updates (Memcheck is single-threaded so races are not a
34 /// concern there. Memcheck uses 2 shadow bits per byte with a slow
35 /// path storage that uses 8 bits per byte).
36 ///
37 /// The default value of shadow is 0, which means "clean" (not poisoned).
38 ///
39 /// Every module initializer should call __msan_init to ensure that the
40 /// shadow memory is ready. On error, __msan_warning is called. Since
41 /// parameters and return values may be passed via registers, we have a
42 /// specialized thread-local shadow for return values
43 /// (__msan_retval_tls) and parameters (__msan_param_tls).
44 ///
45 /// Origin tracking.
46 ///
47 /// MemorySanitizer can track origins (allocation points) of all uninitialized
48 /// values. This behavior is controlled with a flag (msan-track-origins) and is
49 /// disabled by default.
50 ///
51 /// Origins are 4-byte values created and interpreted by the runtime library.
52 /// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
53 /// of application memory. Propagation of origins is basically a bunch of
54 /// "select" instructions that pick the origin of a dirty argument, if an
55 /// instruction has one.
56 ///
57 /// Every 4 aligned, consecutive bytes of application memory have one origin
58 /// value associated with them. If these bytes contain uninitialized data
59 /// coming from 2 different allocations, the last store wins. Because of this,
60 /// MemorySanitizer reports can show unrelated origins, but this is unlikely in
61 /// practice.
62 ///
63 /// Origins are meaningless for fully initialized values, so MemorySanitizer
64 /// avoids storing origin to memory when a fully initialized value is stored.
65 /// This way it avoids needless overwritting origin of the 4-byte region on
66 /// a short (i.e. 1 byte) clean store, and it is also good for performance.
67 ///
68 /// Atomic handling.
69 ///
70 /// Ideally, every atomic store of application value should update the
71 /// corresponding shadow location in an atomic way. Unfortunately, atomic store
72 /// of two disjoint locations can not be done without severe slowdown.
73 ///
74 /// Therefore, we implement an approximation that may err on the safe side.
75 /// In this implementation, every atomically accessed location in the program
76 /// may only change from (partially) uninitialized to fully initialized, but
77 /// not the other way around. We load the shadow _after_ the application load,
78 /// and we store the shadow _before_ the app store. Also, we always store clean
79 /// shadow (if the application store is atomic). This way, if the store-load
80 /// pair constitutes a happens-before arc, shadow store and load are correctly
81 /// ordered such that the load will get either the value that was stored, or
82 /// some later value (which is always clean).
83 ///
84 /// This does not work very well with Compare-And-Swap (CAS) and
85 /// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
86 /// must store the new shadow before the app operation, and load the shadow
87 /// after the app operation. Computers don't work this way. Current
88 /// implementation ignores the load aspect of CAS/RMW, always returning a clean
89 /// value. It implements the store part as a simple atomic store by storing a
90 /// clean shadow.
91 ///
92 /// Instrumenting inline assembly.
93 ///
94 /// For inline assembly code LLVM has little idea about which memory locations
95 /// become initialized depending on the arguments. It can be possible to figure
96 /// out which arguments are meant to point to inputs and outputs, but the
97 /// actual semantics can be only visible at runtime. In the Linux kernel it's
98 /// also possible that the arguments only indicate the offset for a base taken
99 /// from a segment register, so it's dangerous to treat any asm() arguments as
100 /// pointers. We take a conservative approach generating calls to
101 /// __msan_instrument_asm_store(ptr, size)
102 /// , which defer the memory unpoisoning to the runtime library.
103 /// The latter can perform more complex address checks to figure out whether
104 /// it's safe to touch the shadow memory.
105 /// Like with atomic operations, we call __msan_instrument_asm_store() before
106 /// the assembly call, so that changes to the shadow memory will be seen by
107 /// other threads together with main memory initialization.
108 ///
109 /// KernelMemorySanitizer (KMSAN) implementation.
110 ///
111 /// The major differences between KMSAN and MSan instrumentation are:
112 /// - KMSAN always tracks the origins and implies msan-keep-going=true;
113 /// - KMSAN allocates shadow and origin memory for each page separately, so
114 /// there are no explicit accesses to shadow and origin in the
115 /// instrumentation.
116 /// Shadow and origin values for a particular X-byte memory location
117 /// (X=1,2,4,8) are accessed through pointers obtained via the
118 /// __msan_metadata_ptr_for_load_X(ptr)
119 /// __msan_metadata_ptr_for_store_X(ptr)
120 /// functions. The corresponding functions check that the X-byte accesses
121 /// are possible and returns the pointers to shadow and origin memory.
122 /// Arbitrary sized accesses are handled with:
123 /// __msan_metadata_ptr_for_load_n(ptr, size)
124 /// __msan_metadata_ptr_for_store_n(ptr, size);
125 /// - TLS variables are stored in a single per-task struct. A call to a
126 /// function __msan_get_context_state() returning a pointer to that struct
127 /// is inserted into every instrumented function before the entry block;
128 /// - __msan_warning() takes a 32-bit origin parameter;
129 /// - local variables are poisoned with __msan_poison_alloca() upon function
130 /// entry and unpoisoned with __msan_unpoison_alloca() before leaving the
131 /// function;
132 /// - the pass doesn't declare any global variables or add global constructors
133 /// to the translation unit.
134 ///
135 /// Also, KMSAN currently ignores uninitialized memory passed into inline asm
136 /// calls, making sure we're on the safe side wrt. possible false positives.
137 ///
138 /// KernelMemorySanitizer only supports X86_64 at the moment.
139 ///
140 //===----------------------------------------------------------------------===//
141 
143 #include "llvm/ADT/APInt.h"
144 #include "llvm/ADT/ArrayRef.h"
146 #include "llvm/ADT/SmallString.h"
147 #include "llvm/ADT/SmallVector.h"
148 #include "llvm/ADT/StringExtras.h"
149 #include "llvm/ADT/StringRef.h"
150 #include "llvm/ADT/Triple.h"
152 #include "llvm/IR/Argument.h"
153 #include "llvm/IR/Attributes.h"
154 #include "llvm/IR/BasicBlock.h"
155 #include "llvm/IR/CallSite.h"
156 #include "llvm/IR/CallingConv.h"
157 #include "llvm/IR/Constant.h"
158 #include "llvm/IR/Constants.h"
159 #include "llvm/IR/DataLayout.h"
160 #include "llvm/IR/DerivedTypes.h"
161 #include "llvm/IR/Function.h"
162 #include "llvm/IR/GlobalValue.h"
163 #include "llvm/IR/GlobalVariable.h"
164 #include "llvm/IR/IRBuilder.h"
165 #include "llvm/IR/InlineAsm.h"
166 #include "llvm/IR/InstVisitor.h"
167 #include "llvm/IR/InstrTypes.h"
168 #include "llvm/IR/Instruction.h"
169 #include "llvm/IR/Instructions.h"
170 #include "llvm/IR/IntrinsicInst.h"
171 #include "llvm/IR/Intrinsics.h"
172 #include "llvm/IR/LLVMContext.h"
173 #include "llvm/IR/MDBuilder.h"
174 #include "llvm/IR/Module.h"
175 #include "llvm/IR/Type.h"
176 #include "llvm/IR/Value.h"
177 #include "llvm/IR/ValueMap.h"
178 #include "llvm/Pass.h"
180 #include "llvm/Support/Casting.h"
182 #include "llvm/Support/Compiler.h"
183 #include "llvm/Support/Debug.h"
185 #include "llvm/Support/MathExtras.h"
191 #include <algorithm>
192 #include <cassert>
193 #include <cstddef>
194 #include <cstdint>
195 #include <memory>
196 #include <string>
197 #include <tuple>
198 
199 using namespace llvm;
200 
201 #define DEBUG_TYPE "msan"
202 
203 static const unsigned kOriginSize = 4;
204 static const unsigned kMinOriginAlignment = 4;
205 static const unsigned kShadowTLSAlignment = 8;
206 
207 // These constants must be kept in sync with the ones in msan.h.
208 static const unsigned kParamTLSSize = 800;
209 static const unsigned kRetvalTLSSize = 800;
210 
211 // Accesses sizes are powers of two: 1, 2, 4, 8.
212 static const size_t kNumberOfAccessSizes = 4;
213 
214 /// Track origins of uninitialized values.
215 ///
216 /// Adds a section to MemorySanitizer report that points to the allocation
217 /// (stack or heap) the uninitialized bits came from originally.
218 static cl::opt<int> ClTrackOrigins("msan-track-origins",
219  cl::desc("Track origins (allocation sites) of poisoned memory"),
220  cl::Hidden, cl::init(0));
221 
222 static cl::opt<bool> ClKeepGoing("msan-keep-going",
223  cl::desc("keep going after reporting a UMR"),
224  cl::Hidden, cl::init(false));
225 
226 static cl::opt<bool> ClPoisonStack("msan-poison-stack",
227  cl::desc("poison uninitialized stack variables"),
228  cl::Hidden, cl::init(true));
229 
230 static cl::opt<bool> ClPoisonStackWithCall("msan-poison-stack-with-call",
231  cl::desc("poison uninitialized stack variables with a call"),
232  cl::Hidden, cl::init(false));
233 
234 static cl::opt<int> ClPoisonStackPattern("msan-poison-stack-pattern",
235  cl::desc("poison uninitialized stack variables with the given pattern"),
236  cl::Hidden, cl::init(0xff));
237 
238 static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
239  cl::desc("poison undef temps"),
240  cl::Hidden, cl::init(true));
241 
242 static cl::opt<bool> ClHandleICmp("msan-handle-icmp",
243  cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
244  cl::Hidden, cl::init(true));
245 
246 static cl::opt<bool> ClHandleICmpExact("msan-handle-icmp-exact",
247  cl::desc("exact handling of relational integer ICmp"),
248  cl::Hidden, cl::init(false));
249 
250 // When compiling the Linux kernel, we sometimes see false positives related to
251 // MSan being unable to understand that inline assembly calls may initialize
252 // local variables.
253 // This flag makes the compiler conservatively unpoison every memory location
254 // passed into an assembly call. Note that this may cause false positives.
255 // Because it's impossible to figure out the array sizes, we can only unpoison
256 // the first sizeof(type) bytes for each type* pointer.
257 // The instrumentation is only enabled in KMSAN builds, and only if
258 // -msan-handle-asm-conservative is on. This is done because we may want to
259 // quickly disable assembly instrumentation when it breaks.
261  "msan-handle-asm-conservative",
262  cl::desc("conservative handling of inline assembly"), cl::Hidden,
263  cl::init(true));
264 
265 // This flag controls whether we check the shadow of the address
266 // operand of load or store. Such bugs are very rare, since load from
267 // a garbage address typically results in SEGV, but still happen
268 // (e.g. only lower bits of address are garbage, or the access happens
269 // early at program startup where malloc-ed memory is more likely to
270 // be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
271 static cl::opt<bool> ClCheckAccessAddress("msan-check-access-address",
272  cl::desc("report accesses through a pointer which has poisoned shadow"),
273  cl::Hidden, cl::init(true));
274 
275 static cl::opt<bool> ClDumpStrictInstructions("msan-dump-strict-instructions",
276  cl::desc("print out instructions with default strict semantics"),
277  cl::Hidden, cl::init(false));
278 
280  "msan-instrumentation-with-call-threshold",
281  cl::desc(
282  "If the function being instrumented requires more than "
283  "this number of checks and origin stores, use callbacks instead of "
284  "inline checks (-1 means never use callbacks)."),
285  cl::Hidden, cl::init(3500));
286 
287 static cl::opt<bool>
288  ClEnableKmsan("msan-kernel",
289  cl::desc("Enable KernelMemorySanitizer instrumentation"),
290  cl::Hidden, cl::init(false));
291 
292 // This is an experiment to enable handling of cases where shadow is a non-zero
293 // compile-time constant. For some unexplainable reason they were silently
294 // ignored in the instrumentation.
295 static cl::opt<bool> ClCheckConstantShadow("msan-check-constant-shadow",
296  cl::desc("Insert checks for constant shadow values"),
297  cl::Hidden, cl::init(false));
298 
299 // This is off by default because of a bug in gold:
300 // https://sourceware.org/bugzilla/show_bug.cgi?id=19002
301 static cl::opt<bool> ClWithComdat("msan-with-comdat",
302  cl::desc("Place MSan constructors in comdat sections"),
303  cl::Hidden, cl::init(false));
304 
305 // These options allow to specify custom memory map parameters
306 // See MemoryMapParams for details.
307 static cl::opt<unsigned long long> ClAndMask("msan-and-mask",
308  cl::desc("Define custom MSan AndMask"),
309  cl::Hidden, cl::init(0));
310 
311 static cl::opt<unsigned long long> ClXorMask("msan-xor-mask",
312  cl::desc("Define custom MSan XorMask"),
313  cl::Hidden, cl::init(0));
314 
315 static cl::opt<unsigned long long> ClShadowBase("msan-shadow-base",
316  cl::desc("Define custom MSan ShadowBase"),
317  cl::Hidden, cl::init(0));
318 
319 static cl::opt<unsigned long long> ClOriginBase("msan-origin-base",
320  cl::desc("Define custom MSan OriginBase"),
321  cl::Hidden, cl::init(0));
322 
323 static const char *const kMsanModuleCtorName = "msan.module_ctor";
324 static const char *const kMsanInitName = "__msan_init";
325 
326 namespace {
327 
328 // Memory map parameters used in application-to-shadow address calculation.
329 // Offset = (Addr & ~AndMask) ^ XorMask
330 // Shadow = ShadowBase + Offset
331 // Origin = OriginBase + Offset
332 struct MemoryMapParams {
333  uint64_t AndMask;
334  uint64_t XorMask;
335  uint64_t ShadowBase;
336  uint64_t OriginBase;
337 };
338 
339 struct PlatformMemoryMapParams {
340  const MemoryMapParams *bits32;
341  const MemoryMapParams *bits64;
342 };
343 
344 } // end anonymous namespace
345 
346 // i386 Linux
347 static const MemoryMapParams Linux_I386_MemoryMapParams = {
348  0x000080000000, // AndMask
349  0, // XorMask (not used)
350  0, // ShadowBase (not used)
351  0x000040000000, // OriginBase
352 };
353 
354 // x86_64 Linux
355 static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
356 #ifdef MSAN_LINUX_X86_64_OLD_MAPPING
357  0x400000000000, // AndMask
358  0, // XorMask (not used)
359  0, // ShadowBase (not used)
360  0x200000000000, // OriginBase
361 #else
362  0, // AndMask (not used)
363  0x500000000000, // XorMask
364  0, // ShadowBase (not used)
365  0x100000000000, // OriginBase
366 #endif
367 };
368 
369 // mips64 Linux
370 static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
371  0, // AndMask (not used)
372  0x008000000000, // XorMask
373  0, // ShadowBase (not used)
374  0x002000000000, // OriginBase
375 };
376 
377 // ppc64 Linux
378 static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
379  0xE00000000000, // AndMask
380  0x100000000000, // XorMask
381  0x080000000000, // ShadowBase
382  0x1C0000000000, // OriginBase
383 };
384 
385 // aarch64 Linux
386 static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
387  0, // AndMask (not used)
388  0x06000000000, // XorMask
389  0, // ShadowBase (not used)
390  0x01000000000, // OriginBase
391 };
392 
393 // i386 FreeBSD
394 static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
395  0x000180000000, // AndMask
396  0x000040000000, // XorMask
397  0x000020000000, // ShadowBase
398  0x000700000000, // OriginBase
399 };
400 
401 // x86_64 FreeBSD
402 static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
403  0xc00000000000, // AndMask
404  0x200000000000, // XorMask
405  0x100000000000, // ShadowBase
406  0x380000000000, // OriginBase
407 };
408 
409 // x86_64 NetBSD
410 static const MemoryMapParams NetBSD_X86_64_MemoryMapParams = {
411  0, // AndMask
412  0x500000000000, // XorMask
413  0, // ShadowBase
414  0x100000000000, // OriginBase
415 };
416 
417 static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
420 };
421 
422 static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
423  nullptr,
425 };
426 
427 static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
428  nullptr,
430 };
431 
432 static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
433  nullptr,
435 };
436 
437 static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
440 };
441 
442 static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams = {
443  nullptr,
445 };
446 
447 namespace {
448 
449 /// Instrument functions of a module to detect uninitialized reads.
450 ///
451 /// Instantiating MemorySanitizer inserts the msan runtime library API function
452 /// declarations into the module if they don't exist already. Instantiating
453 /// ensures the __msan_init function is in the list of global constructors for
454 /// the module.
455 class MemorySanitizer {
456 public:
457  MemorySanitizer(Module &M, int TrackOrigins = 0, bool Recover = false,
458  bool EnableKmsan = false) {
459  this->CompileKernel =
460  ClEnableKmsan.getNumOccurrences() > 0 ? ClEnableKmsan : EnableKmsan;
461  if (ClTrackOrigins.getNumOccurrences() > 0)
462  this->TrackOrigins = ClTrackOrigins;
463  else
464  this->TrackOrigins = this->CompileKernel ? 2 : TrackOrigins;
465  this->Recover = ClKeepGoing.getNumOccurrences() > 0
466  ? ClKeepGoing
467  : (this->CompileKernel | Recover);
468  initializeModule(M);
469  }
470 
471  // MSan cannot be moved or copied because of MapParams.
472  MemorySanitizer(MemorySanitizer &&) = delete;
473  MemorySanitizer &operator=(MemorySanitizer &&) = delete;
474  MemorySanitizer(const MemorySanitizer &) = delete;
475  MemorySanitizer &operator=(const MemorySanitizer &) = delete;
476 
477  bool sanitizeFunction(Function &F, TargetLibraryInfo &TLI);
478 
479 private:
480  friend struct MemorySanitizerVisitor;
481  friend struct VarArgAMD64Helper;
482  friend struct VarArgMIPS64Helper;
483  friend struct VarArgAArch64Helper;
484  friend struct VarArgPowerPC64Helper;
485 
486  void initializeModule(Module &M);
487  void initializeCallbacks(Module &M);
488  void createKernelApi(Module &M);
489  void createUserspaceApi(Module &M);
490 
491  /// True if we're compiling the Linux kernel.
492  bool CompileKernel;
493  /// Track origins (allocation points) of uninitialized values.
494  int TrackOrigins;
495  bool Recover;
496 
497  LLVMContext *C;
498  Type *IntptrTy;
499  Type *OriginTy;
500 
501  // XxxTLS variables represent the per-thread state in MSan and per-task state
502  // in KMSAN.
503  // For the userspace these point to thread-local globals. In the kernel land
504  // they point to the members of a per-task struct obtained via a call to
505  // __msan_get_context_state().
506 
507  /// Thread-local shadow storage for function parameters.
508  Value *ParamTLS;
509 
510  /// Thread-local origin storage for function parameters.
511  Value *ParamOriginTLS;
512 
513  /// Thread-local shadow storage for function return value.
514  Value *RetvalTLS;
515 
516  /// Thread-local origin storage for function return value.
517  Value *RetvalOriginTLS;
518 
519  /// Thread-local shadow storage for in-register va_arg function
520  /// parameters (x86_64-specific).
521  Value *VAArgTLS;
522 
523  /// Thread-local shadow storage for in-register va_arg function
524  /// parameters (x86_64-specific).
525  Value *VAArgOriginTLS;
526 
527  /// Thread-local shadow storage for va_arg overflow area
528  /// (x86_64-specific).
529  Value *VAArgOverflowSizeTLS;
530 
531  /// Thread-local space used to pass origin value to the UMR reporting
532  /// function.
533  Value *OriginTLS;
534 
535  /// Are the instrumentation callbacks set up?
536  bool CallbacksInitialized = false;
537 
538  /// The run-time callback to print a warning.
539  Value *WarningFn;
540 
541  // These arrays are indexed by log2(AccessSize).
542  Value *MaybeWarningFn[kNumberOfAccessSizes];
543  Value *MaybeStoreOriginFn[kNumberOfAccessSizes];
544 
545  /// Run-time helper that generates a new origin value for a stack
546  /// allocation.
547  Value *MsanSetAllocaOrigin4Fn;
548 
549  /// Run-time helper that poisons stack on function entry.
550  Value *MsanPoisonStackFn;
551 
552  /// Run-time helper that records a store (or any event) of an
553  /// uninitialized value and returns an updated origin id encoding this info.
554  Value *MsanChainOriginFn;
555 
556  /// MSan runtime replacements for memmove, memcpy and memset.
557  Value *MemmoveFn, *MemcpyFn, *MemsetFn;
558 
559  /// KMSAN callback for task-local function argument shadow.
560  Value *MsanGetContextStateFn;
561 
562  /// Functions for poisoning/unpoisoning local variables
563  Value *MsanPoisonAllocaFn, *MsanUnpoisonAllocaFn;
564 
565  /// Each of the MsanMetadataPtrXxx functions returns a pair of shadow/origin
566  /// pointers.
567  Value *MsanMetadataPtrForLoadN, *MsanMetadataPtrForStoreN;
568  Value *MsanMetadataPtrForLoad_1_8[4];
569  Value *MsanMetadataPtrForStore_1_8[4];
570  Value *MsanInstrumentAsmStoreFn;
571 
572  /// Helper to choose between different MsanMetadataPtrXxx().
573  Value *getKmsanShadowOriginAccessFn(bool isStore, int size);
574 
575  /// Memory map parameters used in application-to-shadow calculation.
576  const MemoryMapParams *MapParams;
577 
578  /// Custom memory map parameters used when -msan-shadow-base or
579  // -msan-origin-base is provided.
580  MemoryMapParams CustomMapParams;
581 
582  MDNode *ColdCallWeights;
583 
584  /// Branch weights for origin store.
585  MDNode *OriginStoreWeights;
586 
587  /// An empty volatile inline asm that prevents callback merge.
588  InlineAsm *EmptyAsm;
589 
590  Function *MsanCtorFunction;
591 };
592 
593 /// A legacy function pass for msan instrumentation.
594 ///
595 /// Instruments functions to detect unitialized reads.
596 struct MemorySanitizerLegacyPass : public FunctionPass {
597  // Pass identification, replacement for typeid.
598  static char ID;
599 
600  MemorySanitizerLegacyPass(int TrackOrigins = 0, bool Recover = false,
601  bool EnableKmsan = false)
602  : FunctionPass(ID), TrackOrigins(TrackOrigins), Recover(Recover),
603  EnableKmsan(EnableKmsan) {}
604  StringRef getPassName() const override { return "MemorySanitizerLegacyPass"; }
605 
606  void getAnalysisUsage(AnalysisUsage &AU) const override {
608  }
609 
610  bool runOnFunction(Function &F) override {
611  return MSan->sanitizeFunction(
612  F, getAnalysis<TargetLibraryInfoWrapperPass>().getTLI());
613  }
614  bool doInitialization(Module &M) override;
615 
617  int TrackOrigins;
618  bool Recover;
619  bool EnableKmsan;
620 };
621 
622 } // end anonymous namespace
623 
626  MemorySanitizer Msan(*F.getParent(), TrackOrigins, Recover, EnableKmsan);
627  if (Msan.sanitizeFunction(F, FAM.getResult<TargetLibraryAnalysis>(F)))
628  return PreservedAnalyses::none();
629  return PreservedAnalyses::all();
630 }
631 
633 
634 INITIALIZE_PASS_BEGIN(MemorySanitizerLegacyPass, "msan",
635  "MemorySanitizer: detects uninitialized reads.", false,
636  false)
638 INITIALIZE_PASS_END(MemorySanitizerLegacyPass, "msan",
639  "MemorySanitizer: detects uninitialized reads.", false,
640  false)
641 
643  bool Recover,
644  bool CompileKernel) {
645  return new MemorySanitizerLegacyPass(TrackOrigins, Recover, CompileKernel);
646 }
647 
648 /// Create a non-const global initialized with the given string.
649 ///
650 /// Creates a writable global for Str so that we can pass it to the
651 /// run-time lib. Runtime uses first 4 bytes of the string to store the
652 /// frame ID, so the string needs to be mutable.
654  StringRef Str) {
655  Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
656  return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/false,
657  GlobalValue::PrivateLinkage, StrConst, "");
658 }
659 
660 /// Create KMSAN API callbacks.
661 void MemorySanitizer::createKernelApi(Module &M) {
662  IRBuilder<> IRB(*C);
663 
664  // These will be initialized in insertKmsanPrologue().
665  RetvalTLS = nullptr;
666  RetvalOriginTLS = nullptr;
667  ParamTLS = nullptr;
668  ParamOriginTLS = nullptr;
669  VAArgTLS = nullptr;
670  VAArgOriginTLS = nullptr;
671  VAArgOverflowSizeTLS = nullptr;
672  // OriginTLS is unused in the kernel.
673  OriginTLS = nullptr;
674 
675  // __msan_warning() in the kernel takes an origin.
676  WarningFn = M.getOrInsertFunction("__msan_warning", IRB.getVoidTy(),
677  IRB.getInt32Ty());
678  // Requests the per-task context state (kmsan_context_state*) from the
679  // runtime library.
680  MsanGetContextStateFn = M.getOrInsertFunction(
681  "__msan_get_context_state",
683  StructType::get(ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
684  ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8),
685  ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
687  kParamTLSSize / 8), /* va_arg_origin */
688  IRB.getInt64Ty(),
689  ArrayType::get(OriginTy, kParamTLSSize / 4), OriginTy,
690  OriginTy),
691  0));
692 
693  Type *RetTy = StructType::get(PointerType::get(IRB.getInt8Ty(), 0),
694  PointerType::get(IRB.getInt32Ty(), 0));
695 
696  for (int ind = 0, size = 1; ind < 4; ind++, size <<= 1) {
697  std::string name_load =
698  "__msan_metadata_ptr_for_load_" + std::to_string(size);
699  std::string name_store =
700  "__msan_metadata_ptr_for_store_" + std::to_string(size);
701  MsanMetadataPtrForLoad_1_8[ind] = M.getOrInsertFunction(
702  name_load, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
703  MsanMetadataPtrForStore_1_8[ind] = M.getOrInsertFunction(
704  name_store, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
705  }
706 
707  MsanMetadataPtrForLoadN = M.getOrInsertFunction(
708  "__msan_metadata_ptr_for_load_n", RetTy,
709  PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
710  MsanMetadataPtrForStoreN = M.getOrInsertFunction(
711  "__msan_metadata_ptr_for_store_n", RetTy,
712  PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
713 
714  // Functions for poisoning and unpoisoning memory.
715  MsanPoisonAllocaFn =
716  M.getOrInsertFunction("__msan_poison_alloca", IRB.getVoidTy(),
717  IRB.getInt8PtrTy(), IntptrTy, IRB.getInt8PtrTy());
718  MsanUnpoisonAllocaFn = M.getOrInsertFunction(
719  "__msan_unpoison_alloca", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy);
720 }
721 
723  return M.getOrInsertGlobal(Name, Ty, [&] {
724  return new GlobalVariable(M, Ty, false, GlobalVariable::ExternalLinkage,
725  nullptr, Name, nullptr,
727  });
728 }
729 
730 /// Insert declarations for userspace-specific functions and globals.
731 void MemorySanitizer::createUserspaceApi(Module &M) {
732  IRBuilder<> IRB(*C);
733  // Create the callback.
734  // FIXME: this function should have "Cold" calling conv,
735  // which is not yet implemented.
736  StringRef WarningFnName = Recover ? "__msan_warning"
737  : "__msan_warning_noreturn";
738  WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy());
739 
740  // Create the global TLS variables.
741  RetvalTLS =
742  getOrInsertGlobal(M, "__msan_retval_tls",
743  ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8));
744 
745  RetvalOriginTLS = getOrInsertGlobal(M, "__msan_retval_origin_tls", OriginTy);
746 
747  ParamTLS =
748  getOrInsertGlobal(M, "__msan_param_tls",
749  ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
750 
751  ParamOriginTLS =
752  getOrInsertGlobal(M, "__msan_param_origin_tls",
753  ArrayType::get(OriginTy, kParamTLSSize / 4));
754 
755  VAArgTLS =
756  getOrInsertGlobal(M, "__msan_va_arg_tls",
757  ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
758 
759  VAArgOriginTLS =
760  getOrInsertGlobal(M, "__msan_va_arg_origin_tls",
761  ArrayType::get(OriginTy, kParamTLSSize / 4));
762 
763  VAArgOverflowSizeTLS =
764  getOrInsertGlobal(M, "__msan_va_arg_overflow_size_tls", IRB.getInt64Ty());
765  OriginTLS = getOrInsertGlobal(M, "__msan_origin_tls", IRB.getInt32Ty());
766 
767  for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
768  AccessSizeIndex++) {
769  unsigned AccessSize = 1 << AccessSizeIndex;
770  std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
771  MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
772  FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8),
773  IRB.getInt32Ty());
774 
775  FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
776  MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
777  FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8),
778  IRB.getInt8PtrTy(), IRB.getInt32Ty());
779  }
780 
781  MsanSetAllocaOrigin4Fn = M.getOrInsertFunction(
782  "__msan_set_alloca_origin4", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy,
783  IRB.getInt8PtrTy(), IntptrTy);
784  MsanPoisonStackFn =
785  M.getOrInsertFunction("__msan_poison_stack", IRB.getVoidTy(),
786  IRB.getInt8PtrTy(), IntptrTy);
787 }
788 
789 /// Insert extern declaration of runtime-provided functions and globals.
790 void MemorySanitizer::initializeCallbacks(Module &M) {
791  // Only do this once.
792  if (CallbacksInitialized)
793  return;
794 
795  IRBuilder<> IRB(*C);
796  // Initialize callbacks that are common for kernel and userspace
797  // instrumentation.
798  MsanChainOriginFn = M.getOrInsertFunction(
799  "__msan_chain_origin", IRB.getInt32Ty(), IRB.getInt32Ty());
800  MemmoveFn = M.getOrInsertFunction(
801  "__msan_memmove", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
802  IRB.getInt8PtrTy(), IntptrTy);
803  MemcpyFn = M.getOrInsertFunction(
804  "__msan_memcpy", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
805  IntptrTy);
806  MemsetFn = M.getOrInsertFunction(
807  "__msan_memset", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(),
808  IntptrTy);
809  // We insert an empty inline asm after __msan_report* to avoid callback merge.
810  EmptyAsm = InlineAsm::get(FunctionType::get(IRB.getVoidTy(), false),
811  StringRef(""), StringRef(""),
812  /*hasSideEffects=*/true);
813 
814  MsanInstrumentAsmStoreFn =
815  M.getOrInsertFunction("__msan_instrument_asm_store", IRB.getVoidTy(),
816  PointerType::get(IRB.getInt8Ty(), 0), IntptrTy);
817 
818  if (CompileKernel) {
819  createKernelApi(M);
820  } else {
821  createUserspaceApi(M);
822  }
823  CallbacksInitialized = true;
824 }
825 
826 Value *MemorySanitizer::getKmsanShadowOriginAccessFn(bool isStore, int size) {
827  Value **Fns =
828  isStore ? MsanMetadataPtrForStore_1_8 : MsanMetadataPtrForLoad_1_8;
829  switch (size) {
830  case 1:
831  return Fns[0];
832  case 2:
833  return Fns[1];
834  case 4:
835  return Fns[2];
836  case 8:
837  return Fns[3];
838  default:
839  return nullptr;
840  }
841 }
842 
843 /// Module-level initialization.
844 ///
845 /// inserts a call to __msan_init to the module's constructor list.
846 void MemorySanitizer::initializeModule(Module &M) {
847  auto &DL = M.getDataLayout();
848 
849  bool ShadowPassed = ClShadowBase.getNumOccurrences() > 0;
850  bool OriginPassed = ClOriginBase.getNumOccurrences() > 0;
851  // Check the overrides first
852  if (ShadowPassed || OriginPassed) {
853  CustomMapParams.AndMask = ClAndMask;
854  CustomMapParams.XorMask = ClXorMask;
855  CustomMapParams.ShadowBase = ClShadowBase;
856  CustomMapParams.OriginBase = ClOriginBase;
857  MapParams = &CustomMapParams;
858  } else {
859  Triple TargetTriple(M.getTargetTriple());
860  switch (TargetTriple.getOS()) {
861  case Triple::FreeBSD:
862  switch (TargetTriple.getArch()) {
863  case Triple::x86_64:
864  MapParams = FreeBSD_X86_MemoryMapParams.bits64;
865  break;
866  case Triple::x86:
867  MapParams = FreeBSD_X86_MemoryMapParams.bits32;
868  break;
869  default:
870  report_fatal_error("unsupported architecture");
871  }
872  break;
873  case Triple::NetBSD:
874  switch (TargetTriple.getArch()) {
875  case Triple::x86_64:
876  MapParams = NetBSD_X86_MemoryMapParams.bits64;
877  break;
878  default:
879  report_fatal_error("unsupported architecture");
880  }
881  break;
882  case Triple::Linux:
883  switch (TargetTriple.getArch()) {
884  case Triple::x86_64:
885  MapParams = Linux_X86_MemoryMapParams.bits64;
886  break;
887  case Triple::x86:
888  MapParams = Linux_X86_MemoryMapParams.bits32;
889  break;
890  case Triple::mips64:
891  case Triple::mips64el:
892  MapParams = Linux_MIPS_MemoryMapParams.bits64;
893  break;
894  case Triple::ppc64:
895  case Triple::ppc64le:
896  MapParams = Linux_PowerPC_MemoryMapParams.bits64;
897  break;
898  case Triple::aarch64:
899  case Triple::aarch64_be:
900  MapParams = Linux_ARM_MemoryMapParams.bits64;
901  break;
902  default:
903  report_fatal_error("unsupported architecture");
904  }
905  break;
906  default:
907  report_fatal_error("unsupported operating system");
908  }
909  }
910 
911  C = &(M.getContext());
912  IRBuilder<> IRB(*C);
913  IntptrTy = IRB.getIntPtrTy(DL);
914  OriginTy = IRB.getInt32Ty();
915 
916  ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000);
917  OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000);
918 
919  if (!CompileKernel) {
920  std::tie(MsanCtorFunction, std::ignore) =
923  /*InitArgTypes=*/{},
924  /*InitArgs=*/{},
925  // This callback is invoked when the functions are created the first
926  // time. Hook them into the global ctors list in that case:
927  [&](Function *Ctor, Function *) {
928  if (!ClWithComdat) {
929  appendToGlobalCtors(M, Ctor, 0);
930  return;
931  }
932  Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
933  Ctor->setComdat(MsanCtorComdat);
934  appendToGlobalCtors(M, Ctor, 0, Ctor);
935  });
936 
937  if (TrackOrigins)
938  M.getOrInsertGlobal("__msan_track_origins", IRB.getInt32Ty(), [&] {
939  return new GlobalVariable(
940  M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage,
941  IRB.getInt32(TrackOrigins), "__msan_track_origins");
942  });
943 
944  if (Recover)
945  M.getOrInsertGlobal("__msan_keep_going", IRB.getInt32Ty(), [&] {
946  return new GlobalVariable(M, IRB.getInt32Ty(), true,
948  IRB.getInt32(Recover), "__msan_keep_going");
949  });
950 }
951 }
952 
953 bool MemorySanitizerLegacyPass::doInitialization(Module &M) {
954  MSan.emplace(M, TrackOrigins, Recover, EnableKmsan);
955  return true;
956 }
957 
958 namespace {
959 
960 /// A helper class that handles instrumentation of VarArg
961 /// functions on a particular platform.
962 ///
963 /// Implementations are expected to insert the instrumentation
964 /// necessary to propagate argument shadow through VarArg function
965 /// calls. Visit* methods are called during an InstVisitor pass over
966 /// the function, and should avoid creating new basic blocks. A new
967 /// instance of this class is created for each instrumented function.
968 struct VarArgHelper {
969  virtual ~VarArgHelper() = default;
970 
971  /// Visit a CallSite.
972  virtual void visitCallSite(CallSite &CS, IRBuilder<> &IRB) = 0;
973 
974  /// Visit a va_start call.
975  virtual void visitVAStartInst(VAStartInst &I) = 0;
976 
977  /// Visit a va_copy call.
978  virtual void visitVACopyInst(VACopyInst &I) = 0;
979 
980  /// Finalize function instrumentation.
981  ///
982  /// This method is called after visiting all interesting (see above)
983  /// instructions in a function.
984  virtual void finalizeInstrumentation() = 0;
985 };
986 
987 struct MemorySanitizerVisitor;
988 
989 } // end anonymous namespace
990 
991 static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
992  MemorySanitizerVisitor &Visitor);
993 
994 static unsigned TypeSizeToSizeIndex(unsigned TypeSize) {
995  if (TypeSize <= 8) return 0;
996  return Log2_32_Ceil((TypeSize + 7) / 8);
997 }
998 
999 namespace {
1000 
1001 /// This class does all the work for a given function. Store and Load
1002 /// instructions store and load corresponding shadow and origin
1003 /// values. Most instructions propagate shadow from arguments to their
1004 /// return values. Certain instructions (most importantly, BranchInst)
1005 /// test their argument shadow and print reports (with a runtime call) if it's
1006 /// non-zero.
1007 struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
1008  Function &F;
1009  MemorySanitizer &MS;
1010  SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
1011  ValueMap<Value*, Value*> ShadowMap, OriginMap;
1012  std::unique_ptr<VarArgHelper> VAHelper;
1013  const TargetLibraryInfo *TLI;
1014  BasicBlock *ActualFnStart;
1015 
1016  // The following flags disable parts of MSan instrumentation based on
1017  // blacklist contents and command-line options.
1018  bool InsertChecks;
1019  bool PropagateShadow;
1020  bool PoisonStack;
1021  bool PoisonUndef;
1022  bool CheckReturnValue;
1023 
1024  struct ShadowOriginAndInsertPoint {
1025  Value *Shadow;
1026  Value *Origin;
1027  Instruction *OrigIns;
1028 
1029  ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
1030  : Shadow(S), Origin(O), OrigIns(I) {}
1031  };
1032  SmallVector<ShadowOriginAndInsertPoint, 16> InstrumentationList;
1033  SmallVector<StoreInst *, 16> StoreList;
1034 
1035  MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
1036  const TargetLibraryInfo &TLI)
1037  : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
1038  bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeMemory);
1039  InsertChecks = SanitizeFunction;
1040  PropagateShadow = SanitizeFunction;
1041  PoisonStack = SanitizeFunction && ClPoisonStack;
1042  PoisonUndef = SanitizeFunction && ClPoisonUndef;
1043  // FIXME: Consider using SpecialCaseList to specify a list of functions that
1044  // must always return fully initialized values. For now, we hardcode "main".
1045  CheckReturnValue = SanitizeFunction && (F.getName() == "main");
1046 
1047  MS.initializeCallbacks(*F.getParent());
1048  if (MS.CompileKernel)
1049  ActualFnStart = insertKmsanPrologue(F);
1050  else
1051  ActualFnStart = &F.getEntryBlock();
1052 
1053  LLVM_DEBUG(if (!InsertChecks) dbgs()
1054  << "MemorySanitizer is not inserting checks into '"
1055  << F.getName() << "'\n");
1056  }
1057 
1058  Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
1059  if (MS.TrackOrigins <= 1) return V;
1060  return IRB.CreateCall(MS.MsanChainOriginFn, V);
1061  }
1062 
1063  Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
1064  const DataLayout &DL = F.getParent()->getDataLayout();
1065  unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1066  if (IntptrSize == kOriginSize) return Origin;
1067  assert(IntptrSize == kOriginSize * 2);
1068  Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
1069  return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
1070  }
1071 
1072  /// Fill memory range with the given origin value.
1073  void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
1074  unsigned Size, unsigned Alignment) {
1075  const DataLayout &DL = F.getParent()->getDataLayout();
1076  unsigned IntptrAlignment = DL.getABITypeAlignment(MS.IntptrTy);
1077  unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1078  assert(IntptrAlignment >= kMinOriginAlignment);
1079  assert(IntptrSize >= kOriginSize);
1080 
1081  unsigned Ofs = 0;
1082  unsigned CurrentAlignment = Alignment;
1083  if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
1084  Value *IntptrOrigin = originToIntptr(IRB, Origin);
1085  Value *IntptrOriginPtr =
1086  IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
1087  for (unsigned i = 0; i < Size / IntptrSize; ++i) {
1088  Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
1089  : IntptrOriginPtr;
1090  IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
1091  Ofs += IntptrSize / kOriginSize;
1092  CurrentAlignment = IntptrAlignment;
1093  }
1094  }
1095 
1096  for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) {
1097  Value *GEP =
1098  i ? IRB.CreateConstGEP1_32(nullptr, OriginPtr, i) : OriginPtr;
1099  IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
1100  CurrentAlignment = kMinOriginAlignment;
1101  }
1102  }
1103 
1104  void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
1105  Value *OriginPtr, unsigned Alignment, bool AsCall) {
1106  const DataLayout &DL = F.getParent()->getDataLayout();
1107  unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1108  unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
1109  if (Shadow->getType()->isAggregateType()) {
1110  paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1111  OriginAlignment);
1112  } else {
1113  Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB);
1114  Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow);
1115  if (ConstantShadow) {
1116  if (ClCheckConstantShadow && !ConstantShadow->isZeroValue())
1117  paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1118  OriginAlignment);
1119  return;
1120  }
1121 
1122  unsigned TypeSizeInBits =
1123  DL.getTypeSizeInBits(ConvertedShadow->getType());
1124  unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1125  if (AsCall && SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1126  Value *Fn = MS.MaybeStoreOriginFn[SizeIndex];
1127  Value *ConvertedShadow2 = IRB.CreateZExt(
1128  ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1129  IRB.CreateCall(Fn, {ConvertedShadow2,
1130  IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()),
1131  Origin});
1132  } else {
1133  Value *Cmp = IRB.CreateICmpNE(
1134  ConvertedShadow, getCleanShadow(ConvertedShadow), "_mscmp");
1136  Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
1137  IRBuilder<> IRBNew(CheckTerm);
1138  paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), OriginPtr, StoreSize,
1139  OriginAlignment);
1140  }
1141  }
1142  }
1143 
1144  void materializeStores(bool InstrumentWithCalls) {
1145  for (StoreInst *SI : StoreList) {
1146  IRBuilder<> IRB(SI);
1147  Value *Val = SI->getValueOperand();
1148  Value *Addr = SI->getPointerOperand();
1149  Value *Shadow = SI->isAtomic() ? getCleanShadow(Val) : getShadow(Val);
1150  Value *ShadowPtr, *OriginPtr;
1151  Type *ShadowTy = Shadow->getType();
1152  unsigned Alignment = SI->getAlignment();
1153  unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1154  std::tie(ShadowPtr, OriginPtr) =
1155  getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ true);
1156 
1157  StoreInst *NewSI = IRB.CreateAlignedStore(Shadow, ShadowPtr, Alignment);
1158  LLVM_DEBUG(dbgs() << " STORE: " << *NewSI << "\n");
1159  (void)NewSI;
1160 
1161  if (SI->isAtomic())
1162  SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
1163 
1164  if (MS.TrackOrigins && !SI->isAtomic())
1165  storeOrigin(IRB, Addr, Shadow, getOrigin(Val), OriginPtr,
1166  OriginAlignment, InstrumentWithCalls);
1167  }
1168  }
1169 
1170  /// Helper function to insert a warning at IRB's current insert point.
1171  void insertWarningFn(IRBuilder<> &IRB, Value *Origin) {
1172  if (!Origin)
1173  Origin = (Value *)IRB.getInt32(0);
1174  if (MS.CompileKernel) {
1175  IRB.CreateCall(MS.WarningFn, Origin);
1176  } else {
1177  if (MS.TrackOrigins) {
1178  IRB.CreateStore(Origin, MS.OriginTLS);
1179  }
1180  IRB.CreateCall(MS.WarningFn, {});
1181  }
1182  IRB.CreateCall(MS.EmptyAsm, {});
1183  // FIXME: Insert UnreachableInst if !MS.Recover?
1184  // This may invalidate some of the following checks and needs to be done
1185  // at the very end.
1186  }
1187 
1188  void materializeOneCheck(Instruction *OrigIns, Value *Shadow, Value *Origin,
1189  bool AsCall) {
1190  IRBuilder<> IRB(OrigIns);
1191  LLVM_DEBUG(dbgs() << " SHAD0 : " << *Shadow << "\n");
1192  Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB);
1193  LLVM_DEBUG(dbgs() << " SHAD1 : " << *ConvertedShadow << "\n");
1194 
1195  Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow);
1196  if (ConstantShadow) {
1197  if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) {
1198  insertWarningFn(IRB, Origin);
1199  }
1200  return;
1201  }
1202 
1203  const DataLayout &DL = OrigIns->getModule()->getDataLayout();
1204 
1205  unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1206  unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1207  if (AsCall && SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1208  Value *Fn = MS.MaybeWarningFn[SizeIndex];
1209  Value *ConvertedShadow2 =
1210  IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1211  IRB.CreateCall(Fn, {ConvertedShadow2, MS.TrackOrigins && Origin
1212  ? Origin
1213  : (Value *)IRB.getInt32(0)});
1214  } else {
1215  Value *Cmp = IRB.CreateICmpNE(ConvertedShadow,
1216  getCleanShadow(ConvertedShadow), "_mscmp");
1218  Cmp, OrigIns,
1219  /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
1220 
1221  IRB.SetInsertPoint(CheckTerm);
1222  insertWarningFn(IRB, Origin);
1223  LLVM_DEBUG(dbgs() << " CHECK: " << *Cmp << "\n");
1224  }
1225  }
1226 
1227  void materializeChecks(bool InstrumentWithCalls) {
1228  for (const auto &ShadowData : InstrumentationList) {
1229  Instruction *OrigIns = ShadowData.OrigIns;
1230  Value *Shadow = ShadowData.Shadow;
1231  Value *Origin = ShadowData.Origin;
1232  materializeOneCheck(OrigIns, Shadow, Origin, InstrumentWithCalls);
1233  }
1234  LLVM_DEBUG(dbgs() << "DONE:\n" << F);
1235  }
1236 
1237  BasicBlock *insertKmsanPrologue(Function &F) {
1238  BasicBlock *ret =
1241  Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
1242  Constant *Zero = IRB.getInt32(0);
1243  MS.ParamTLS =
1244  IRB.CreateGEP(ContextState, {Zero, IRB.getInt32(0)}, "param_shadow");
1245  MS.RetvalTLS =
1246  IRB.CreateGEP(ContextState, {Zero, IRB.getInt32(1)}, "retval_shadow");
1247  MS.VAArgTLS =
1248  IRB.CreateGEP(ContextState, {Zero, IRB.getInt32(2)}, "va_arg_shadow");
1249  MS.VAArgOriginTLS =
1250  IRB.CreateGEP(ContextState, {Zero, IRB.getInt32(3)}, "va_arg_origin");
1251  MS.VAArgOverflowSizeTLS = IRB.CreateGEP(
1252  ContextState, {Zero, IRB.getInt32(4)}, "va_arg_overflow_size");
1253  MS.ParamOriginTLS =
1254  IRB.CreateGEP(ContextState, {Zero, IRB.getInt32(5)}, "param_origin");
1255  MS.RetvalOriginTLS =
1256  IRB.CreateGEP(ContextState, {Zero, IRB.getInt32(6)}, "retval_origin");
1257  return ret;
1258  }
1259 
1260  /// Add MemorySanitizer instrumentation to a function.
1261  bool runOnFunction() {
1262  // In the presence of unreachable blocks, we may see Phi nodes with
1263  // incoming nodes from such blocks. Since InstVisitor skips unreachable
1264  // blocks, such nodes will not have any shadow value associated with them.
1265  // It's easier to remove unreachable blocks than deal with missing shadow.
1267 
1268  // Iterate all BBs in depth-first order and create shadow instructions
1269  // for all instructions (where applicable).
1270  // For PHI nodes we create dummy shadow PHIs which will be finalized later.
1271  for (BasicBlock *BB : depth_first(ActualFnStart))
1272  visit(*BB);
1273 
1274  // Finalize PHI nodes.
1275  for (PHINode *PN : ShadowPHINodes) {
1276  PHINode *PNS = cast<PHINode>(getShadow(PN));
1277  PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
1278  size_t NumValues = PN->getNumIncomingValues();
1279  for (size_t v = 0; v < NumValues; v++) {
1280  PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
1281  if (PNO) PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v));
1282  }
1283  }
1284 
1285  VAHelper->finalizeInstrumentation();
1286 
1287  bool InstrumentWithCalls = ClInstrumentationWithCallThreshold >= 0 &&
1288  InstrumentationList.size() + StoreList.size() >
1290 
1291  // Insert shadow value checks.
1292  materializeChecks(InstrumentWithCalls);
1293 
1294  // Delayed instrumentation of StoreInst.
1295  // This may not add new address checks.
1296  materializeStores(InstrumentWithCalls);
1297 
1298  return true;
1299  }
1300 
1301  /// Compute the shadow type that corresponds to a given Value.
1302  Type *getShadowTy(Value *V) {
1303  return getShadowTy(V->getType());
1304  }
1305 
1306  /// Compute the shadow type that corresponds to a given Type.
1307  Type *getShadowTy(Type *OrigTy) {
1308  if (!OrigTy->isSized()) {
1309  return nullptr;
1310  }
1311  // For integer type, shadow is the same as the original type.
1312  // This may return weird-sized types like i1.
1313  if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
1314  return IT;
1315  const DataLayout &DL = F.getParent()->getDataLayout();
1316  if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
1317  uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
1318  return VectorType::get(IntegerType::get(*MS.C, EltSize),
1319  VT->getNumElements());
1320  }
1321  if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
1322  return ArrayType::get(getShadowTy(AT->getElementType()),
1323  AT->getNumElements());
1324  }
1325  if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
1326  SmallVector<Type*, 4> Elements;
1327  for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1328  Elements.push_back(getShadowTy(ST->getElementType(i)));
1329  StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
1330  LLVM_DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
1331  return Res;
1332  }
1333  uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
1334  return IntegerType::get(*MS.C, TypeSize);
1335  }
1336 
1337  /// Flatten a vector type.
1338  Type *getShadowTyNoVec(Type *ty) {
1339  if (VectorType *vt = dyn_cast<VectorType>(ty))
1340  return IntegerType::get(*MS.C, vt->getBitWidth());
1341  return ty;
1342  }
1343 
1344  /// Convert a shadow value to it's flattened variant.
1345  Value *convertToShadowTyNoVec(Value *V, IRBuilder<> &IRB) {
1346  Type *Ty = V->getType();
1347  Type *NoVecTy = getShadowTyNoVec(Ty);
1348  if (Ty == NoVecTy) return V;
1349  return IRB.CreateBitCast(V, NoVecTy);
1350  }
1351 
1352  /// Compute the integer shadow offset that corresponds to a given
1353  /// application address.
1354  ///
1355  /// Offset = (Addr & ~AndMask) ^ XorMask
1356  Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1357  Value *OffsetLong = IRB.CreatePointerCast(Addr, MS.IntptrTy);
1358 
1359  uint64_t AndMask = MS.MapParams->AndMask;
1360  if (AndMask)
1361  OffsetLong =
1362  IRB.CreateAnd(OffsetLong, ConstantInt::get(MS.IntptrTy, ~AndMask));
1363 
1364  uint64_t XorMask = MS.MapParams->XorMask;
1365  if (XorMask)
1366  OffsetLong =
1367  IRB.CreateXor(OffsetLong, ConstantInt::get(MS.IntptrTy, XorMask));
1368  return OffsetLong;
1369  }
1370 
1371  /// Compute the shadow and origin addresses corresponding to a given
1372  /// application address.
1373  ///
1374  /// Shadow = ShadowBase + Offset
1375  /// Origin = (OriginBase + Offset) & ~3ULL
1376  std::pair<Value *, Value *> getShadowOriginPtrUserspace(Value *Addr,
1377  IRBuilder<> &IRB,
1378  Type *ShadowTy,
1379  unsigned Alignment) {
1380  Value *ShadowOffset = getShadowPtrOffset(Addr, IRB);
1381  Value *ShadowLong = ShadowOffset;
1382  uint64_t ShadowBase = MS.MapParams->ShadowBase;
1383  if (ShadowBase != 0) {
1384  ShadowLong =
1385  IRB.CreateAdd(ShadowLong,
1386  ConstantInt::get(MS.IntptrTy, ShadowBase));
1387  }
1388  Value *ShadowPtr =
1389  IRB.CreateIntToPtr(ShadowLong, PointerType::get(ShadowTy, 0));
1390  Value *OriginPtr = nullptr;
1391  if (MS.TrackOrigins) {
1392  Value *OriginLong = ShadowOffset;
1393  uint64_t OriginBase = MS.MapParams->OriginBase;
1394  if (OriginBase != 0)
1395  OriginLong = IRB.CreateAdd(OriginLong,
1396  ConstantInt::get(MS.IntptrTy, OriginBase));
1397  if (Alignment < kMinOriginAlignment) {
1398  uint64_t Mask = kMinOriginAlignment - 1;
1399  OriginLong =
1400  IRB.CreateAnd(OriginLong, ConstantInt::get(MS.IntptrTy, ~Mask));
1401  }
1402  OriginPtr =
1403  IRB.CreateIntToPtr(OriginLong, PointerType::get(IRB.getInt32Ty(), 0));
1404  }
1405  return std::make_pair(ShadowPtr, OriginPtr);
1406  }
1407 
1408  std::pair<Value *, Value *>
1409  getShadowOriginPtrKernel(Value *Addr, IRBuilder<> &IRB, Type *ShadowTy,
1410  unsigned Alignment, bool isStore) {
1411  Value *ShadowOriginPtrs;
1412  const DataLayout &DL = F.getParent()->getDataLayout();
1413  int Size = DL.getTypeStoreSize(ShadowTy);
1414 
1415  Value *Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
1416  Value *AddrCast =
1417  IRB.CreatePointerCast(Addr, PointerType::get(IRB.getInt8Ty(), 0));
1418  if (Getter) {
1419  ShadowOriginPtrs = IRB.CreateCall(Getter, AddrCast);
1420  } else {
1421  Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
1422  ShadowOriginPtrs = IRB.CreateCall(isStore ? MS.MsanMetadataPtrForStoreN
1423  : MS.MsanMetadataPtrForLoadN,
1424  {AddrCast, SizeVal});
1425  }
1426  Value *ShadowPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 0);
1427  ShadowPtr = IRB.CreatePointerCast(ShadowPtr, PointerType::get(ShadowTy, 0));
1428  Value *OriginPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 1);
1429 
1430  return std::make_pair(ShadowPtr, OriginPtr);
1431  }
1432 
1433  std::pair<Value *, Value *> getShadowOriginPtr(Value *Addr, IRBuilder<> &IRB,
1434  Type *ShadowTy,
1435  unsigned Alignment,
1436  bool isStore) {
1437  std::pair<Value *, Value *> ret;
1438  if (MS.CompileKernel)
1439  ret = getShadowOriginPtrKernel(Addr, IRB, ShadowTy, Alignment, isStore);
1440  else
1441  ret = getShadowOriginPtrUserspace(Addr, IRB, ShadowTy, Alignment);
1442  return ret;
1443  }
1444 
1445  /// Compute the shadow address for a given function argument.
1446  ///
1447  /// Shadow = ParamTLS+ArgOffset.
1448  Value *getShadowPtrForArgument(Value *A, IRBuilder<> &IRB,
1449  int ArgOffset) {
1450  Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1451  if (ArgOffset)
1452  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1453  return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0),
1454  "_msarg");
1455  }
1456 
1457  /// Compute the origin address for a given function argument.
1458  Value *getOriginPtrForArgument(Value *A, IRBuilder<> &IRB,
1459  int ArgOffset) {
1460  if (!MS.TrackOrigins)
1461  return nullptr;
1462  Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1463  if (ArgOffset)
1464  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1465  return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
1466  "_msarg_o");
1467  }
1468 
1469  /// Compute the shadow address for a retval.
1470  Value *getShadowPtrForRetval(Value *A, IRBuilder<> &IRB) {
1471  return IRB.CreatePointerCast(MS.RetvalTLS,
1472  PointerType::get(getShadowTy(A), 0),
1473  "_msret");
1474  }
1475 
1476  /// Compute the origin address for a retval.
1477  Value *getOriginPtrForRetval(IRBuilder<> &IRB) {
1478  // We keep a single origin for the entire retval. Might be too optimistic.
1479  return MS.RetvalOriginTLS;
1480  }
1481 
1482  /// Set SV to be the shadow value for V.
1483  void setShadow(Value *V, Value *SV) {
1484  assert(!ShadowMap.count(V) && "Values may only have one shadow");
1485  ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V);
1486  }
1487 
1488  /// Set Origin to be the origin value for V.
1489  void setOrigin(Value *V, Value *Origin) {
1490  if (!MS.TrackOrigins) return;
1491  assert(!OriginMap.count(V) && "Values may only have one origin");
1492  LLVM_DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n");
1493  OriginMap[V] = Origin;
1494  }
1495 
1496  Constant *getCleanShadow(Type *OrigTy) {
1497  Type *ShadowTy = getShadowTy(OrigTy);
1498  if (!ShadowTy)
1499  return nullptr;
1500  return Constant::getNullValue(ShadowTy);
1501  }
1502 
1503  /// Create a clean shadow value for a given value.
1504  ///
1505  /// Clean shadow (all zeroes) means all bits of the value are defined
1506  /// (initialized).
1507  Constant *getCleanShadow(Value *V) {
1508  return getCleanShadow(V->getType());
1509  }
1510 
1511  /// Create a dirty shadow of a given shadow type.
1512  Constant *getPoisonedShadow(Type *ShadowTy) {
1513  assert(ShadowTy);
1514  if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy))
1515  return Constant::getAllOnesValue(ShadowTy);
1516  if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1517  SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1518  getPoisonedShadow(AT->getElementType()));
1519  return ConstantArray::get(AT, Vals);
1520  }
1521  if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1523  for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1524  Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1525  return ConstantStruct::get(ST, Vals);
1526  }
1527  llvm_unreachable("Unexpected shadow type");
1528  }
1529 
1530  /// Create a dirty shadow for a given value.
1531  Constant *getPoisonedShadow(Value *V) {
1532  Type *ShadowTy = getShadowTy(V);
1533  if (!ShadowTy)
1534  return nullptr;
1535  return getPoisonedShadow(ShadowTy);
1536  }
1537 
1538  /// Create a clean (zero) origin.
1539  Value *getCleanOrigin() {
1540  return Constant::getNullValue(MS.OriginTy);
1541  }
1542 
1543  /// Get the shadow value for a given Value.
1544  ///
1545  /// This function either returns the value set earlier with setShadow,
1546  /// or extracts if from ParamTLS (for function arguments).
1547  Value *getShadow(Value *V) {
1548  if (!PropagateShadow) return getCleanShadow(V);
1549  if (Instruction *I = dyn_cast<Instruction>(V)) {
1550  if (I->getMetadata("nosanitize"))
1551  return getCleanShadow(V);
1552  // For instructions the shadow is already stored in the map.
1553  Value *Shadow = ShadowMap[V];
1554  if (!Shadow) {
1555  LLVM_DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1556  (void)I;
1557  assert(Shadow && "No shadow for a value");
1558  }
1559  return Shadow;
1560  }
1561  if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1562  Value *AllOnes = PoisonUndef ? getPoisonedShadow(V) : getCleanShadow(V);
1563  LLVM_DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1564  (void)U;
1565  return AllOnes;
1566  }
1567  if (Argument *A = dyn_cast<Argument>(V)) {
1568  // For arguments we compute the shadow on demand and store it in the map.
1569  Value **ShadowPtr = &ShadowMap[V];
1570  if (*ShadowPtr)
1571  return *ShadowPtr;
1572  Function *F = A->getParent();
1573  IRBuilder<> EntryIRB(ActualFnStart->getFirstNonPHI());
1574  unsigned ArgOffset = 0;
1575  const DataLayout &DL = F->getParent()->getDataLayout();
1576  for (auto &FArg : F->args()) {
1577  if (!FArg.getType()->isSized()) {
1578  LLVM_DEBUG(dbgs() << "Arg is not sized\n");
1579  continue;
1580  }
1581  unsigned Size =
1582  FArg.hasByValAttr()
1583  ? DL.getTypeAllocSize(FArg.getType()->getPointerElementType())
1584  : DL.getTypeAllocSize(FArg.getType());
1585  if (A == &FArg) {
1586  bool Overflow = ArgOffset + Size > kParamTLSSize;
1587  Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset);
1588  if (FArg.hasByValAttr()) {
1589  // ByVal pointer itself has clean shadow. We copy the actual
1590  // argument shadow to the underlying memory.
1591  // Figure out maximal valid memcpy alignment.
1592  unsigned ArgAlign = FArg.getParamAlignment();
1593  if (ArgAlign == 0) {
1594  Type *EltType = A->getType()->getPointerElementType();
1595  ArgAlign = DL.getABITypeAlignment(EltType);
1596  }
1597  Value *CpShadowPtr =
1598  getShadowOriginPtr(V, EntryIRB, EntryIRB.getInt8Ty(), ArgAlign,
1599  /*isStore*/ true)
1600  .first;
1601  // TODO(glider): need to copy origins.
1602  if (Overflow) {
1603  // ParamTLS overflow.
1604  EntryIRB.CreateMemSet(
1605  CpShadowPtr, Constant::getNullValue(EntryIRB.getInt8Ty()),
1606  Size, ArgAlign);
1607  } else {
1608  unsigned CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
1609  Value *Cpy = EntryIRB.CreateMemCpy(CpShadowPtr, CopyAlign, Base,
1610  CopyAlign, Size);
1611  LLVM_DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n");
1612  (void)Cpy;
1613  }
1614  *ShadowPtr = getCleanShadow(V);
1615  } else {
1616  if (Overflow) {
1617  // ParamTLS overflow.
1618  *ShadowPtr = getCleanShadow(V);
1619  } else {
1620  *ShadowPtr =
1621  EntryIRB.CreateAlignedLoad(Base, kShadowTLSAlignment);
1622  }
1623  }
1624  LLVM_DEBUG(dbgs()
1625  << " ARG: " << FArg << " ==> " << **ShadowPtr << "\n");
1626  if (MS.TrackOrigins && !Overflow) {
1627  Value *OriginPtr =
1628  getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset);
1629  setOrigin(A, EntryIRB.CreateLoad(OriginPtr));
1630  } else {
1631  setOrigin(A, getCleanOrigin());
1632  }
1633  }
1634  ArgOffset += alignTo(Size, kShadowTLSAlignment);
1635  }
1636  assert(*ShadowPtr && "Could not find shadow for an argument");
1637  return *ShadowPtr;
1638  }
1639  // For everything else the shadow is zero.
1640  return getCleanShadow(V);
1641  }
1642 
1643  /// Get the shadow for i-th argument of the instruction I.
1644  Value *getShadow(Instruction *I, int i) {
1645  return getShadow(I->getOperand(i));
1646  }
1647 
1648  /// Get the origin for a value.
1649  Value *getOrigin(Value *V) {
1650  if (!MS.TrackOrigins) return nullptr;
1651  if (!PropagateShadow) return getCleanOrigin();
1652  if (isa<Constant>(V)) return getCleanOrigin();
1653  assert((isa<Instruction>(V) || isa<Argument>(V)) &&
1654  "Unexpected value type in getOrigin()");
1655  if (Instruction *I = dyn_cast<Instruction>(V)) {
1656  if (I->getMetadata("nosanitize"))
1657  return getCleanOrigin();
1658  }
1659  Value *Origin = OriginMap[V];
1660  assert(Origin && "Missing origin");
1661  return Origin;
1662  }
1663 
1664  /// Get the origin for i-th argument of the instruction I.
1665  Value *getOrigin(Instruction *I, int i) {
1666  return getOrigin(I->getOperand(i));
1667  }
1668 
1669  /// Remember the place where a shadow check should be inserted.
1670  ///
1671  /// This location will be later instrumented with a check that will print a
1672  /// UMR warning in runtime if the shadow value is not 0.
1673  void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
1674  assert(Shadow);
1675  if (!InsertChecks) return;
1676 #ifndef NDEBUG
1677  Type *ShadowTy = Shadow->getType();
1678  assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy)) &&
1679  "Can only insert checks for integer and vector shadow types");
1680 #endif
1681  InstrumentationList.push_back(
1682  ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
1683  }
1684 
1685  /// Remember the place where a shadow check should be inserted.
1686  ///
1687  /// This location will be later instrumented with a check that will print a
1688  /// UMR warning in runtime if the value is not fully defined.
1689  void insertShadowCheck(Value *Val, Instruction *OrigIns) {
1690  assert(Val);
1691  Value *Shadow, *Origin;
1692  if (ClCheckConstantShadow) {
1693  Shadow = getShadow(Val);
1694  if (!Shadow) return;
1695  Origin = getOrigin(Val);
1696  } else {
1697  Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
1698  if (!Shadow) return;
1699  Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
1700  }
1701  insertShadowCheck(Shadow, Origin, OrigIns);
1702  }
1703 
1704  AtomicOrdering addReleaseOrdering(AtomicOrdering a) {
1705  switch (a) {
1711  return AtomicOrdering::Release;
1717  }
1718  llvm_unreachable("Unknown ordering");
1719  }
1720 
1721  AtomicOrdering addAcquireOrdering(AtomicOrdering a) {
1722  switch (a) {
1728  return AtomicOrdering::Acquire;
1734  }
1735  llvm_unreachable("Unknown ordering");
1736  }
1737 
1738  // ------------------- Visitors.
1740  void visit(Instruction &I) {
1741  if (!I.getMetadata("nosanitize"))
1743  }
1744 
1745  /// Instrument LoadInst
1746  ///
1747  /// Loads the corresponding shadow and (optionally) origin.
1748  /// Optionally, checks that the load address is fully defined.
1749  void visitLoadInst(LoadInst &I) {
1750  assert(I.getType()->isSized() && "Load type must have size");
1751  assert(!I.getMetadata("nosanitize"));
1752  IRBuilder<> IRB(I.getNextNode());
1753  Type *ShadowTy = getShadowTy(&I);
1754  Value *Addr = I.getPointerOperand();
1755  Value *ShadowPtr, *OriginPtr;
1756  unsigned Alignment = I.getAlignment();
1757  if (PropagateShadow) {
1758  std::tie(ShadowPtr, OriginPtr) =
1759  getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
1760  setShadow(&I, IRB.CreateAlignedLoad(ShadowPtr, Alignment, "_msld"));
1761  } else {
1762  setShadow(&I, getCleanShadow(&I));
1763  }
1764 
1766  insertShadowCheck(I.getPointerOperand(), &I);
1767 
1768  if (I.isAtomic())
1769  I.setOrdering(addAcquireOrdering(I.getOrdering()));
1770 
1771  if (MS.TrackOrigins) {
1772  if (PropagateShadow) {
1773  unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1774  setOrigin(&I, IRB.CreateAlignedLoad(OriginPtr, OriginAlignment));
1775  } else {
1776  setOrigin(&I, getCleanOrigin());
1777  }
1778  }
1779  }
1780 
1781  /// Instrument StoreInst
1782  ///
1783  /// Stores the corresponding shadow and (optionally) origin.
1784  /// Optionally, checks that the store address is fully defined.
1785  void visitStoreInst(StoreInst &I) {
1786  StoreList.push_back(&I);
1788  insertShadowCheck(I.getPointerOperand(), &I);
1789  }
1790 
1791  void handleCASOrRMW(Instruction &I) {
1792  assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
1793 
1794  IRBuilder<> IRB(&I);
1795  Value *Addr = I.getOperand(0);
1796  Value *ShadowPtr = getShadowOriginPtr(Addr, IRB, I.getType(),
1797  /*Alignment*/ 1, /*isStore*/ true)
1798  .first;
1799 
1801  insertShadowCheck(Addr, &I);
1802 
1803  // Only test the conditional argument of cmpxchg instruction.
1804  // The other argument can potentially be uninitialized, but we can not
1805  // detect this situation reliably without possible false positives.
1806  if (isa<AtomicCmpXchgInst>(I))
1807  insertShadowCheck(I.getOperand(1), &I);
1808 
1809  IRB.CreateStore(getCleanShadow(&I), ShadowPtr);
1810 
1811  setShadow(&I, getCleanShadow(&I));
1812  setOrigin(&I, getCleanOrigin());
1813  }
1814 
1815  void visitAtomicRMWInst(AtomicRMWInst &I) {
1816  handleCASOrRMW(I);
1817  I.setOrdering(addReleaseOrdering(I.getOrdering()));
1818  }
1819 
1820  void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
1821  handleCASOrRMW(I);
1822  I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
1823  }
1824 
1825  // Vector manipulation.
1826  void visitExtractElementInst(ExtractElementInst &I) {
1827  insertShadowCheck(I.getOperand(1), &I);
1828  IRBuilder<> IRB(&I);
1829  setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
1830  "_msprop"));
1831  setOrigin(&I, getOrigin(&I, 0));
1832  }
1833 
1834  void visitInsertElementInst(InsertElementInst &I) {
1835  insertShadowCheck(I.getOperand(2), &I);
1836  IRBuilder<> IRB(&I);
1837  setShadow(&I, IRB.CreateInsertElement(getShadow(&I, 0), getShadow(&I, 1),
1838  I.getOperand(2), "_msprop"));
1839  setOriginForNaryOp(I);
1840  }
1841 
1842  void visitShuffleVectorInst(ShuffleVectorInst &I) {
1843  insertShadowCheck(I.getOperand(2), &I);
1844  IRBuilder<> IRB(&I);
1845  setShadow(&I, IRB.CreateShuffleVector(getShadow(&I, 0), getShadow(&I, 1),
1846  I.getOperand(2), "_msprop"));
1847  setOriginForNaryOp(I);
1848  }
1849 
1850  // Casts.
1851  void visitSExtInst(SExtInst &I) {
1852  IRBuilder<> IRB(&I);
1853  setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
1854  setOrigin(&I, getOrigin(&I, 0));
1855  }
1856 
1857  void visitZExtInst(ZExtInst &I) {
1858  IRBuilder<> IRB(&I);
1859  setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
1860  setOrigin(&I, getOrigin(&I, 0));
1861  }
1862 
1863  void visitTruncInst(TruncInst &I) {
1864  IRBuilder<> IRB(&I);
1865  setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
1866  setOrigin(&I, getOrigin(&I, 0));
1867  }
1868 
1869  void visitBitCastInst(BitCastInst &I) {
1870  // Special case: if this is the bitcast (there is exactly 1 allowed) between
1871  // a musttail call and a ret, don't instrument. New instructions are not
1872  // allowed after a musttail call.
1873  if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
1874  if (CI->isMustTailCall())
1875  return;
1876  IRBuilder<> IRB(&I);
1877  setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
1878  setOrigin(&I, getOrigin(&I, 0));
1879  }
1880 
1881  void visitPtrToIntInst(PtrToIntInst &I) {
1882  IRBuilder<> IRB(&I);
1883  setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
1884  "_msprop_ptrtoint"));
1885  setOrigin(&I, getOrigin(&I, 0));
1886  }
1887 
1888  void visitIntToPtrInst(IntToPtrInst &I) {
1889  IRBuilder<> IRB(&I);
1890  setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
1891  "_msprop_inttoptr"));
1892  setOrigin(&I, getOrigin(&I, 0));
1893  }
1894 
1895  void visitFPToSIInst(CastInst& I) { handleShadowOr(I); }
1896  void visitFPToUIInst(CastInst& I) { handleShadowOr(I); }
1897  void visitSIToFPInst(CastInst& I) { handleShadowOr(I); }
1898  void visitUIToFPInst(CastInst& I) { handleShadowOr(I); }
1899  void visitFPExtInst(CastInst& I) { handleShadowOr(I); }
1900  void visitFPTruncInst(CastInst& I) { handleShadowOr(I); }
1901 
1902  /// Propagate shadow for bitwise AND.
1903  ///
1904  /// This code is exact, i.e. if, for example, a bit in the left argument
1905  /// is defined and 0, then neither the value not definedness of the
1906  /// corresponding bit in B don't affect the resulting shadow.
1907  void visitAnd(BinaryOperator &I) {
1908  IRBuilder<> IRB(&I);
1909  // "And" of 0 and a poisoned value results in unpoisoned value.
1910  // 1&1 => 1; 0&1 => 0; p&1 => p;
1911  // 1&0 => 0; 0&0 => 0; p&0 => 0;
1912  // 1&p => p; 0&p => 0; p&p => p;
1913  // S = (S1 & S2) | (V1 & S2) | (S1 & V2)
1914  Value *S1 = getShadow(&I, 0);
1915  Value *S2 = getShadow(&I, 1);
1916  Value *V1 = I.getOperand(0);
1917  Value *V2 = I.getOperand(1);
1918  if (V1->getType() != S1->getType()) {
1919  V1 = IRB.CreateIntCast(V1, S1->getType(), false);
1920  V2 = IRB.CreateIntCast(V2, S2->getType(), false);
1921  }
1922  Value *S1S2 = IRB.CreateAnd(S1, S2);
1923  Value *V1S2 = IRB.CreateAnd(V1, S2);
1924  Value *S1V2 = IRB.CreateAnd(S1, V2);
1925  setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2)));
1926  setOriginForNaryOp(I);
1927  }
1928 
1929  void visitOr(BinaryOperator &I) {
1930  IRBuilder<> IRB(&I);
1931  // "Or" of 1 and a poisoned value results in unpoisoned value.
1932  // 1|1 => 1; 0|1 => 1; p|1 => 1;
1933  // 1|0 => 1; 0|0 => 0; p|0 => p;
1934  // 1|p => 1; 0|p => p; p|p => p;
1935  // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
1936  Value *S1 = getShadow(&I, 0);
1937  Value *S2 = getShadow(&I, 1);
1938  Value *V1 = IRB.CreateNot(I.getOperand(0));
1939  Value *V2 = IRB.CreateNot(I.getOperand(1));
1940  if (V1->getType() != S1->getType()) {
1941  V1 = IRB.CreateIntCast(V1, S1->getType(), false);
1942  V2 = IRB.CreateIntCast(V2, S2->getType(), false);
1943  }
1944  Value *S1S2 = IRB.CreateAnd(S1, S2);
1945  Value *V1S2 = IRB.CreateAnd(V1, S2);
1946  Value *S1V2 = IRB.CreateAnd(S1, V2);
1947  setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2)));
1948  setOriginForNaryOp(I);
1949  }
1950 
1951  /// Default propagation of shadow and/or origin.
1952  ///
1953  /// This class implements the general case of shadow propagation, used in all
1954  /// cases where we don't know and/or don't care about what the operation
1955  /// actually does. It converts all input shadow values to a common type
1956  /// (extending or truncating as necessary), and bitwise OR's them.
1957  ///
1958  /// This is much cheaper than inserting checks (i.e. requiring inputs to be
1959  /// fully initialized), and less prone to false positives.
1960  ///
1961  /// This class also implements the general case of origin propagation. For a
1962  /// Nary operation, result origin is set to the origin of an argument that is
1963  /// not entirely initialized. If there is more than one such arguments, the
1964  /// rightmost of them is picked. It does not matter which one is picked if all
1965  /// arguments are initialized.
1966  template <bool CombineShadow>
1967  class Combiner {
1968  Value *Shadow = nullptr;
1969  Value *Origin = nullptr;
1970  IRBuilder<> &IRB;
1971  MemorySanitizerVisitor *MSV;
1972 
1973  public:
1974  Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
1975  : IRB(IRB), MSV(MSV) {}
1976 
1977  /// Add a pair of shadow and origin values to the mix.
1978  Combiner &Add(Value *OpShadow, Value *OpOrigin) {
1979  if (CombineShadow) {
1980  assert(OpShadow);
1981  if (!Shadow)
1982  Shadow = OpShadow;
1983  else {
1984  OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
1985  Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
1986  }
1987  }
1988 
1989  if (MSV->MS.TrackOrigins) {
1990  assert(OpOrigin);
1991  if (!Origin) {
1992  Origin = OpOrigin;
1993  } else {
1994  Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
1995  // No point in adding something that might result in 0 origin value.
1996  if (!ConstOrigin || !ConstOrigin->isNullValue()) {
1997  Value *FlatShadow = MSV->convertToShadowTyNoVec(OpShadow, IRB);
1998  Value *Cond =
1999  IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow));
2000  Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2001  }
2002  }
2003  }
2004  return *this;
2005  }
2006 
2007  /// Add an application value to the mix.
2008  Combiner &Add(Value *V) {
2009  Value *OpShadow = MSV->getShadow(V);
2010  Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr;
2011  return Add(OpShadow, OpOrigin);
2012  }
2013 
2014  /// Set the current combined values as the given instruction's shadow
2015  /// and origin.
2016  void Done(Instruction *I) {
2017  if (CombineShadow) {
2018  assert(Shadow);
2019  Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2020  MSV->setShadow(I, Shadow);
2021  }
2022  if (MSV->MS.TrackOrigins) {
2023  assert(Origin);
2024  MSV->setOrigin(I, Origin);
2025  }
2026  }
2027  };
2028 
2029  using ShadowAndOriginCombiner = Combiner<true>;
2030  using OriginCombiner = Combiner<false>;
2031 
2032  /// Propagate origin for arbitrary operation.
2033  void setOriginForNaryOp(Instruction &I) {
2034  if (!MS.TrackOrigins) return;
2035  IRBuilder<> IRB(&I);
2036  OriginCombiner OC(this, IRB);
2037  for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI)
2038  OC.Add(OI->get());
2039  OC.Done(&I);
2040  }
2041 
2042  size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
2043  assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
2044  "Vector of pointers is not a valid shadow type");
2045  return Ty->isVectorTy() ?
2047  Ty->getPrimitiveSizeInBits();
2048  }
2049 
2050  /// Cast between two shadow types, extending or truncating as
2051  /// necessary.
2052  Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
2053  bool Signed = false) {
2054  Type *srcTy = V->getType();
2055  size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
2056  size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
2057  if (srcSizeInBits > 1 && dstSizeInBits == 1)
2058  return IRB.CreateICmpNE(V, getCleanShadow(V));
2059 
2060  if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
2061  return IRB.CreateIntCast(V, dstTy, Signed);
2062  if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
2063  dstTy->getVectorNumElements() == srcTy->getVectorNumElements())
2064  return IRB.CreateIntCast(V, dstTy, Signed);
2065  Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
2066  Value *V2 =
2067  IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
2068  return IRB.CreateBitCast(V2, dstTy);
2069  // TODO: handle struct types.
2070  }
2071 
2072  /// Cast an application value to the type of its own shadow.
2073  Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
2074  Type *ShadowTy = getShadowTy(V);
2075  if (V->getType() == ShadowTy)
2076  return V;
2077  if (V->getType()->isPtrOrPtrVectorTy())
2078  return IRB.CreatePtrToInt(V, ShadowTy);
2079  else
2080  return IRB.CreateBitCast(V, ShadowTy);
2081  }
2082 
2083  /// Propagate shadow for arbitrary operation.
2084  void handleShadowOr(Instruction &I) {
2085  IRBuilder<> IRB(&I);
2086  ShadowAndOriginCombiner SC(this, IRB);
2087  for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI)
2088  SC.Add(OI->get());
2089  SC.Done(&I);
2090  }
2091 
2092  // Handle multiplication by constant.
2093  //
2094  // Handle a special case of multiplication by constant that may have one or
2095  // more zeros in the lower bits. This makes corresponding number of lower bits
2096  // of the result zero as well. We model it by shifting the other operand
2097  // shadow left by the required number of bits. Effectively, we transform
2098  // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
2099  // We use multiplication by 2**N instead of shift to cover the case of
2100  // multiplication by 0, which may occur in some elements of a vector operand.
2101  void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
2102  Value *OtherArg) {
2103  Constant *ShadowMul;
2104  Type *Ty = ConstArg->getType();
2105  if (Ty->isVectorTy()) {
2106  unsigned NumElements = Ty->getVectorNumElements();
2107  Type *EltTy = Ty->getSequentialElementType();
2108  SmallVector<Constant *, 16> Elements;
2109  for (unsigned Idx = 0; Idx < NumElements; ++Idx) {
2110  if (ConstantInt *Elt =
2111  dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
2112  const APInt &V = Elt->getValue();
2113  APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
2114  Elements.push_back(ConstantInt::get(EltTy, V2));
2115  } else {
2116  Elements.push_back(ConstantInt::get(EltTy, 1));
2117  }
2118  }
2119  ShadowMul = ConstantVector::get(Elements);
2120  } else {
2121  if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
2122  const APInt &V = Elt->getValue();
2123  APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
2124  ShadowMul = ConstantInt::get(Ty, V2);
2125  } else {
2126  ShadowMul = ConstantInt::get(Ty, 1);
2127  }
2128  }
2129 
2130  IRBuilder<> IRB(&I);
2131  setShadow(&I,
2132  IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
2133  setOrigin(&I, getOrigin(OtherArg));
2134  }
2135 
2136  void visitMul(BinaryOperator &I) {
2137  Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
2138  Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
2139  if (constOp0 && !constOp1)
2140  handleMulByConstant(I, constOp0, I.getOperand(1));
2141  else if (constOp1 && !constOp0)
2142  handleMulByConstant(I, constOp1, I.getOperand(0));
2143  else
2144  handleShadowOr(I);
2145  }
2146 
2147  void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
2148  void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
2149  void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
2150  void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
2151  void visitSub(BinaryOperator &I) { handleShadowOr(I); }
2152  void visitXor(BinaryOperator &I) { handleShadowOr(I); }
2153 
2154  void handleIntegerDiv(Instruction &I) {
2155  IRBuilder<> IRB(&I);
2156  // Strict on the second argument.
2157  insertShadowCheck(I.getOperand(1), &I);
2158  setShadow(&I, getShadow(&I, 0));
2159  setOrigin(&I, getOrigin(&I, 0));
2160  }
2161 
2162  void visitUDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2163  void visitSDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2164  void visitURem(BinaryOperator &I) { handleIntegerDiv(I); }
2165  void visitSRem(BinaryOperator &I) { handleIntegerDiv(I); }
2166 
2167  // Floating point division is side-effect free. We can not require that the
2168  // divisor is fully initialized and must propagate shadow. See PR37523.
2169  void visitFDiv(BinaryOperator &I) { handleShadowOr(I); }
2170  void visitFRem(BinaryOperator &I) { handleShadowOr(I); }
2171 
2172  /// Instrument == and != comparisons.
2173  ///
2174  /// Sometimes the comparison result is known even if some of the bits of the
2175  /// arguments are not.
2176  void handleEqualityComparison(ICmpInst &I) {
2177  IRBuilder<> IRB(&I);
2178  Value *A = I.getOperand(0);
2179  Value *B = I.getOperand(1);
2180  Value *Sa = getShadow(A);
2181  Value *Sb = getShadow(B);
2182 
2183  // Get rid of pointers and vectors of pointers.
2184  // For ints (and vectors of ints), types of A and Sa match,
2185  // and this is a no-op.
2186  A = IRB.CreatePointerCast(A, Sa->getType());
2187  B = IRB.CreatePointerCast(B, Sb->getType());
2188 
2189  // A == B <==> (C = A^B) == 0
2190  // A != B <==> (C = A^B) != 0
2191  // Sc = Sa | Sb
2192  Value *C = IRB.CreateXor(A, B);
2193  Value *Sc = IRB.CreateOr(Sa, Sb);
2194  // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
2195  // Result is defined if one of the following is true
2196  // * there is a defined 1 bit in C
2197  // * C is fully defined
2198  // Si = !(C & ~Sc) && Sc
2199  Value *Zero = Constant::getNullValue(Sc->getType());
2200  Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
2201  Value *Si =
2202  IRB.CreateAnd(IRB.CreateICmpNE(Sc, Zero),
2203  IRB.CreateICmpEQ(
2204  IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero));
2205  Si->setName("_msprop_icmp");
2206  setShadow(&I, Si);
2207  setOriginForNaryOp(I);
2208  }
2209 
2210  /// Build the lowest possible value of V, taking into account V's
2211  /// uninitialized bits.
2212  Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2213  bool isSigned) {
2214  if (isSigned) {
2215  // Split shadow into sign bit and other bits.
2216  Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2217  Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2218  // Maximise the undefined shadow bit, minimize other undefined bits.
2219  return
2220  IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)), SaSignBit);
2221  } else {
2222  // Minimize undefined bits.
2223  return IRB.CreateAnd(A, IRB.CreateNot(Sa));
2224  }
2225  }
2226 
2227  /// Build the highest possible value of V, taking into account V's
2228  /// uninitialized bits.
2229  Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2230  bool isSigned) {
2231  if (isSigned) {
2232  // Split shadow into sign bit and other bits.
2233  Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2234  Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2235  // Minimise the undefined shadow bit, maximise other undefined bits.
2236  return
2237  IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)), SaOtherBits);
2238  } else {
2239  // Maximize undefined bits.
2240  return IRB.CreateOr(A, Sa);
2241  }
2242  }
2243 
2244  /// Instrument relational comparisons.
2245  ///
2246  /// This function does exact shadow propagation for all relational
2247  /// comparisons of integers, pointers and vectors of those.
2248  /// FIXME: output seems suboptimal when one of the operands is a constant
2249  void handleRelationalComparisonExact(ICmpInst &I) {
2250  IRBuilder<> IRB(&I);
2251  Value *A = I.getOperand(0);
2252  Value *B = I.getOperand(1);
2253  Value *Sa = getShadow(A);
2254  Value *Sb = getShadow(B);
2255 
2256  // Get rid of pointers and vectors of pointers.
2257  // For ints (and vectors of ints), types of A and Sa match,
2258  // and this is a no-op.
2259  A = IRB.CreatePointerCast(A, Sa->getType());
2260  B = IRB.CreatePointerCast(B, Sb->getType());
2261 
2262  // Let [a0, a1] be the interval of possible values of A, taking into account
2263  // its undefined bits. Let [b0, b1] be the interval of possible values of B.
2264  // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
2265  bool IsSigned = I.isSigned();
2266  Value *S1 = IRB.CreateICmp(I.getPredicate(),
2267  getLowestPossibleValue(IRB, A, Sa, IsSigned),
2268  getHighestPossibleValue(IRB, B, Sb, IsSigned));
2269  Value *S2 = IRB.CreateICmp(I.getPredicate(),
2270  getHighestPossibleValue(IRB, A, Sa, IsSigned),
2271  getLowestPossibleValue(IRB, B, Sb, IsSigned));
2272  Value *Si = IRB.CreateXor(S1, S2);
2273  setShadow(&I, Si);
2274  setOriginForNaryOp(I);
2275  }
2276 
2277  /// Instrument signed relational comparisons.
2278  ///
2279  /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
2280  /// bit of the shadow. Everything else is delegated to handleShadowOr().
2281  void handleSignedRelationalComparison(ICmpInst &I) {
2282  Constant *constOp;
2283  Value *op = nullptr;
2284  CmpInst::Predicate pre;
2285  if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
2286  op = I.getOperand(0);
2287  pre = I.getPredicate();
2288  } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) {
2289  op = I.getOperand(1);
2290  pre = I.getSwappedPredicate();
2291  } else {
2292  handleShadowOr(I);
2293  return;
2294  }
2295 
2296  if ((constOp->isNullValue() &&
2297  (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) ||
2298  (constOp->isAllOnesValue() &&
2299  (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) {
2300  IRBuilder<> IRB(&I);
2301  Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
2302  "_msprop_icmp_s");
2303  setShadow(&I, Shadow);
2304  setOrigin(&I, getOrigin(op));
2305  } else {
2306  handleShadowOr(I);
2307  }
2308  }
2309 
2310  void visitICmpInst(ICmpInst &I) {
2311  if (!ClHandleICmp) {
2312  handleShadowOr(I);
2313  return;
2314  }
2315  if (I.isEquality()) {
2316  handleEqualityComparison(I);
2317  return;
2318  }
2319 
2320  assert(I.isRelational());
2321  if (ClHandleICmpExact) {
2322  handleRelationalComparisonExact(I);
2323  return;
2324  }
2325  if (I.isSigned()) {
2326  handleSignedRelationalComparison(I);
2327  return;
2328  }
2329 
2330  assert(I.isUnsigned());
2331  if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
2332  handleRelationalComparisonExact(I);
2333  return;
2334  }
2335 
2336  handleShadowOr(I);
2337  }
2338 
2339  void visitFCmpInst(FCmpInst &I) {
2340  handleShadowOr(I);
2341  }
2342 
2343  void handleShift(BinaryOperator &I) {
2344  IRBuilder<> IRB(&I);
2345  // If any of the S2 bits are poisoned, the whole thing is poisoned.
2346  // Otherwise perform the same shift on S1.
2347  Value *S1 = getShadow(&I, 0);
2348  Value *S2 = getShadow(&I, 1);
2349  Value *S2Conv = IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)),
2350  S2->getType());
2351  Value *V2 = I.getOperand(1);
2352  Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
2353  setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2354  setOriginForNaryOp(I);
2355  }
2356 
2357  void visitShl(BinaryOperator &I) { handleShift(I); }
2358  void visitAShr(BinaryOperator &I) { handleShift(I); }
2359  void visitLShr(BinaryOperator &I) { handleShift(I); }
2360 
2361  /// Instrument llvm.memmove
2362  ///
2363  /// At this point we don't know if llvm.memmove will be inlined or not.
2364  /// If we don't instrument it and it gets inlined,
2365  /// our interceptor will not kick in and we will lose the memmove.
2366  /// If we instrument the call here, but it does not get inlined,
2367  /// we will memove the shadow twice: which is bad in case
2368  /// of overlapping regions. So, we simply lower the intrinsic to a call.
2369  ///
2370  /// Similar situation exists for memcpy and memset.
2371  void visitMemMoveInst(MemMoveInst &I) {
2372  IRBuilder<> IRB(&I);
2373  IRB.CreateCall(
2374  MS.MemmoveFn,
2375  {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2376  IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2377  IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2378  I.eraseFromParent();
2379  }
2380 
2381  // Similar to memmove: avoid copying shadow twice.
2382  // This is somewhat unfortunate as it may slowdown small constant memcpys.
2383  // FIXME: consider doing manual inline for small constant sizes and proper
2384  // alignment.
2385  void visitMemCpyInst(MemCpyInst &I) {
2386  IRBuilder<> IRB(&I);
2387  IRB.CreateCall(
2388  MS.MemcpyFn,
2389  {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2390  IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2391  IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2392  I.eraseFromParent();
2393  }
2394 
2395  // Same as memcpy.
2396  void visitMemSetInst(MemSetInst &I) {
2397  IRBuilder<> IRB(&I);
2398  IRB.CreateCall(
2399  MS.MemsetFn,
2400  {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2401  IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
2402  IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2403  I.eraseFromParent();
2404  }
2405 
2406  void visitVAStartInst(VAStartInst &I) {
2407  VAHelper->visitVAStartInst(I);
2408  }
2409 
2410  void visitVACopyInst(VACopyInst &I) {
2411  VAHelper->visitVACopyInst(I);
2412  }
2413 
2414  /// Handle vector store-like intrinsics.
2415  ///
2416  /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2417  /// has 1 pointer argument and 1 vector argument, returns void.
2418  bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2419  IRBuilder<> IRB(&I);
2420  Value* Addr = I.getArgOperand(0);
2421  Value *Shadow = getShadow(&I, 1);
2422  Value *ShadowPtr, *OriginPtr;
2423 
2424  // We don't know the pointer alignment (could be unaligned SSE store!).
2425  // Have to assume to worst case.
2426  std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2427  Addr, IRB, Shadow->getType(), /*Alignment*/ 1, /*isStore*/ true);
2428  IRB.CreateAlignedStore(Shadow, ShadowPtr, 1);
2429 
2431  insertShadowCheck(Addr, &I);
2432 
2433  // FIXME: factor out common code from materializeStores
2434  if (MS.TrackOrigins) IRB.CreateStore(getOrigin(&I, 1), OriginPtr);
2435  return true;
2436  }
2437 
2438  /// Handle vector load-like intrinsics.
2439  ///
2440  /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2441  /// has 1 pointer argument, returns a vector.
2442  bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2443  IRBuilder<> IRB(&I);
2444  Value *Addr = I.getArgOperand(0);
2445 
2446  Type *ShadowTy = getShadowTy(&I);
2447  Value *ShadowPtr, *OriginPtr;
2448  if (PropagateShadow) {
2449  // We don't know the pointer alignment (could be unaligned SSE load!).
2450  // Have to assume to worst case.
2451  unsigned Alignment = 1;
2452  std::tie(ShadowPtr, OriginPtr) =
2453  getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2454  setShadow(&I, IRB.CreateAlignedLoad(ShadowPtr, Alignment, "_msld"));
2455  } else {
2456  setShadow(&I, getCleanShadow(&I));
2457  }
2458 
2460  insertShadowCheck(Addr, &I);
2461 
2462  if (MS.TrackOrigins) {
2463  if (PropagateShadow)
2464  setOrigin(&I, IRB.CreateLoad(OriginPtr));
2465  else
2466  setOrigin(&I, getCleanOrigin());
2467  }
2468  return true;
2469  }
2470 
2471  /// Handle (SIMD arithmetic)-like intrinsics.
2472  ///
2473  /// Instrument intrinsics with any number of arguments of the same type,
2474  /// equal to the return type. The type should be simple (no aggregates or
2475  /// pointers; vectors are fine).
2476  /// Caller guarantees that this intrinsic does not access memory.
2477  bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2478  Type *RetTy = I.getType();
2479  if (!(RetTy->isIntOrIntVectorTy() ||
2480  RetTy->isFPOrFPVectorTy() ||
2481  RetTy->isX86_MMXTy()))
2482  return false;
2483 
2484  unsigned NumArgOperands = I.getNumArgOperands();
2485 
2486  for (unsigned i = 0; i < NumArgOperands; ++i) {
2487  Type *Ty = I.getArgOperand(i)->getType();
2488  if (Ty != RetTy)
2489  return false;
2490  }
2491 
2492  IRBuilder<> IRB(&I);
2493  ShadowAndOriginCombiner SC(this, IRB);
2494  for (unsigned i = 0; i < NumArgOperands; ++i)
2495  SC.Add(I.getArgOperand(i));
2496  SC.Done(&I);
2497 
2498  return true;
2499  }
2500 
2501  /// Heuristically instrument unknown intrinsics.
2502  ///
2503  /// The main purpose of this code is to do something reasonable with all
2504  /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
2505  /// We recognize several classes of intrinsics by their argument types and
2506  /// ModRefBehaviour and apply special intrumentation when we are reasonably
2507  /// sure that we know what the intrinsic does.
2508  ///
2509  /// We special-case intrinsics where this approach fails. See llvm.bswap
2510  /// handling as an example of that.
2511  bool handleUnknownIntrinsic(IntrinsicInst &I) {
2512  unsigned NumArgOperands = I.getNumArgOperands();
2513  if (NumArgOperands == 0)
2514  return false;
2515 
2516  if (NumArgOperands == 2 &&
2517  I.getArgOperand(0)->getType()->isPointerTy() &&
2518  I.getArgOperand(1)->getType()->isVectorTy() &&
2519  I.getType()->isVoidTy() &&
2520  !I.onlyReadsMemory()) {
2521  // This looks like a vector store.
2522  return handleVectorStoreIntrinsic(I);
2523  }
2524 
2525  if (NumArgOperands == 1 &&
2526  I.getArgOperand(0)->getType()->isPointerTy() &&
2527  I.getType()->isVectorTy() &&
2528  I.onlyReadsMemory()) {
2529  // This looks like a vector load.
2530  return handleVectorLoadIntrinsic(I);
2531  }
2532 
2533  if (I.doesNotAccessMemory())
2534  if (maybeHandleSimpleNomemIntrinsic(I))
2535  return true;
2536 
2537  // FIXME: detect and handle SSE maskstore/maskload
2538  return false;
2539  }
2540 
2541  void handleBswap(IntrinsicInst &I) {
2542  IRBuilder<> IRB(&I);
2543  Value *Op = I.getArgOperand(0);
2544  Type *OpType = Op->getType();
2545  Function *BswapFunc = Intrinsic::getDeclaration(
2546  F.getParent(), Intrinsic::bswap, makeArrayRef(&OpType, 1));
2547  setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
2548  setOrigin(&I, getOrigin(Op));
2549  }
2550 
2551  // Instrument vector convert instrinsic.
2552  //
2553  // This function instruments intrinsics like cvtsi2ss:
2554  // %Out = int_xxx_cvtyyy(%ConvertOp)
2555  // or
2556  // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
2557  // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
2558  // number \p Out elements, and (if has 2 arguments) copies the rest of the
2559  // elements from \p CopyOp.
2560  // In most cases conversion involves floating-point value which may trigger a
2561  // hardware exception when not fully initialized. For this reason we require
2562  // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
2563  // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
2564  // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
2565  // return a fully initialized value.
2566  void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements) {
2567  IRBuilder<> IRB(&I);
2568  Value *CopyOp, *ConvertOp;
2569 
2570  switch (I.getNumArgOperands()) {
2571  case 3:
2572  assert(isa<ConstantInt>(I.getArgOperand(2)) && "Invalid rounding mode");
2574  case 2:
2575  CopyOp = I.getArgOperand(0);
2576  ConvertOp = I.getArgOperand(1);
2577  break;
2578  case 1:
2579  ConvertOp = I.getArgOperand(0);
2580  CopyOp = nullptr;
2581  break;
2582  default:
2583  llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
2584  }
2585 
2586  // The first *NumUsedElements* elements of ConvertOp are converted to the
2587  // same number of output elements. The rest of the output is copied from
2588  // CopyOp, or (if not available) filled with zeroes.
2589  // Combine shadow for elements of ConvertOp that are used in this operation,
2590  // and insert a check.
2591  // FIXME: consider propagating shadow of ConvertOp, at least in the case of
2592  // int->any conversion.
2593  Value *ConvertShadow = getShadow(ConvertOp);
2594  Value *AggShadow = nullptr;
2595  if (ConvertOp->getType()->isVectorTy()) {
2596  AggShadow = IRB.CreateExtractElement(
2597  ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
2598  for (int i = 1; i < NumUsedElements; ++i) {
2599  Value *MoreShadow = IRB.CreateExtractElement(
2600  ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
2601  AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
2602  }
2603  } else {
2604  AggShadow = ConvertShadow;
2605  }
2606  assert(AggShadow->getType()->isIntegerTy());
2607  insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
2608 
2609  // Build result shadow by zero-filling parts of CopyOp shadow that come from
2610  // ConvertOp.
2611  if (CopyOp) {
2612  assert(CopyOp->getType() == I.getType());
2613  assert(CopyOp->getType()->isVectorTy());
2614  Value *ResultShadow = getShadow(CopyOp);
2615  Type *EltTy = ResultShadow->getType()->getVectorElementType();
2616  for (int i = 0; i < NumUsedElements; ++i) {
2617  ResultShadow = IRB.CreateInsertElement(
2618  ResultShadow, ConstantInt::getNullValue(EltTy),
2619  ConstantInt::get(IRB.getInt32Ty(), i));
2620  }
2621  setShadow(&I, ResultShadow);
2622  setOrigin(&I, getOrigin(CopyOp));
2623  } else {
2624  setShadow(&I, getCleanShadow(&I));
2625  setOrigin(&I, getCleanOrigin());
2626  }
2627  }
2628 
2629  // Given a scalar or vector, extract lower 64 bits (or less), and return all
2630  // zeroes if it is zero, and all ones otherwise.
2631  Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2632  if (S->getType()->isVectorTy())
2633  S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
2634  assert(S->getType()->getPrimitiveSizeInBits() <= 64);
2635  Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2636  return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2637  }
2638 
2639  // Given a vector, extract its first element, and return all
2640  // zeroes if it is zero, and all ones otherwise.
2641  Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2642  Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
2643  Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
2644  return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2645  }
2646 
2647  Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
2648  Type *T = S->getType();
2649  assert(T->isVectorTy());
2650  Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2651  return IRB.CreateSExt(S2, T);
2652  }
2653 
2654  // Instrument vector shift instrinsic.
2655  //
2656  // This function instruments intrinsics like int_x86_avx2_psll_w.
2657  // Intrinsic shifts %In by %ShiftSize bits.
2658  // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
2659  // size, and the rest is ignored. Behavior is defined even if shift size is
2660  // greater than register (or field) width.
2661  void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
2662  assert(I.getNumArgOperands() == 2);
2663  IRBuilder<> IRB(&I);
2664  // If any of the S2 bits are poisoned, the whole thing is poisoned.
2665  // Otherwise perform the same shift on S1.
2666  Value *S1 = getShadow(&I, 0);
2667  Value *S2 = getShadow(&I, 1);
2668  Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2)
2669  : Lower64ShadowExtend(IRB, S2, getShadowTy(&I));
2670  Value *V1 = I.getOperand(0);
2671  Value *V2 = I.getOperand(1);
2672  Value *Shift = IRB.CreateCall(I.getCalledValue(),
2673  {IRB.CreateBitCast(S1, V1->getType()), V2});
2674  Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
2675  setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2676  setOriginForNaryOp(I);
2677  }
2678 
2679  // Get an X86_MMX-sized vector type.
2680  Type *getMMXVectorTy(unsigned EltSizeInBits) {
2681  const unsigned X86_MMXSizeInBits = 64;
2682  return VectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
2683  X86_MMXSizeInBits / EltSizeInBits);
2684  }
2685 
2686  // Returns a signed counterpart for an (un)signed-saturate-and-pack
2687  // intrinsic.
2688  Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
2689  switch (id) {
2690  case Intrinsic::x86_sse2_packsswb_128:
2691  case Intrinsic::x86_sse2_packuswb_128:
2692  return Intrinsic::x86_sse2_packsswb_128;
2693 
2694  case Intrinsic::x86_sse2_packssdw_128:
2695  case Intrinsic::x86_sse41_packusdw:
2696  return Intrinsic::x86_sse2_packssdw_128;
2697 
2698  case Intrinsic::x86_avx2_packsswb:
2699  case Intrinsic::x86_avx2_packuswb:
2700  return Intrinsic::x86_avx2_packsswb;
2701 
2702  case Intrinsic::x86_avx2_packssdw:
2703  case Intrinsic::x86_avx2_packusdw:
2704  return Intrinsic::x86_avx2_packssdw;
2705 
2706  case Intrinsic::x86_mmx_packsswb:
2707  case Intrinsic::x86_mmx_packuswb:
2708  return Intrinsic::x86_mmx_packsswb;
2709 
2710  case Intrinsic::x86_mmx_packssdw:
2711  return Intrinsic::x86_mmx_packssdw;
2712  default:
2713  llvm_unreachable("unexpected intrinsic id");
2714  }
2715  }
2716 
2717  // Instrument vector pack instrinsic.
2718  //
2719  // This function instruments intrinsics like x86_mmx_packsswb, that
2720  // packs elements of 2 input vectors into half as many bits with saturation.
2721  // Shadow is propagated with the signed variant of the same intrinsic applied
2722  // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
2723  // EltSizeInBits is used only for x86mmx arguments.
2724  void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
2725  assert(I.getNumArgOperands() == 2);
2726  bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2727  IRBuilder<> IRB(&I);
2728  Value *S1 = getShadow(&I, 0);
2729  Value *S2 = getShadow(&I, 1);
2730  assert(isX86_MMX || S1->getType()->isVectorTy());
2731 
2732  // SExt and ICmpNE below must apply to individual elements of input vectors.
2733  // In case of x86mmx arguments, cast them to appropriate vector types and
2734  // back.
2735  Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType();
2736  if (isX86_MMX) {
2737  S1 = IRB.CreateBitCast(S1, T);
2738  S2 = IRB.CreateBitCast(S2, T);
2739  }
2740  Value *S1_ext = IRB.CreateSExt(
2741  IRB.CreateICmpNE(S1, Constant::getNullValue(T)), T);
2742  Value *S2_ext = IRB.CreateSExt(
2743  IRB.CreateICmpNE(S2, Constant::getNullValue(T)), T);
2744  if (isX86_MMX) {
2745  Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
2746  S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
2747  S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
2748  }
2749 
2750  Function *ShadowFn = Intrinsic::getDeclaration(
2751  F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
2752 
2753  Value *S =
2754  IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
2755  if (isX86_MMX) S = IRB.CreateBitCast(S, getShadowTy(&I));
2756  setShadow(&I, S);
2757  setOriginForNaryOp(I);
2758  }
2759 
2760  // Instrument sum-of-absolute-differencies intrinsic.
2761  void handleVectorSadIntrinsic(IntrinsicInst &I) {
2762  const unsigned SignificantBitsPerResultElement = 16;
2763  bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2764  Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType();
2765  unsigned ZeroBitsPerResultElement =
2766  ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
2767 
2768  IRBuilder<> IRB(&I);
2769  Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2770  S = IRB.CreateBitCast(S, ResTy);
2771  S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2772  ResTy);
2773  S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
2774  S = IRB.CreateBitCast(S, getShadowTy(&I));
2775  setShadow(&I, S);
2776  setOriginForNaryOp(I);
2777  }
2778 
2779  // Instrument multiply-add intrinsic.
2780  void handleVectorPmaddIntrinsic(IntrinsicInst &I,
2781  unsigned EltSizeInBits = 0) {
2782  bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2783  Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType();
2784  IRBuilder<> IRB(&I);
2785  Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2786  S = IRB.CreateBitCast(S, ResTy);
2787  S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2788  ResTy);
2789  S = IRB.CreateBitCast(S, getShadowTy(&I));
2790  setShadow(&I, S);
2791  setOriginForNaryOp(I);
2792  }
2793 
2794  // Instrument compare-packed intrinsic.
2795  // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
2796  // all-ones shadow.
2797  void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
2798  IRBuilder<> IRB(&I);
2799  Type *ResTy = getShadowTy(&I);
2800  Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2801  Value *S = IRB.CreateSExt(
2802  IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
2803  setShadow(&I, S);
2804  setOriginForNaryOp(I);
2805  }
2806 
2807  // Instrument compare-scalar intrinsic.
2808  // This handles both cmp* intrinsics which return the result in the first
2809  // element of a vector, and comi* which return the result as i32.
2810  void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
2811  IRBuilder<> IRB(&I);
2812  Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2813  Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
2814  setShadow(&I, S);
2815  setOriginForNaryOp(I);
2816  }
2817 
2818  void handleStmxcsr(IntrinsicInst &I) {
2819  IRBuilder<> IRB(&I);
2820  Value* Addr = I.getArgOperand(0);
2821  Type *Ty = IRB.getInt32Ty();
2822  Value *ShadowPtr =
2823  getShadowOriginPtr(Addr, IRB, Ty, /*Alignment*/ 1, /*isStore*/ true)
2824  .first;
2825 
2826  IRB.CreateStore(getCleanShadow(Ty),
2827  IRB.CreatePointerCast(ShadowPtr, Ty->getPointerTo()));
2828 
2830  insertShadowCheck(Addr, &I);
2831  }
2832 
2833  void handleLdmxcsr(IntrinsicInst &I) {
2834  if (!InsertChecks) return;
2835 
2836  IRBuilder<> IRB(&I);
2837  Value *Addr = I.getArgOperand(0);
2838  Type *Ty = IRB.getInt32Ty();
2839  unsigned Alignment = 1;
2840  Value *ShadowPtr, *OriginPtr;
2841  std::tie(ShadowPtr, OriginPtr) =
2842  getShadowOriginPtr(Addr, IRB, Ty, Alignment, /*isStore*/ false);
2843 
2845  insertShadowCheck(Addr, &I);
2846 
2847  Value *Shadow = IRB.CreateAlignedLoad(ShadowPtr, Alignment, "_ldmxcsr");
2848  Value *Origin =
2849  MS.TrackOrigins ? IRB.CreateLoad(OriginPtr) : getCleanOrigin();
2850  insertShadowCheck(Shadow, Origin, &I);
2851  }
2852 
2853  void handleMaskedStore(IntrinsicInst &I) {
2854  IRBuilder<> IRB(&I);
2855  Value *V = I.getArgOperand(0);
2856  Value *Addr = I.getArgOperand(1);
2857  unsigned Align = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
2858  Value *Mask = I.getArgOperand(3);
2859  Value *Shadow = getShadow(V);
2860 
2861  Value *ShadowPtr;
2862  Value *OriginPtr;
2863  std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2864  Addr, IRB, Shadow->getType(), Align, /*isStore*/ true);
2865 
2866  if (ClCheckAccessAddress) {
2867  insertShadowCheck(Addr, &I);
2868  // Uninitialized mask is kind of like uninitialized address, but not as
2869  // scary.
2870  insertShadowCheck(Mask, &I);
2871  }
2872 
2873  IRB.CreateMaskedStore(Shadow, ShadowPtr, Align, Mask);
2874 
2875  if (MS.TrackOrigins) {
2876  auto &DL = F.getParent()->getDataLayout();
2877  paintOrigin(IRB, getOrigin(V), OriginPtr,
2878  DL.getTypeStoreSize(Shadow->getType()),
2879  std::max(Align, kMinOriginAlignment));
2880  }
2881  }
2882 
2883  bool handleMaskedLoad(IntrinsicInst &I) {
2884  IRBuilder<> IRB(&I);
2885  Value *Addr = I.getArgOperand(0);
2886  unsigned Align = cast<ConstantInt>(I.getArgOperand(1))->getZExtValue();
2887  Value *Mask = I.getArgOperand(2);
2888  Value *PassThru = I.getArgOperand(3);
2889 
2890  Type *ShadowTy = getShadowTy(&I);
2891  Value *ShadowPtr, *OriginPtr;
2892  if (PropagateShadow) {
2893  std::tie(ShadowPtr, OriginPtr) =
2894  getShadowOriginPtr(Addr, IRB, ShadowTy, Align, /*isStore*/ false);
2895  setShadow(&I, IRB.CreateMaskedLoad(ShadowPtr, Align, Mask,
2896  getShadow(PassThru), "_msmaskedld"));
2897  } else {
2898  setShadow(&I, getCleanShadow(&I));
2899  }
2900 
2901  if (ClCheckAccessAddress) {
2902  insertShadowCheck(Addr, &I);
2903  insertShadowCheck(Mask, &I);
2904  }
2905 
2906  if (MS.TrackOrigins) {
2907  if (PropagateShadow) {
2908  // Choose between PassThru's and the loaded value's origins.
2909  Value *MaskedPassThruShadow = IRB.CreateAnd(
2910  getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
2911 
2912  Value *Acc = IRB.CreateExtractElement(
2913  MaskedPassThruShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
2914  for (int i = 1, N = PassThru->getType()->getVectorNumElements(); i < N;
2915  ++i) {
2916  Value *More = IRB.CreateExtractElement(
2917  MaskedPassThruShadow, ConstantInt::get(IRB.getInt32Ty(), i));
2918  Acc = IRB.CreateOr(Acc, More);
2919  }
2920 
2921  Value *Origin = IRB.CreateSelect(
2922  IRB.CreateICmpNE(Acc, Constant::getNullValue(Acc->getType())),
2923  getOrigin(PassThru), IRB.CreateLoad(OriginPtr));
2924 
2925  setOrigin(&I, Origin);
2926  } else {
2927  setOrigin(&I, getCleanOrigin());
2928  }
2929  }
2930  return true;
2931  }
2932 
2933 
2934  void visitIntrinsicInst(IntrinsicInst &I) {
2935  switch (I.getIntrinsicID()) {
2936  case Intrinsic::bswap:
2937  handleBswap(I);
2938  break;
2939  case Intrinsic::masked_store:
2940  handleMaskedStore(I);
2941  break;
2942  case Intrinsic::masked_load:
2943  handleMaskedLoad(I);
2944  break;
2945  case Intrinsic::x86_sse_stmxcsr:
2946  handleStmxcsr(I);
2947  break;
2948  case Intrinsic::x86_sse_ldmxcsr:
2949  handleLdmxcsr(I);
2950  break;
2951  case Intrinsic::x86_avx512_vcvtsd2usi64:
2952  case Intrinsic::x86_avx512_vcvtsd2usi32:
2953  case Intrinsic::x86_avx512_vcvtss2usi64:
2954  case Intrinsic::x86_avx512_vcvtss2usi32:
2955  case Intrinsic::x86_avx512_cvttss2usi64:
2956  case Intrinsic::x86_avx512_cvttss2usi:
2957  case Intrinsic::x86_avx512_cvttsd2usi64:
2958  case Intrinsic::x86_avx512_cvttsd2usi:
2959  case Intrinsic::x86_avx512_cvtusi2ss:
2960  case Intrinsic::x86_avx512_cvtusi642sd:
2961  case Intrinsic::x86_avx512_cvtusi642ss:
2962  case Intrinsic::x86_sse2_cvtsd2si64:
2963  case Intrinsic::x86_sse2_cvtsd2si:
2964  case Intrinsic::x86_sse2_cvtsd2ss:
2965  case Intrinsic::x86_sse2_cvttsd2si64:
2966  case Intrinsic::x86_sse2_cvttsd2si:
2967  case Intrinsic::x86_sse_cvtss2si64:
2968  case Intrinsic::x86_sse_cvtss2si:
2969  case Intrinsic::x86_sse_cvttss2si64:
2970  case Intrinsic::x86_sse_cvttss2si:
2971  handleVectorConvertIntrinsic(I, 1);
2972  break;
2973  case Intrinsic::x86_sse_cvtps2pi:
2974  case Intrinsic::x86_sse_cvttps2pi:
2975  handleVectorConvertIntrinsic(I, 2);
2976  break;
2977 
2978  case Intrinsic::x86_avx512_psll_w_512:
2979  case Intrinsic::x86_avx512_psll_d_512:
2980  case Intrinsic::x86_avx512_psll_q_512:
2981  case Intrinsic::x86_avx512_pslli_w_512:
2982  case Intrinsic::x86_avx512_pslli_d_512:
2983  case Intrinsic::x86_avx512_pslli_q_512:
2984  case Intrinsic::x86_avx512_psrl_w_512:
2985  case Intrinsic::x86_avx512_psrl_d_512:
2986  case Intrinsic::x86_avx512_psrl_q_512:
2987  case Intrinsic::x86_avx512_psra_w_512:
2988  case Intrinsic::x86_avx512_psra_d_512:
2989  case Intrinsic::x86_avx512_psra_q_512:
2990  case Intrinsic::x86_avx512_psrli_w_512:
2991  case Intrinsic::x86_avx512_psrli_d_512:
2992  case Intrinsic::x86_avx512_psrli_q_512:
2993  case Intrinsic::x86_avx512_psrai_w_512:
2994  case Intrinsic::x86_avx512_psrai_d_512:
2995  case Intrinsic::x86_avx512_psrai_q_512:
2996  case Intrinsic::x86_avx512_psra_q_256:
2997  case Intrinsic::x86_avx512_psra_q_128:
2998  case Intrinsic::x86_avx512_psrai_q_256:
2999  case Intrinsic::x86_avx512_psrai_q_128:
3000  case Intrinsic::x86_avx2_psll_w:
3001  case Intrinsic::x86_avx2_psll_d:
3002  case Intrinsic::x86_avx2_psll_q:
3003  case Intrinsic::x86_avx2_pslli_w:
3004  case Intrinsic::x86_avx2_pslli_d:
3005  case Intrinsic::x86_avx2_pslli_q:
3006  case Intrinsic::x86_avx2_psrl_w:
3007  case Intrinsic::x86_avx2_psrl_d:
3008  case Intrinsic::x86_avx2_psrl_q:
3009  case Intrinsic::x86_avx2_psra_w:
3010  case Intrinsic::x86_avx2_psra_d:
3011  case Intrinsic::x86_avx2_psrli_w:
3012  case Intrinsic::x86_avx2_psrli_d:
3013  case Intrinsic::x86_avx2_psrli_q:
3014  case Intrinsic::x86_avx2_psrai_w:
3015  case Intrinsic::x86_avx2_psrai_d:
3016  case Intrinsic::x86_sse2_psll_w:
3017  case Intrinsic::x86_sse2_psll_d:
3018  case Intrinsic::x86_sse2_psll_q:
3019  case Intrinsic::x86_sse2_pslli_w:
3020  case Intrinsic::x86_sse2_pslli_d:
3021  case Intrinsic::x86_sse2_pslli_q:
3022  case Intrinsic::x86_sse2_psrl_w:
3023  case Intrinsic::x86_sse2_psrl_d:
3024  case Intrinsic::x86_sse2_psrl_q:
3025  case Intrinsic::x86_sse2_psra_w:
3026  case Intrinsic::x86_sse2_psra_d:
3027  case Intrinsic::x86_sse2_psrli_w:
3028  case Intrinsic::x86_sse2_psrli_d:
3029  case Intrinsic::x86_sse2_psrli_q:
3030  case Intrinsic::x86_sse2_psrai_w:
3031  case Intrinsic::x86_sse2_psrai_d:
3032  case Intrinsic::x86_mmx_psll_w:
3033  case Intrinsic::x86_mmx_psll_d:
3034  case Intrinsic::x86_mmx_psll_q:
3035  case Intrinsic::x86_mmx_pslli_w:
3036  case Intrinsic::x86_mmx_pslli_d:
3037  case Intrinsic::x86_mmx_pslli_q:
3038  case Intrinsic::x86_mmx_psrl_w:
3039  case Intrinsic::x86_mmx_psrl_d:
3040  case Intrinsic::x86_mmx_psrl_q:
3041  case Intrinsic::x86_mmx_psra_w:
3042  case Intrinsic::x86_mmx_psra_d:
3043  case Intrinsic::x86_mmx_psrli_w:
3044  case Intrinsic::x86_mmx_psrli_d:
3045  case Intrinsic::x86_mmx_psrli_q:
3046  case Intrinsic::x86_mmx_psrai_w:
3047  case Intrinsic::x86_mmx_psrai_d:
3048  handleVectorShiftIntrinsic(I, /* Variable */ false);
3049  break;
3050  case Intrinsic::x86_avx2_psllv_d:
3051  case Intrinsic::x86_avx2_psllv_d_256:
3052  case Intrinsic::x86_avx512_psllv_d_512:
3053  case Intrinsic::x86_avx2_psllv_q:
3054  case Intrinsic::x86_avx2_psllv_q_256:
3055  case Intrinsic::x86_avx512_psllv_q_512:
3056  case Intrinsic::x86_avx2_psrlv_d:
3057  case Intrinsic::x86_avx2_psrlv_d_256:
3058  case Intrinsic::x86_avx512_psrlv_d_512:
3059  case Intrinsic::x86_avx2_psrlv_q:
3060  case Intrinsic::x86_avx2_psrlv_q_256:
3061  case Intrinsic::x86_avx512_psrlv_q_512:
3062  case Intrinsic::x86_avx2_psrav_d:
3063  case Intrinsic::x86_avx2_psrav_d_256:
3064  case Intrinsic::x86_avx512_psrav_d_512:
3065  case Intrinsic::x86_avx512_psrav_q_128:
3066  case Intrinsic::x86_avx512_psrav_q_256:
3067  case Intrinsic::x86_avx512_psrav_q_512:
3068  handleVectorShiftIntrinsic(I, /* Variable */ true);
3069  break;
3070 
3071  case Intrinsic::x86_sse2_packsswb_128:
3072  case Intrinsic::x86_sse2_packssdw_128:
3073  case Intrinsic::x86_sse2_packuswb_128:
3074  case Intrinsic::x86_sse41_packusdw:
3075  case Intrinsic::x86_avx2_packsswb:
3076  case Intrinsic::x86_avx2_packssdw:
3077  case Intrinsic::x86_avx2_packuswb:
3078  case Intrinsic::x86_avx2_packusdw:
3079  handleVectorPackIntrinsic(I);
3080  break;
3081 
3082  case Intrinsic::x86_mmx_packsswb:
3083  case Intrinsic::x86_mmx_packuswb:
3084  handleVectorPackIntrinsic(I, 16);
3085  break;
3086 
3087  case Intrinsic::x86_mmx_packssdw:
3088  handleVectorPackIntrinsic(I, 32);
3089  break;
3090 
3091  case Intrinsic::x86_mmx_psad_bw:
3092  case Intrinsic::x86_sse2_psad_bw:
3093  case Intrinsic::x86_avx2_psad_bw:
3094  handleVectorSadIntrinsic(I);
3095  break;
3096 
3097  case Intrinsic::x86_sse2_pmadd_wd:
3098  case Intrinsic::x86_avx2_pmadd_wd:
3099  case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
3100  case Intrinsic::x86_avx2_pmadd_ub_sw:
3101  handleVectorPmaddIntrinsic(I);
3102  break;
3103 
3104  case Intrinsic::x86_ssse3_pmadd_ub_sw:
3105  handleVectorPmaddIntrinsic(I, 8);
3106  break;
3107 
3108  case Intrinsic::x86_mmx_pmadd_wd:
3109  handleVectorPmaddIntrinsic(I, 16);
3110  break;
3111 
3112  case Intrinsic::x86_sse_cmp_ss:
3113  case Intrinsic::x86_sse2_cmp_sd:
3114  case Intrinsic::x86_sse_comieq_ss:
3115  case Intrinsic::x86_sse_comilt_ss:
3116  case Intrinsic::x86_sse_comile_ss:
3117  case Intrinsic::x86_sse_comigt_ss:
3118  case Intrinsic::x86_sse_comige_ss:
3119  case Intrinsic::x86_sse_comineq_ss:
3120  case Intrinsic::x86_sse_ucomieq_ss:
3121  case Intrinsic::x86_sse_ucomilt_ss:
3122  case Intrinsic::x86_sse_ucomile_ss:
3123  case Intrinsic::x86_sse_ucomigt_ss:
3124  case Intrinsic::x86_sse_ucomige_ss:
3125  case Intrinsic::x86_sse_ucomineq_ss:
3126  case Intrinsic::x86_sse2_comieq_sd:
3127  case Intrinsic::x86_sse2_comilt_sd:
3128  case Intrinsic::x86_sse2_comile_sd:
3129  case Intrinsic::x86_sse2_comigt_sd:
3130  case Intrinsic::x86_sse2_comige_sd:
3131  case Intrinsic::x86_sse2_comineq_sd:
3132  case Intrinsic::x86_sse2_ucomieq_sd:
3133  case Intrinsic::x86_sse2_ucomilt_sd:
3134  case Intrinsic::x86_sse2_ucomile_sd:
3135  case Intrinsic::x86_sse2_ucomigt_sd:
3136  case Intrinsic::x86_sse2_ucomige_sd:
3137  case Intrinsic::x86_sse2_ucomineq_sd:
3138  handleVectorCompareScalarIntrinsic(I);
3139  break;
3140 
3141  case Intrinsic::x86_sse_cmp_ps:
3142  case Intrinsic::x86_sse2_cmp_pd:
3143  // FIXME: For x86_avx_cmp_pd_256 and x86_avx_cmp_ps_256 this function
3144  // generates reasonably looking IR that fails in the backend with "Do not
3145  // know how to split the result of this operator!".
3146  handleVectorComparePackedIntrinsic(I);
3147  break;
3148 
3149  case Intrinsic::is_constant:
3150  // The result of llvm.is.constant() is always defined.
3151  setShadow(&I, getCleanShadow(&I));
3152  setOrigin(&I, getCleanOrigin());
3153  break;
3154 
3155  default:
3156  if (!handleUnknownIntrinsic(I))
3157  visitInstruction(I);
3158  break;
3159  }
3160  }
3161 
3162  void visitCallSite(CallSite CS) {
3163  Instruction &I = *CS.getInstruction();
3164  assert(!I.getMetadata("nosanitize"));
3165  assert((CS.isCall() || CS.isInvoke()) && "Unknown type of CallSite");
3166  if (CS.isCall()) {
3167  CallInst *Call = cast<CallInst>(&I);
3168 
3169  // For inline asm, do the usual thing: check argument shadow and mark all
3170  // outputs as clean. Note that any side effects of the inline asm that are
3171  // not immediately visible in its constraints are not handled.
3172  if (Call->isInlineAsm()) {
3173  if (ClHandleAsmConservative && MS.CompileKernel)
3174  visitAsmInstruction(I);
3175  else
3176  visitInstruction(I);
3177  return;
3178  }
3179 
3180  assert(!isa<IntrinsicInst>(&I) && "intrinsics are handled elsewhere");
3181 
3182  // We are going to insert code that relies on the fact that the callee
3183  // will become a non-readonly function after it is instrumented by us. To
3184  // prevent this code from being optimized out, mark that function
3185  // non-readonly in advance.
3186  if (Function *Func = Call->getCalledFunction()) {
3187  // Clear out readonly/readnone attributes.
3188  AttrBuilder B;
3189  B.addAttribute(Attribute::ReadOnly)
3190  .addAttribute(Attribute::ReadNone);
3192  }
3193 
3195  }
3196  IRBuilder<> IRB(&I);
3197 
3198  unsigned ArgOffset = 0;
3199  LLVM_DEBUG(dbgs() << " CallSite: " << I << "\n");
3200  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
3201  ArgIt != End; ++ArgIt) {
3202  Value *A = *ArgIt;
3203  unsigned i = ArgIt - CS.arg_begin();
3204  if (!A->getType()->isSized()) {
3205  LLVM_DEBUG(dbgs() << "Arg " << i << " is not sized: " << I << "\n");
3206  continue;
3207  }
3208  unsigned Size = 0;
3209  Value *Store = nullptr;
3210  // Compute the Shadow for arg even if it is ByVal, because
3211  // in that case getShadow() will copy the actual arg shadow to
3212  // __msan_param_tls.
3213  Value *ArgShadow = getShadow(A);
3214  Value *ArgShadowBase = getShadowPtrForArgument(A, IRB, ArgOffset);
3215  LLVM_DEBUG(dbgs() << " Arg#" << i << ": " << *A
3216  << " Shadow: " << *ArgShadow << "\n");
3217  bool ArgIsInitialized = false;
3218  const DataLayout &DL = F.getParent()->getDataLayout();
3219  if (CS.paramHasAttr(i, Attribute::ByVal)) {
3220  assert(A->getType()->isPointerTy() &&
3221  "ByVal argument is not a pointer!");
3222  Size = DL.getTypeAllocSize(A->getType()->getPointerElementType());
3223  if (ArgOffset + Size > kParamTLSSize) break;
3224  unsigned ParamAlignment = CS.getParamAlignment(i);
3225  unsigned Alignment = std::min(ParamAlignment, kShadowTLSAlignment);
3226  Value *AShadowPtr =
3227  getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), Alignment,
3228  /*isStore*/ false)
3229  .first;
3230 
3231  Store = IRB.CreateMemCpy(ArgShadowBase, Alignment, AShadowPtr,
3232  Alignment, Size);
3233  // TODO(glider): need to copy origins.
3234  } else {
3235  Size = DL.getTypeAllocSize(A->getType());
3236  if (ArgOffset + Size > kParamTLSSize) break;
3237  Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
3238  kShadowTLSAlignment);
3239  Constant *Cst = dyn_cast<Constant>(ArgShadow);
3240  if (Cst && Cst->isNullValue()) ArgIsInitialized = true;
3241  }
3242  if (MS.TrackOrigins && !ArgIsInitialized)
3243  IRB.CreateStore(getOrigin(A),
3244  getOriginPtrForArgument(A, IRB, ArgOffset));
3245  (void)Store;
3246  assert(Size != 0 && Store != nullptr);
3247  LLVM_DEBUG(dbgs() << " Param:" << *Store << "\n");
3248  ArgOffset += alignTo(Size, 8);
3249  }
3250  LLVM_DEBUG(dbgs() << " done with call args\n");
3251 
3252  FunctionType *FT = CS.getFunctionType();
3253  if (FT->isVarArg()) {
3254  VAHelper->visitCallSite(CS, IRB);
3255  }
3256 
3257  // Now, get the shadow for the RetVal.
3258  if (!I.getType()->isSized()) return;
3259  // Don't emit the epilogue for musttail call returns.
3260  if (CS.isCall() && cast<CallInst>(&I)->isMustTailCall()) return;
3261  IRBuilder<> IRBBefore(&I);
3262  // Until we have full dynamic coverage, make sure the retval shadow is 0.
3263  Value *Base = getShadowPtrForRetval(&I, IRBBefore);
3264  IRBBefore.CreateAlignedStore(getCleanShadow(&I), Base, kShadowTLSAlignment);
3265  BasicBlock::iterator NextInsn;
3266  if (CS.isCall()) {
3267  NextInsn = ++I.getIterator();
3268  assert(NextInsn != I.getParent()->end());
3269  } else {
3270  BasicBlock *NormalDest = cast<InvokeInst>(&I)->getNormalDest();
3271  if (!NormalDest->getSinglePredecessor()) {
3272  // FIXME: this case is tricky, so we are just conservative here.
3273  // Perhaps we need to split the edge between this BB and NormalDest,
3274  // but a naive attempt to use SplitEdge leads to a crash.
3275  setShadow(&I, getCleanShadow(&I));
3276  setOrigin(&I, getCleanOrigin());
3277  return;
3278  }
3279  // FIXME: NextInsn is likely in a basic block that has not been visited yet.
3280  // Anything inserted there will be instrumented by MSan later!
3281  NextInsn = NormalDest->getFirstInsertionPt();
3282  assert(NextInsn != NormalDest->end() &&
3283  "Could not find insertion point for retval shadow load");
3284  }
3285  IRBuilder<> IRBAfter(&*NextInsn);
3286  Value *RetvalShadow =
3287  IRBAfter.CreateAlignedLoad(getShadowPtrForRetval(&I, IRBAfter),
3288  kShadowTLSAlignment, "_msret");
3289  setShadow(&I, RetvalShadow);
3290  if (MS.TrackOrigins)
3291  setOrigin(&I, IRBAfter.CreateLoad(getOriginPtrForRetval(IRBAfter)));
3292  }
3293 
3294  bool isAMustTailRetVal(Value *RetVal) {
3295  if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
3296  RetVal = I->getOperand(0);
3297  }
3298  if (auto *I = dyn_cast<CallInst>(RetVal)) {
3299  return I->isMustTailCall();
3300  }
3301  return false;
3302  }
3303 
3304  void visitReturnInst(ReturnInst &I) {
3305  IRBuilder<> IRB(&I);
3306  Value *RetVal = I.getReturnValue();
3307  if (!RetVal) return;
3308  // Don't emit the epilogue for musttail call returns.
3309  if (isAMustTailRetVal(RetVal)) return;
3310  Value *ShadowPtr = getShadowPtrForRetval(RetVal, IRB);
3311  if (CheckReturnValue) {
3312  insertShadowCheck(RetVal, &I);
3313  Value *Shadow = getCleanShadow(RetVal);
3314  IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
3315  } else {
3316  Value *Shadow = getShadow(RetVal);
3317  IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
3318  if (MS.TrackOrigins)
3319  IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB));
3320  }
3321  }
3322 
3323  void visitPHINode(PHINode &I) {
3324  IRBuilder<> IRB(&I);
3325  if (!PropagateShadow) {
3326  setShadow(&I, getCleanShadow(&I));
3327  setOrigin(&I, getCleanOrigin());
3328  return;
3329  }
3330 
3331  ShadowPHINodes.push_back(&I);
3332  setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
3333  "_msphi_s"));
3334  if (MS.TrackOrigins)
3335  setOrigin(&I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(),
3336  "_msphi_o"));
3337  }
3338 
3339  Value *getLocalVarDescription(AllocaInst &I) {
3340  SmallString<2048> StackDescriptionStorage;
3341  raw_svector_ostream StackDescription(StackDescriptionStorage);
3342  // We create a string with a description of the stack allocation and
3343  // pass it into __msan_set_alloca_origin.
3344  // It will be printed by the run-time if stack-originated UMR is found.
3345  // The first 4 bytes of the string are set to '----' and will be replaced
3346  // by __msan_va_arg_overflow_size_tls at the first call.
3347  StackDescription << "----" << I.getName() << "@" << F.getName();
3349  StackDescription.str());
3350  }
3351 
3352  void instrumentAllocaUserspace(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
3353  if (PoisonStack && ClPoisonStackWithCall) {
3354  IRB.CreateCall(MS.MsanPoisonStackFn,
3355  {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
3356  } else {
3357  Value *ShadowBase, *OriginBase;
3358  std::tie(ShadowBase, OriginBase) =
3359  getShadowOriginPtr(&I, IRB, IRB.getInt8Ty(), 1, /*isStore*/ true);
3360 
3361  Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0);
3362  IRB.CreateMemSet(ShadowBase, PoisonValue, Len, I.getAlignment());
3363  }
3364 
3365  if (PoisonStack && MS.TrackOrigins) {
3366  Value *Descr = getLocalVarDescription(I);
3367  IRB.CreateCall(MS.MsanSetAllocaOrigin4Fn,
3368  {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
3369  IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy()),
3370  IRB.CreatePointerCast(&F, MS.IntptrTy)});
3371  }
3372  }
3373 
3374  void instrumentAllocaKmsan(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
3375  Value *Descr = getLocalVarDescription(I);
3376  if (PoisonStack) {
3377  IRB.CreateCall(MS.MsanPoisonAllocaFn,
3378  {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
3379  IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy())});
3380  } else {
3381  IRB.CreateCall(MS.MsanUnpoisonAllocaFn,
3382  {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
3383  }
3384  }
3385 
3386  void visitAllocaInst(AllocaInst &I) {
3387  setShadow(&I, getCleanShadow(&I));
3388  setOrigin(&I, getCleanOrigin());
3389  IRBuilder<> IRB(I.getNextNode());
3390  const DataLayout &DL = F.getParent()->getDataLayout();
3391  uint64_t TypeSize = DL.getTypeAllocSize(I.getAllocatedType());
3392  Value *Len = ConstantInt::get(MS.IntptrTy, TypeSize);
3393  if (I.isArrayAllocation())
3394  Len = IRB.CreateMul(Len, I.getArraySize());
3395 
3396  if (MS.CompileKernel)
3397  instrumentAllocaKmsan(I, IRB, Len);
3398  else
3399  instrumentAllocaUserspace(I, IRB, Len);
3400  }
3401 
3402  void visitSelectInst(SelectInst& I) {
3403  IRBuilder<> IRB(&I);
3404  // a = select b, c, d
3405  Value *B = I.getCondition();
3406  Value *C = I.getTrueValue();
3407  Value *D = I.getFalseValue();
3408  Value *Sb = getShadow(B);
3409  Value *Sc = getShadow(C);
3410  Value *Sd = getShadow(D);
3411 
3412  // Result shadow if condition shadow is 0.
3413  Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
3414  Value *Sa1;
3415  if (I.getType()->isAggregateType()) {
3416  // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
3417  // an extra "select". This results in much more compact IR.
3418  // Sa = select Sb, poisoned, (select b, Sc, Sd)
3419  Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
3420  } else {
3421  // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
3422  // If Sb (condition is poisoned), look for bits in c and d that are equal
3423  // and both unpoisoned.
3424  // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
3425 
3426  // Cast arguments to shadow-compatible type.
3427  C = CreateAppToShadowCast(IRB, C);
3428  D = CreateAppToShadowCast(IRB, D);
3429 
3430  // Result shadow if condition shadow is 1.
3431  Sa1 = IRB.CreateOr(IRB.CreateXor(C, D), IRB.CreateOr(Sc, Sd));
3432  }
3433  Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
3434  setShadow(&I, Sa);
3435  if (MS.TrackOrigins) {
3436  // Origins are always i32, so any vector conditions must be flattened.
3437  // FIXME: consider tracking vector origins for app vectors?
3438  if (B->getType()->isVectorTy()) {
3439  Type *FlatTy = getShadowTyNoVec(B->getType());
3440  B = IRB.CreateICmpNE(IRB.CreateBitCast(B, FlatTy),
3441  ConstantInt::getNullValue(FlatTy));
3442  Sb = IRB.CreateICmpNE(IRB.CreateBitCast(Sb, FlatTy),
3443  ConstantInt::getNullValue(FlatTy));
3444  }
3445  // a = select b, c, d
3446  // Oa = Sb ? Ob : (b ? Oc : Od)
3447  setOrigin(
3448  &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()),
3449  IRB.CreateSelect(B, getOrigin(I.getTrueValue()),
3450  getOrigin(I.getFalseValue()))));
3451  }
3452  }
3453 
3454  void visitLandingPadInst(LandingPadInst &I) {
3455  // Do nothing.
3456  // See https://github.com/google/sanitizers/issues/504
3457  setShadow(&I, getCleanShadow(&I));
3458  setOrigin(&I, getCleanOrigin());
3459  }
3460 
3461  void visitCatchSwitchInst(CatchSwitchInst &I) {
3462  setShadow(&I, getCleanShadow(&I));
3463  setOrigin(&I, getCleanOrigin());
3464  }
3465 
3466  void visitFuncletPadInst(FuncletPadInst &I) {
3467  setShadow(&I, getCleanShadow(&I));
3468  setOrigin(&I, getCleanOrigin());
3469  }
3470 
3471  void visitGetElementPtrInst(GetElementPtrInst &I) {
3472  handleShadowOr(I);
3473  }
3474 
3475  void visitExtractValueInst(ExtractValueInst &I) {
3476  IRBuilder<> IRB(&I);
3477  Value *Agg = I.getAggregateOperand();
3478  LLVM_DEBUG(dbgs() << "ExtractValue: " << I << "\n");
3479  Value *AggShadow = getShadow(Agg);
3480  LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
3481  Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
3482  LLVM_DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n");
3483  setShadow(&I, ResShadow);
3484  setOriginForNaryOp(I);
3485  }
3486 
3487  void visitInsertValueInst(InsertValueInst &I) {
3488  IRBuilder<> IRB(&I);
3489  LLVM_DEBUG(dbgs() << "InsertValue: " << I << "\n");
3490  Value *AggShadow = getShadow(I.getAggregateOperand());
3491  Value *InsShadow = getShadow(I.getInsertedValueOperand());
3492  LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
3493  LLVM_DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n");
3494  Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
3495  LLVM_DEBUG(dbgs() << " Res: " << *Res << "\n");
3496  setShadow(&I, Res);
3497  setOriginForNaryOp(I);
3498  }
3499 
3500  void dumpInst(Instruction &I) {
3501  if (CallInst *CI = dyn_cast<CallInst>(&I)) {
3502  errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
3503  } else {
3504  errs() << "ZZZ " << I.getOpcodeName() << "\n";
3505  }
3506  errs() << "QQQ " << I << "\n";
3507  }
3508 
3509  void visitResumeInst(ResumeInst &I) {
3510  LLVM_DEBUG(dbgs() << "Resume: " << I << "\n");
3511  // Nothing to do here.
3512  }
3513 
3514  void visitCleanupReturnInst(CleanupReturnInst &CRI) {
3515  LLVM_DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
3516  // Nothing to do here.
3517  }
3518 
3519  void visitCatchReturnInst(CatchReturnInst &CRI) {
3520  LLVM_DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
3521  // Nothing to do here.
3522  }
3523 
3524  void instrumentAsmArgument(Value *Operand, Instruction &I, IRBuilder<> &IRB,
3525  const DataLayout &DL, bool isOutput) {
3526  // For each assembly argument, we check its value for being initialized.
3527  // If the argument is a pointer, we assume it points to a single element
3528  // of the corresponding type (or to a 8-byte word, if the type is unsized).
3529  // Each such pointer is instrumented with a call to the runtime library.
3530  Type *OpType = Operand->getType();
3531  // Check the operand value itself.
3532  insertShadowCheck(Operand, &I);
3533  if (!OpType->isPointerTy() || !isOutput) {
3534  assert(!isOutput);
3535  return;
3536  }
3537  Type *ElType = OpType->getPointerElementType();
3538  if (!ElType->isSized())
3539  return;
3540  int Size = DL.getTypeStoreSize(ElType);
3541  Value *Ptr = IRB.CreatePointerCast(Operand, IRB.getInt8PtrTy());
3542  Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
3543  IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Ptr, SizeVal});
3544  }
3545 
3546  /// Get the number of output arguments returned by pointers.
3547  int getNumOutputArgs(InlineAsm *IA, CallInst *CI) {
3548  int NumRetOutputs = 0;
3549  int NumOutputs = 0;
3550  Type *RetTy = dyn_cast<Value>(CI)->getType();
3551  if (!RetTy->isVoidTy()) {
3552  // Register outputs are returned via the CallInst return value.
3553  StructType *ST = dyn_cast_or_null<StructType>(RetTy);
3554  if (ST)
3555  NumRetOutputs = ST->getNumElements();
3556  else
3557  NumRetOutputs = 1;
3558  }
3560  for (size_t i = 0, n = Constraints.size(); i < n; i++) {
3561  InlineAsm::ConstraintInfo Info = Constraints[i];
3562  switch (Info.Type) {
3563  case InlineAsm::isOutput:
3564  NumOutputs++;
3565  break;
3566  default:
3567  break;
3568  }
3569  }
3570  return NumOutputs - NumRetOutputs;
3571  }
3572 
3573  void visitAsmInstruction(Instruction &I) {
3574  // Conservative inline assembly handling: check for poisoned shadow of
3575  // asm() arguments, then unpoison the result and all the memory locations
3576  // pointed to by those arguments.
3577  // An inline asm() statement in C++ contains lists of input and output
3578  // arguments used by the assembly code. These are mapped to operands of the
3579  // CallInst as follows:
3580  // - nR register outputs ("=r) are returned by value in a single structure
3581  // (SSA value of the CallInst);
3582  // - nO other outputs ("=m" and others) are returned by pointer as first
3583  // nO operands of the CallInst;
3584  // - nI inputs ("r", "m" and others) are passed to CallInst as the
3585  // remaining nI operands.
3586  // The total number of asm() arguments in the source is nR+nO+nI, and the
3587  // corresponding CallInst has nO+nI+1 operands (the last operand is the
3588  // function to be called).
3589  const DataLayout &DL = F.getParent()->getDataLayout();
3590  CallInst *CI = dyn_cast<CallInst>(&I);
3591  IRBuilder<> IRB(&I);
3592  InlineAsm *IA = cast<InlineAsm>(CI->getCalledValue());
3593  int OutputArgs = getNumOutputArgs(IA, CI);
3594  // The last operand of a CallInst is the function itself.
3595  int NumOperands = CI->getNumOperands() - 1;
3596 
3597  // Check input arguments. Doing so before unpoisoning output arguments, so
3598  // that we won't overwrite uninit values before checking them.
3599  for (int i = OutputArgs; i < NumOperands; i++) {
3600  Value *Operand = CI->getOperand(i);
3601  instrumentAsmArgument(Operand, I, IRB, DL, /*isOutput*/ false);
3602  }
3603  // Unpoison output arguments. This must happen before the actual InlineAsm
3604  // call, so that the shadow for memory published in the asm() statement
3605  // remains valid.
3606  for (int i = 0; i < OutputArgs; i++) {
3607  Value *Operand = CI->getOperand(i);
3608  instrumentAsmArgument(Operand, I, IRB, DL, /*isOutput*/ true);
3609  }
3610 
3611  setShadow(&I, getCleanShadow(&I));
3612  setOrigin(&I, getCleanOrigin());
3613  }
3614 
3615  void visitInstruction(Instruction &I) {
3616  // Everything else: stop propagating and check for poisoned shadow.
3618  dumpInst(I);
3619  LLVM_DEBUG(dbgs() << "DEFAULT: " << I << "\n");
3620  for (size_t i = 0, n = I.getNumOperands(); i < n; i++) {
3621  Value *Operand = I.getOperand(i);
3622  if (Operand->getType()->isSized())
3623  insertShadowCheck(Operand, &I);
3624  }
3625  setShadow(&I, getCleanShadow(&I));
3626  setOrigin(&I, getCleanOrigin());
3627  }
3628 };
3629 
3630 /// AMD64-specific implementation of VarArgHelper.
3631 struct VarArgAMD64Helper : public VarArgHelper {
3632  // An unfortunate workaround for asymmetric lowering of va_arg stuff.
3633  // See a comment in visitCallSite for more details.
3634  static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7
3635  static const unsigned AMD64FpEndOffsetSSE = 176;
3636  // If SSE is disabled, fp_offset in va_list is zero.
3637  static const unsigned AMD64FpEndOffsetNoSSE = AMD64GpEndOffset;
3638 
3639  unsigned AMD64FpEndOffset;
3640  Function &F;
3641  MemorySanitizer &MS;
3642  MemorySanitizerVisitor &MSV;
3643  Value *VAArgTLSCopy = nullptr;
3644  Value *VAArgTLSOriginCopy = nullptr;
3645  Value *VAArgOverflowSize = nullptr;
3646 
3647  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3648 
3649  enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
3650 
3651  VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
3652  MemorySanitizerVisitor &MSV)
3653  : F(F), MS(MS), MSV(MSV) {
3654  AMD64FpEndOffset = AMD64FpEndOffsetSSE;
3655  for (const auto &Attr : F.getAttributes().getFnAttributes()) {
3656  if (Attr.isStringAttribute() &&
3657  (Attr.getKindAsString() == "target-features")) {
3658  if (Attr.getValueAsString().contains("-sse"))
3659  AMD64FpEndOffset = AMD64FpEndOffsetNoSSE;
3660  break;
3661  }
3662  }
3663  }
3664 
3665  ArgKind classifyArgument(Value* arg) {
3666  // A very rough approximation of X86_64 argument classification rules.
3667  Type *T = arg->getType();
3668  if (T->isFPOrFPVectorTy() || T->isX86_MMXTy())
3669  return AK_FloatingPoint;
3670  if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
3671  return AK_GeneralPurpose;
3672  if (T->isPointerTy())
3673  return AK_GeneralPurpose;
3674  return AK_Memory;
3675  }
3676 
3677  // For VarArg functions, store the argument shadow in an ABI-specific format
3678  // that corresponds to va_list layout.
3679  // We do this because Clang lowers va_arg in the frontend, and this pass
3680  // only sees the low level code that deals with va_list internals.
3681  // A much easier alternative (provided that Clang emits va_arg instructions)
3682  // would have been to associate each live instance of va_list with a copy of
3683  // MSanParamTLS, and extract shadow on va_arg() call in the argument list
3684  // order.
3685  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3686  unsigned GpOffset = 0;
3687  unsigned FpOffset = AMD64GpEndOffset;
3688  unsigned OverflowOffset = AMD64FpEndOffset;
3689  const DataLayout &DL = F.getParent()->getDataLayout();
3690  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
3691  ArgIt != End; ++ArgIt) {
3692  Value *A = *ArgIt;
3693  unsigned ArgNo = CS.getArgumentNo(ArgIt);
3694  bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
3695  bool IsByVal = CS.paramHasAttr(ArgNo, Attribute::ByVal);
3696  if (IsByVal) {
3697  // ByVal arguments always go to the overflow area.
3698  // Fixed arguments passed through the overflow area will be stepped
3699  // over by va_start, so don't count them towards the offset.
3700  if (IsFixed)
3701  continue;
3702  assert(A->getType()->isPointerTy());
3703  Type *RealTy = A->getType()->getPointerElementType();
3704  uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
3705  Value *ShadowBase = getShadowPtrForVAArgument(
3706  RealTy, IRB, OverflowOffset, alignTo(ArgSize, 8));
3707  Value *OriginBase = nullptr;
3708  if (MS.TrackOrigins)
3709  OriginBase = getOriginPtrForVAArgument(RealTy, IRB, OverflowOffset);
3710  OverflowOffset += alignTo(ArgSize, 8);
3711  if (!ShadowBase)
3712  continue;
3713  Value *ShadowPtr, *OriginPtr;
3714  std::tie(ShadowPtr, OriginPtr) =
3715  MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), kShadowTLSAlignment,
3716  /*isStore*/ false);
3717 
3718  IRB.CreateMemCpy(ShadowBase, kShadowTLSAlignment, ShadowPtr,
3719  kShadowTLSAlignment, ArgSize);
3720  if (MS.TrackOrigins)
3721  IRB.CreateMemCpy(OriginBase, kShadowTLSAlignment, OriginPtr,
3722  kShadowTLSAlignment, ArgSize);
3723  } else {
3724  ArgKind AK = classifyArgument(A);
3725  if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset)
3726  AK = AK_Memory;
3727  if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset)
3728  AK = AK_Memory;
3729  Value *ShadowBase, *OriginBase = nullptr;
3730  switch (AK) {
3731  case AK_GeneralPurpose:
3732  ShadowBase =
3733  getShadowPtrForVAArgument(A->getType(), IRB, GpOffset, 8);
3734  if (MS.TrackOrigins)
3735  OriginBase =
3736  getOriginPtrForVAArgument(A->getType(), IRB, GpOffset);
3737  GpOffset += 8;
3738  break;
3739  case AK_FloatingPoint:
3740  ShadowBase =
3741  getShadowPtrForVAArgument(A->getType(), IRB, FpOffset, 16);
3742  if (MS.TrackOrigins)
3743  OriginBase =
3744  getOriginPtrForVAArgument(A->getType(), IRB, FpOffset);
3745  FpOffset += 16;
3746  break;
3747  case AK_Memory:
3748  if (IsFixed)
3749  continue;
3750  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
3751  ShadowBase =
3752  getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset, 8);
3753  if (MS.TrackOrigins)
3754  OriginBase =
3755  getOriginPtrForVAArgument(A->getType(), IRB, OverflowOffset);
3756  OverflowOffset += alignTo(ArgSize, 8);
3757  }
3758  // Take fixed arguments into account for GpOffset and FpOffset,
3759  // but don't actually store shadows for them.
3760  // TODO(glider): don't call get*PtrForVAArgument() for them.
3761  if (IsFixed)
3762  continue;
3763  if (!ShadowBase)
3764  continue;
3765  Value *Shadow = MSV.getShadow(A);
3766  IRB.CreateAlignedStore(Shadow, ShadowBase, kShadowTLSAlignment);
3767  if (MS.TrackOrigins) {
3768  Value *Origin = MSV.getOrigin(A);
3769  unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
3770  MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
3771  std::max(kShadowTLSAlignment, kMinOriginAlignment));
3772  }
3773  }
3774  }
3775  Constant *OverflowSize =
3776  ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
3777  IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
3778  }
3779 
3780  /// Compute the shadow address for a given va_arg.
3781  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
3782  unsigned ArgOffset, unsigned ArgSize) {
3783  // Make sure we don't overflow __msan_va_arg_tls.
3784  if (ArgOffset + ArgSize > kParamTLSSize)
3785  return nullptr;
3786  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
3787  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3788  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
3789  "_msarg_va_s");
3790  }
3791 
3792  /// Compute the origin address for a given va_arg.
3793  Value *getOriginPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, int ArgOffset) {
3794  Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
3795  // getOriginPtrForVAArgument() is always called after
3796  // getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
3797  // overflow.
3798  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3799  return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
3800  "_msarg_va_o");
3801  }
3802 
3803  void unpoisonVAListTagForInst(IntrinsicInst &I) {
3804  IRBuilder<> IRB(&I);
3805  Value *VAListTag = I.getArgOperand(0);
3806  Value *ShadowPtr, *OriginPtr;
3807  unsigned Alignment = 8;
3808  std::tie(ShadowPtr, OriginPtr) =
3809  MSV.getShadowOriginPtr(VAListTag, IRB, IRB.getInt8Ty(), Alignment,
3810  /*isStore*/ true);
3811 
3812  // Unpoison the whole __va_list_tag.
3813  // FIXME: magic ABI constants.
3814  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3815  /* size */ 24, Alignment, false);
3816  // We shouldn't need to zero out the origins, as they're only checked for
3817  // nonzero shadow.
3818  }
3819 
3820  void visitVAStartInst(VAStartInst &I) override {
3822  return;
3823  VAStartInstrumentationList.push_back(&I);
3824  unpoisonVAListTagForInst(I);
3825  }
3826 
3827  void visitVACopyInst(VACopyInst &I) override {
3828  if (F.getCallingConv() == CallingConv::Win64) return;
3829  unpoisonVAListTagForInst(I);
3830  }
3831 
3832  void finalizeInstrumentation() override {
3833  assert(!VAArgOverflowSize && !VAArgTLSCopy &&
3834  "finalizeInstrumentation called twice");
3835  if (!VAStartInstrumentationList.empty()) {
3836  // If there is a va_start in this function, make a backup copy of
3837  // va_arg_tls somewhere in the function entry block.
3838  IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
3839  VAArgOverflowSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
3840  Value *CopySize =
3841  IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset),
3842  VAArgOverflowSize);
3843  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3844  IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
3845  if (MS.TrackOrigins) {
3846  VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3847  IRB.CreateMemCpy(VAArgTLSOriginCopy, 8, MS.VAArgOriginTLS, 8, CopySize);
3848  }
3849  }
3850 
3851  // Instrument va_start.
3852  // Copy va_list shadow from the backup copy of the TLS contents.
3853  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
3854  CallInst *OrigInst = VAStartInstrumentationList[i];
3855  IRBuilder<> IRB(OrigInst->getNextNode());
3856  Value *VAListTag = OrigInst->getArgOperand(0);
3857 
3858  Value *RegSaveAreaPtrPtr = IRB.CreateIntToPtr(
3859  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3860  ConstantInt::get(MS.IntptrTy, 16)),
3862  Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr);
3863  Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
3864  unsigned Alignment = 16;
3865  std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
3866  MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
3867  Alignment, /*isStore*/ true);
3868  IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
3869  AMD64FpEndOffset);
3870  if (MS.TrackOrigins)
3871  IRB.CreateMemCpy(RegSaveAreaOriginPtr, Alignment, VAArgTLSOriginCopy,
3872  Alignment, AMD64FpEndOffset);
3873  Value *OverflowArgAreaPtrPtr = IRB.CreateIntToPtr(
3874  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3875  ConstantInt::get(MS.IntptrTy, 8)),
3877  Value *OverflowArgAreaPtr = IRB.CreateLoad(OverflowArgAreaPtrPtr);
3878  Value *OverflowArgAreaShadowPtr, *OverflowArgAreaOriginPtr;
3879  std::tie(OverflowArgAreaShadowPtr, OverflowArgAreaOriginPtr) =
3880  MSV.getShadowOriginPtr(OverflowArgAreaPtr, IRB, IRB.getInt8Ty(),
3881  Alignment, /*isStore*/ true);
3882  Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy,
3883  AMD64FpEndOffset);
3884  IRB.CreateMemCpy(OverflowArgAreaShadowPtr, Alignment, SrcPtr, Alignment,
3885  VAArgOverflowSize);
3886  if (MS.TrackOrigins) {
3887  SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSOriginCopy,
3888  AMD64FpEndOffset);
3889  IRB.CreateMemCpy(OverflowArgAreaOriginPtr, Alignment, SrcPtr, Alignment,
3890  VAArgOverflowSize);
3891  }
3892  }
3893  }
3894 };
3895 
3896 /// MIPS64-specific implementation of VarArgHelper.
3897 struct VarArgMIPS64Helper : public VarArgHelper {
3898  Function &F;
3899  MemorySanitizer &MS;
3900  MemorySanitizerVisitor &MSV;
3901  Value *VAArgTLSCopy = nullptr;
3902  Value *VAArgSize = nullptr;
3903 
3904  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3905 
3906  VarArgMIPS64Helper(Function &F, MemorySanitizer &MS,
3907  MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
3908 
3909  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3910  unsigned VAArgOffset = 0;
3911  const DataLayout &DL = F.getParent()->getDataLayout();
3912  for (CallSite::arg_iterator ArgIt = CS.arg_begin() +
3913  CS.getFunctionType()->getNumParams(), End = CS.arg_end();
3914  ArgIt != End; ++ArgIt) {
3915  Triple TargetTriple(F.getParent()->getTargetTriple());
3916  Value *A = *ArgIt;
3917  Value *Base;
3918  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
3919  if (TargetTriple.getArch() == Triple::mips64) {
3920  // Adjusting the shadow for argument with size < 8 to match the placement
3921  // of bits in big endian system
3922  if (ArgSize < 8)
3923  VAArgOffset += (8 - ArgSize);
3924  }
3925  Base = getShadowPtrForVAArgument(A->getType(), IRB, VAArgOffset, ArgSize);
3926  VAArgOffset += ArgSize;
3927  VAArgOffset = alignTo(VAArgOffset, 8);
3928  if (!Base)
3929  continue;
3930  IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
3931  }
3932 
3933  Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(), VAArgOffset);
3934  // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
3935  // a new class member i.e. it is the total size of all VarArgs.
3936  IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
3937  }
3938 
3939  /// Compute the shadow address for a given va_arg.
3940  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
3941  unsigned ArgOffset, unsigned ArgSize) {
3942  // Make sure we don't overflow __msan_va_arg_tls.
3943  if (ArgOffset + ArgSize > kParamTLSSize)
3944  return nullptr;
3945  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
3946  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3947  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
3948  "_msarg");
3949  }
3950 
3951  void visitVAStartInst(VAStartInst &I) override {
3952  IRBuilder<> IRB(&I);
3953  VAStartInstrumentationList.push_back(&I);
3954  Value *VAListTag = I.getArgOperand(0);
3955  Value *ShadowPtr, *OriginPtr;
3956  unsigned Alignment = 8;
3957  std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
3958  VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
3959  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3960  /* size */ 8, Alignment, false);
3961  }
3962 
3963  void visitVACopyInst(VACopyInst &I) override {
3964  IRBuilder<> IRB(&I);
3965  VAStartInstrumentationList.push_back(&I);
3966  Value *VAListTag = I.getArgOperand(0);
3967  Value *ShadowPtr, *OriginPtr;
3968  unsigned Alignment = 8;
3969  std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
3970  VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
3971  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3972  /* size */ 8, Alignment, false);
3973  }
3974 
3975  void finalizeInstrumentation() override {
3976  assert(!VAArgSize && !VAArgTLSCopy &&
3977  "finalizeInstrumentation called twice");
3978  IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
3979  VAArgSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
3980  Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
3981  VAArgSize);
3982 
3983  if (!VAStartInstrumentationList.empty()) {
3984  // If there is a va_start in this function, make a backup copy of
3985  // va_arg_tls somewhere in the function entry block.
3986  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3987  IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
3988  }
3989 
3990  // Instrument va_start.
3991  // Copy va_list shadow from the backup copy of the TLS contents.
3992  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
3993  CallInst *OrigInst = VAStartInstrumentationList[i];
3994  IRBuilder<> IRB(OrigInst->getNextNode());
3995  Value *VAListTag = OrigInst->getArgOperand(0);
3996  Value *RegSaveAreaPtrPtr =
3997  IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3999  Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr);
4000  Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4001  unsigned Alignment = 8;
4002  std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4003  MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4004  Alignment, /*isStore*/ true);
4005  IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4006  CopySize);
4007  }
4008  }
4009 };
4010 
4011 /// AArch64-specific implementation of VarArgHelper.
4012 struct VarArgAArch64Helper : public VarArgHelper {
4013  static const unsigned kAArch64GrArgSize = 64;
4014  static const unsigned kAArch64VrArgSize = 128;
4015 
4016  static const unsigned AArch64GrBegOffset = 0;
4017  static const unsigned AArch64GrEndOffset = kAArch64GrArgSize;
4018  // Make VR space aligned to 16 bytes.
4019  static const unsigned AArch64VrBegOffset = AArch64GrEndOffset;
4020  static const unsigned AArch64VrEndOffset = AArch64VrBegOffset
4021  + kAArch64VrArgSize;
4022  static const unsigned AArch64VAEndOffset = AArch64VrEndOffset;
4023 
4024  Function &F;
4025  MemorySanitizer &MS;
4026  MemorySanitizerVisitor &MSV;
4027  Value *VAArgTLSCopy = nullptr;
4028  Value *VAArgOverflowSize = nullptr;
4029 
4030  SmallVector<CallInst*, 16> VAStartInstrumentationList;
4031 
4032  enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
4033 
4034  VarArgAArch64Helper(Function &F, MemorySanitizer &MS,
4035  MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
4036 
4037  ArgKind classifyArgument(Value* arg) {
4038  Type *T = arg->getType();
4039  if (T->isFPOrFPVectorTy())
4040  return AK_FloatingPoint;
4041  if ((T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
4042  || (T->isPointerTy()))
4043  return AK_GeneralPurpose;
4044  return AK_Memory;
4045  }
4046 
4047  // The instrumentation stores the argument shadow in a non ABI-specific
4048  // format because it does not know which argument is named (since Clang,
4049  // like x86_64 case, lowers the va_args in the frontend and this pass only
4050  // sees the low level code that deals with va_list internals).
4051  // The first seven GR registers are saved in the first 56 bytes of the
4052  // va_arg tls arra, followers by the first 8 FP/SIMD registers, and then
4053  // the remaining arguments.
4054  // Using constant offset within the va_arg TLS array allows fast copy
4055  // in the finalize instrumentation.
4056  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
4057  unsigned GrOffset = AArch64GrBegOffset;
4058  unsigned VrOffset = AArch64VrBegOffset;
4059  unsigned OverflowOffset = AArch64VAEndOffset;
4060 
4061  const DataLayout &DL = F.getParent()->getDataLayout();
4062  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
4063  ArgIt != End; ++ArgIt) {
4064  Value *A = *ArgIt;
4065  unsigned ArgNo = CS.getArgumentNo(ArgIt);
4066  bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
4067  ArgKind AK = classifyArgument(A);
4068  if (AK == AK_GeneralPurpose && GrOffset >= AArch64GrEndOffset)
4069  AK = AK_Memory;
4070  if (AK == AK_FloatingPoint && VrOffset >= AArch64VrEndOffset)
4071  AK = AK_Memory;
4072  Value *Base;
4073  switch (AK) {
4074  case AK_GeneralPurpose:
4075  Base = getShadowPtrForVAArgument(A->getType(), IRB, GrOffset, 8);
4076  GrOffset += 8;
4077  break;
4078  case AK_FloatingPoint:
4079  Base = getShadowPtrForVAArgument(A->getType(), IRB, VrOffset, 8);
4080  VrOffset += 16;
4081  break;
4082  case AK_Memory:
4083  // Don't count fixed arguments in the overflow area - va_start will
4084  // skip right over them.
4085  if (IsFixed)
4086  continue;
4087  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4088  Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset,
4089  alignTo(ArgSize, 8));
4090  OverflowOffset += alignTo(ArgSize, 8);
4091  break;
4092  }
4093  // Count Gp/Vr fixed arguments to their respective offsets, but don't
4094  // bother to actually store a shadow.
4095  if (IsFixed)
4096  continue;
4097  if (!Base)
4098  continue;
4099  IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4100  }
4101  Constant *OverflowSize =
4102  ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AArch64VAEndOffset);
4103  IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
4104  }
4105 
4106  /// Compute the shadow address for a given va_arg.
4107  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4108  unsigned ArgOffset, unsigned ArgSize) {
4109  // Make sure we don't overflow __msan_va_arg_tls.
4110  if (ArgOffset + ArgSize > kParamTLSSize)
4111  return nullptr;
4112  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4113  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4114  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4115  "_msarg");
4116  }
4117 
4118  void visitVAStartInst(VAStartInst &I) override {
4119  IRBuilder<> IRB(&I);
4120  VAStartInstrumentationList.push_back(&I);
4121  Value *VAListTag = I.getArgOperand(0);
4122  Value *ShadowPtr, *OriginPtr;
4123  unsigned Alignment = 8;
4124  std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4125  VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4126  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4127  /* size */ 32, Alignment, false);
4128  }
4129 
4130  void visitVACopyInst(VACopyInst &I) override {
4131  IRBuilder<> IRB(&I);
4132  VAStartInstrumentationList.push_back(&I);
4133  Value *VAListTag = I.getArgOperand(0);
4134  Value *ShadowPtr, *OriginPtr;
4135  unsigned Alignment = 8;
4136  std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4137  VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4138  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4139  /* size */ 32, Alignment, false);
4140  }
4141 
4142  // Retrieve a va_list field of 'void*' size.
4143  Value* getVAField64(IRBuilder<> &IRB, Value *VAListTag, int offset) {
4144  Value *SaveAreaPtrPtr =
4145  IRB.CreateIntToPtr(
4146  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4147  ConstantInt::get(MS.IntptrTy, offset)),
4148  Type::getInt64PtrTy(*MS.C));
4149  return IRB.CreateLoad(SaveAreaPtrPtr);
4150  }
4151 
4152  // Retrieve a va_list field of 'int' size.
4153  Value* getVAField32(IRBuilder<> &IRB, Value *VAListTag, int offset) {
4154  Value *SaveAreaPtr =
4155  IRB.CreateIntToPtr(
4156  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4157  ConstantInt::get(MS.IntptrTy, offset)),
4158  Type::getInt32PtrTy(*MS.C));
4159  Value *SaveArea32 = IRB.CreateLoad(SaveAreaPtr);
4160  return IRB.CreateSExt(SaveArea32, MS.IntptrTy);
4161  }
4162 
4163  void finalizeInstrumentation() override {
4164  assert(!VAArgOverflowSize && !VAArgTLSCopy &&
4165  "finalizeInstrumentation called twice");
4166  if (!VAStartInstrumentationList.empty()) {
4167  // If there is a va_start in this function, make a backup copy of
4168  // va_arg_tls somewhere in the function entry block.
4169  IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
4170  VAArgOverflowSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
4171  Value *CopySize =
4172  IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AArch64VAEndOffset),
4173  VAArgOverflowSize);
4174  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4175  IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
4176  }
4177 
4178  Value *GrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64GrArgSize);
4179  Value *VrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64VrArgSize);
4180 
4181  // Instrument va_start, copy va_list shadow from the backup copy of
4182  // the TLS contents.
4183  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4184  CallInst *OrigInst = VAStartInstrumentationList[i];
4185  IRBuilder<> IRB(OrigInst->getNextNode());
4186 
4187  Value *VAListTag = OrigInst->getArgOperand(0);
4188 
4189  // The variadic ABI for AArch64 creates two areas to save the incoming
4190  // argument registers (one for 64-bit general register xn-x7 and another
4191  // for 128-bit FP/SIMD vn-v7).
4192  // We need then to propagate the shadow arguments on both regions
4193  // 'va::__gr_top + va::__gr_offs' and 'va::__vr_top + va::__vr_offs'.
4194  // The remaning arguments are saved on shadow for 'va::stack'.
4195  // One caveat is it requires only to propagate the non-named arguments,
4196  // however on the call site instrumentation 'all' the arguments are
4197  // saved. So to copy the shadow values from the va_arg TLS array
4198  // we need to adjust the offset for both GR and VR fields based on
4199  // the __{gr,vr}_offs value (since they are stores based on incoming
4200  // named arguments).
4201 
4202  // Read the stack pointer from the va_list.
4203  Value *StackSaveAreaPtr = getVAField64(IRB, VAListTag, 0);
4204 
4205  // Read both the __gr_top and __gr_off and add them up.
4206  Value *GrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 8);
4207  Value *GrOffSaveArea = getVAField32(IRB, VAListTag, 24);
4208 
4209  Value *GrRegSaveAreaPtr = IRB.CreateAdd(GrTopSaveAreaPtr, GrOffSaveArea);
4210 
4211  // Read both the __vr_top and __vr_off and add them up.
4212  Value *VrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 16);
4213  Value *VrOffSaveArea = getVAField32(IRB, VAListTag, 28);
4214 
4215  Value *VrRegSaveAreaPtr = IRB.CreateAdd(VrTopSaveAreaPtr, VrOffSaveArea);
4216 
4217  // It does not know how many named arguments is being used and, on the
4218  // callsite all the arguments were saved. Since __gr_off is defined as
4219  // '0 - ((8 - named_gr) * 8)', the idea is to just propagate the variadic
4220  // argument by ignoring the bytes of shadow from named arguments.
4221  Value *GrRegSaveAreaShadowPtrOff =
4222  IRB.CreateAdd(GrArgSize, GrOffSaveArea);
4223 
4224  Value *GrRegSaveAreaShadowPtr =
4225  MSV.getShadowOriginPtr(GrRegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4226  /*Alignment*/ 8, /*isStore*/ true)
4227  .first;
4228 
4229  Value *GrSrcPtr = IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4230  GrRegSaveAreaShadowPtrOff);
4231  Value *GrCopySize = IRB.CreateSub(GrArgSize, GrRegSaveAreaShadowPtrOff);
4232 
4233  IRB.CreateMemCpy(GrRegSaveAreaShadowPtr, 8, GrSrcPtr, 8, GrCopySize);
4234 
4235  // Again, but for FP/SIMD values.
4236  Value *VrRegSaveAreaShadowPtrOff =
4237  IRB.CreateAdd(VrArgSize, VrOffSaveArea);
4238 
4239  Value *VrRegSaveAreaShadowPtr =
4240  MSV.getShadowOriginPtr(VrRegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4241  /*Alignment*/ 8, /*isStore*/ true)
4242  .first;
4243 
4244  Value *VrSrcPtr = IRB.CreateInBoundsGEP(
4245  IRB.getInt8Ty(),
4246  IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4247  IRB.getInt32(AArch64VrBegOffset)),
4248  VrRegSaveAreaShadowPtrOff);
4249  Value *VrCopySize = IRB.CreateSub(VrArgSize, VrRegSaveAreaShadowPtrOff);
4250 
4251  IRB.CreateMemCpy(VrRegSaveAreaShadowPtr, 8, VrSrcPtr, 8, VrCopySize);
4252 
4253  // And finally for remaining arguments.
4254  Value *StackSaveAreaShadowPtr =
4255  MSV.getShadowOriginPtr(StackSaveAreaPtr, IRB, IRB.getInt8Ty(),
4256  /*Alignment*/ 16, /*isStore*/ true)
4257  .first;
4258 
4259  Value *StackSrcPtr =
4260  IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4261  IRB.getInt32(AArch64VAEndOffset));
4262 
4263  IRB.CreateMemCpy(StackSaveAreaShadowPtr, 16, StackSrcPtr, 16,
4264  VAArgOverflowSize);
4265  }
4266  }
4267 };
4268 
4269 /// PowerPC64-specific implementation of VarArgHelper.
4270 struct VarArgPowerPC64Helper : public VarArgHelper {
4271  Function &F;
4272  MemorySanitizer &MS;
4273  MemorySanitizerVisitor &MSV;
4274  Value *VAArgTLSCopy = nullptr;
4275  Value *VAArgSize = nullptr;
4276 
4277  SmallVector<CallInst*, 16> VAStartInstrumentationList;
4278 
4279  VarArgPowerPC64Helper(Function &F, MemorySanitizer &MS,
4280  MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
4281 
4282  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
4283  // For PowerPC, we need to deal with alignment of stack arguments -
4284  // they are mostly aligned to 8 bytes, but vectors and i128 arrays
4285  // are aligned to 16 bytes, byvals can be aligned to 8 or 16 bytes,
4286  // and QPX vectors are aligned to 32 bytes. For that reason, we
4287  // compute current offset from stack pointer (which is always properly
4288  // aligned), and offset for the first vararg, then subtract them.
4289  unsigned VAArgBase;
4290  Triple TargetTriple(F.getParent()->getTargetTriple());
4291  // Parameter save area starts at 48 bytes from frame pointer for ABIv1,
4292  // and 32 bytes for ABIv2. This is usually determined by target
4293  // endianness, but in theory could be overriden by function attribute.
4294  // For simplicity, we ignore it here (it'd only matter for QPX vectors).
4295  if (TargetTriple.getArch() == Triple::ppc64)
4296  VAArgBase = 48;
4297  else
4298  VAArgBase = 32;
4299  unsigned VAArgOffset = VAArgBase;
4300  const DataLayout &DL = F.getParent()->getDataLayout();
4301  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
4302  ArgIt != End; ++ArgIt) {
4303  Value *A = *ArgIt;
4304  unsigned ArgNo = CS.getArgumentNo(ArgIt);
4305  bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
4306  bool IsByVal = CS.paramHasAttr(ArgNo, Attribute::ByVal);
4307  if (IsByVal) {
4308  assert(A->getType()->isPointerTy());
4309  Type *RealTy = A->getType()->getPointerElementType();
4310  uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
4311  uint64_t ArgAlign = CS.getParamAlignment(ArgNo);
4312  if (ArgAlign < 8)
4313  ArgAlign = 8;
4314  VAArgOffset = alignTo(VAArgOffset, ArgAlign);
4315  if (!IsFixed) {
4316  Value *Base = getShadowPtrForVAArgument(
4317  RealTy, IRB, VAArgOffset - VAArgBase, ArgSize);
4318  if (Base) {
4319  Value *AShadowPtr, *AOriginPtr;
4320  std::tie(AShadowPtr, AOriginPtr) =
4321  MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(),
4322  kShadowTLSAlignment, /*isStore*/ false);
4323 
4324  IRB.CreateMemCpy(Base, kShadowTLSAlignment, AShadowPtr,
4325  kShadowTLSAlignment, ArgSize);
4326  }
4327  }
4328  VAArgOffset += alignTo(ArgSize, 8);
4329  } else {
4330  Value *Base;
4331  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4332  uint64_t ArgAlign = 8;
4333  if (A->getType()->isArrayTy()) {
4334  // Arrays are aligned to element size, except for long double
4335  // arrays, which are aligned to 8 bytes.
4336  Type *ElementTy = A->getType()->getArrayElementType();
4337  if (!ElementTy->isPPC_FP128Ty())
4338  ArgAlign = DL.getTypeAllocSize(ElementTy);
4339  } else if (A->getType()->isVectorTy()) {
4340  // Vectors are naturally aligned.
4341  ArgAlign = DL.getTypeAllocSize(A->getType());
4342  }
4343  if (ArgAlign < 8)
4344  ArgAlign = 8;
4345  VAArgOffset = alignTo(VAArgOffset, ArgAlign);
4346  if (DL.isBigEndian()) {
4347  // Adjusting the shadow for argument with size < 8 to match the placement
4348  // of bits in big endian system
4349  if (ArgSize < 8)
4350  VAArgOffset += (8 - ArgSize);
4351  }
4352  if (!IsFixed) {
4353  Base = getShadowPtrForVAArgument(A->getType(), IRB,
4354  VAArgOffset - VAArgBase, ArgSize);
4355  if (Base)
4356  IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4357  }
4358  VAArgOffset += ArgSize;
4359  VAArgOffset = alignTo(VAArgOffset, 8);
4360  }
4361  if (IsFixed)
4362  VAArgBase = VAArgOffset;
4363  }
4364 
4365  Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(),
4366  VAArgOffset - VAArgBase);
4367  // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
4368  // a new class member i.e. it is the total size of all VarArgs.
4369  IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
4370  }
4371 
4372  /// Compute the shadow address for a given va_arg.
4373  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4374  unsigned ArgOffset, unsigned ArgSize) {
4375  // Make sure we don't overflow __msan_va_arg_tls.
4376  if (ArgOffset + ArgSize > kParamTLSSize)
4377  return nullptr;
4378  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4379  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4380  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4381  "_msarg");
4382  }
4383 
4384  void visitVAStartInst(VAStartInst &I) override {
4385  IRBuilder<> IRB(&I);
4386  VAStartInstrumentationList.push_back(&I);
4387  Value *VAListTag = I.getArgOperand(0);
4388  Value *ShadowPtr, *OriginPtr;
4389  unsigned Alignment = 8;
4390  std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4391  VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4392  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4393  /* size */ 8, Alignment, false);
4394  }
4395 
4396  void visitVACopyInst(VACopyInst &I) override {
4397  IRBuilder<> IRB(&I);
4398  Value *VAListTag = I.getArgOperand(0);
4399  Value *ShadowPtr, *OriginPtr;
4400  unsigned Alignment = 8;
4401  std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4402  VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4403  // Unpoison the whole __va_list_tag.
4404  // FIXME: magic ABI constants.
4405  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4406  /* size */ 8, Alignment, false);
4407  }
4408 
4409  void finalizeInstrumentation() override {
4410  assert(!VAArgSize && !VAArgTLSCopy &&
4411  "finalizeInstrumentation called twice");
4412  IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
4413  VAArgSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
4414  Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
4415  VAArgSize);
4416 
4417  if (!VAStartInstrumentationList.empty()) {
4418  // If there is a va_start in this function, make a backup copy of
4419  // va_arg_tls somewhere in the function entry block.
4420  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4421  IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
4422  }
4423 
4424  // Instrument va_start.
4425  // Copy va_list shadow from the backup copy of the TLS contents.
4426  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4427  CallInst *OrigInst = VAStartInstrumentationList[i];
4428  IRBuilder<> IRB(OrigInst->getNextNode());
4429  Value *VAListTag = OrigInst->getArgOperand(0);
4430  Value *RegSaveAreaPtrPtr =
4431  IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4433  Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr);
4434  Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4435  unsigned Alignment = 8;
4436  std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4437  MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4438  Alignment, /*isStore*/ true);
4439  IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4440  CopySize);
4441  }
4442  }
4443 };
4444 
4445 /// A no-op implementation of VarArgHelper.
4446 struct VarArgNoOpHelper : public VarArgHelper {
4447  VarArgNoOpHelper(Function &F, MemorySanitizer &MS,
4448  MemorySanitizerVisitor &MSV) {}
4449 
4450  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {}
4451 
4452  void visitVAStartInst(VAStartInst &I) override {}
4453 
4454  void visitVACopyInst(VACopyInst &I) override {}
4455 
4456  void finalizeInstrumentation() override {}
4457 };
4458 
4459 } // end anonymous namespace
4460 
4461 static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
4462  MemorySanitizerVisitor &Visitor) {
4463  // VarArg handling is only implemented on AMD64. False positives are possible
4464  // on other platforms.
4465  Triple TargetTriple(Func.getParent()->getTargetTriple());
4466  if (TargetTriple.getArch() == Triple::x86_64)
4467  return new VarArgAMD64Helper(Func, Msan, Visitor);
4468  else if (TargetTriple.isMIPS64())
4469  return new VarArgMIPS64Helper(Func, Msan, Visitor);
4470  else if (TargetTriple.getArch() == Triple::aarch64)
4471  return new VarArgAArch64Helper(Func, Msan, Visitor);
4472  else if (TargetTriple.getArch() == Triple::ppc64 ||
4473  TargetTriple.getArch() == Triple::ppc64le)
4474  return new VarArgPowerPC64Helper(Func, Msan, Visitor);
4475  else
4476  return new VarArgNoOpHelper(Func, Msan, Visitor);
4477 }
4478 
4479 bool MemorySanitizer::sanitizeFunction(Function &F, TargetLibraryInfo &TLI) {
4480  if (!CompileKernel && (&F == MsanCtorFunction))
4481  return false;
4482  MemorySanitizerVisitor Visitor(F, *this, TLI);
4483 
4484  // Clear out readonly/readnone attributes.
4485  AttrBuilder B;
4486  B.addAttribute(Attribute::ReadOnly)
4487  .addAttribute(Attribute::ReadNone);
4489 
4490  return Visitor.runOnFunction();
4491 }
Value * CreateInBoundsGEP(Value *Ptr, ArrayRef< Value *> IdxList, const Twine &Name="")
Definition: IRBuilder.h:1476
Type * getVectorElementType() const
Definition: Type.h:370
uint64_t CallInst * C
Return a value (possibly void), from a function.
User::op_iterator arg_iterator
The type of iterator to use when looping over actual arguments at this call site. ...
Definition: CallSite.h:212
SymbolTableList< Instruction >::iterator eraseFromParent()
This method unlinks &#39;this&#39; from the containing basic block and deletes it.
Definition: Instruction.cpp:67
unsigned Log2_32_Ceil(uint32_t Value)
Return the ceil log base 2 of the specified value, 32 if the value is zero.
Definition: MathExtras.h:551
A parsed version of the target data layout string in and methods for querying it. ...
Definition: DataLayout.h:110
constexpr char Align[]
Key for Kernel::Arg::Metadata::mAlign.
const std::string & getTargetTriple() const
Get the target triple which is a string describing the target host.
Definition: Module.h:239
static const MemoryMapParams Linux_PowerPC64_MemoryMapParams
Value * CreateICmp(CmpInst::Predicate P, Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1948
Value * CreateConstGEP1_32(Value *Ptr, unsigned Idx0, const Twine &Name="")
Definition: IRBuilder.h:1515
static Constant * getString(LLVMContext &Context, StringRef Initializer, bool AddNull=true)
This method constructs a CDS and initializes it with a text string.
Definition: Constants.cpp:2566
bool isAllOnesValue() const
Return true if this is the value that would be returned by getAllOnesValue.
Definition: Constants.cpp:99
raw_ostream & errs()
This returns a reference to a raw_ostream for standard error.
void addIncoming(Value *V, BasicBlock *BB)
Add an incoming value to the end of the PHI list.
This instruction extracts a struct member or array element value from an aggregate value...
Value * CreateBinOp(Instruction::BinaryOps Opc, Value *LHS, Value *RHS, const Twine &Name="", MDNode *FPMathTag=nullptr)
Definition: IRBuilder.h:1297
GCNRegPressure max(const GCNRegPressure &P1, const GCNRegPressure &P2)
This class represents an incoming formal argument to a Function.
Definition: Argument.h:29
static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams
AllocaInst * CreateAlloca(Type *Ty, unsigned AddrSpace, Value *ArraySize=nullptr, const Twine &Name="")
Definition: IRBuilder.h:1343
bool doesNotAccessMemory(unsigned OpNo) const
Definition: InstrTypes.h:1428
Base class for instruction visitors.
Definition: InstVisitor.h:80
Value * getAggregateOperand()
Value * CreateICmpNE(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1842
Atomic ordering constants.
NodeTy * getNextNode()
Get the next node, or nullptr for the list tail.
Definition: ilist_node.h:288
PassT::Result & getResult(IRUnitT &IR, ExtraArgTs... ExtraArgs)
Get the result of an analysis pass for a given IR unit.
Definition: PassManager.h:769
LLVM_ATTRIBUTE_NORETURN void report_fatal_error(Error Err, bool gen_crash_diag=true)
Report a serious error, calling any installed error handler.
Definition: Error.cpp:139
This class represents lattice values for constants.
Definition: AllocatorList.h:23
BinaryOps getOpcode() const
Definition: InstrTypes.h:315
unsigned getParamAlignment(unsigned ArgNo) const
Extract the alignment for a call or parameter (0=unknown).
Definition: CallSite.h:405
bool isAtomic() const
Return true if this instruction has an AtomicOrdering of unordered or higher.
Constant * getOrInsertFunction(StringRef Name, FunctionType *T, AttributeList AttributeList)
Look up the specified function in the module symbol table.
Definition: Module.cpp:143
LoadInst * CreateLoad(Type *Ty, Value *Ptr, const char *Name)
Provided to resolve &#39;CreateLoad(Ty, Ptr, "...")&#39; correctly, instead of converting the string to &#39;bool...
Definition: IRBuilder.h:1356
Value * CreateXor(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1199
A Module instance is used to store all the information related to an LLVM module. ...
Definition: Module.h:64
bool isSized(SmallPtrSetImpl< Type *> *Visited=nullptr) const
Return true if it makes sense to take the size of this type.
Definition: Type.h:264
LoadInst * CreateAlignedLoad(Type *Ty, Value *Ptr, unsigned Align, const char *Name)
Provided to resolve &#39;CreateAlignedLoad(Ptr, Align, "...")&#39; correctly, instead of converting the strin...
Definition: IRBuilder.h:1392
void setOrdering(AtomicOrdering Ordering)
Sets the ordering constraint of this rmw instruction.
Definition: Instructions.h:778
static const MemoryMapParams Linux_I386_MemoryMapParams
Same, but only replaced by something equivalent.
Definition: GlobalValue.h:53
an instruction that atomically checks whether a specified value is in a memory location, and, if it is, stores a new value there.
Definition: Instructions.h:528
static const MemoryMapParams NetBSD_X86_64_MemoryMapParams
static cl::opt< bool > ClPoisonStackWithCall("msan-poison-stack-with-call", cl::desc("poison uninitialized stack variables with a call"), cl::Hidden, cl::init(false))
This class represents zero extension of integer types.
static const unsigned kRetvalTLSSize
unsigned getNumElements() const
Random access to the elements.
Definition: DerivedTypes.h:312
bool removeUnreachableBlocks(Function &F, LazyValueInfo *LVI=nullptr, DomTreeUpdater *DTU=nullptr, MemorySSAUpdater *MSSAU=nullptr)
Remove all blocks that can not be reached from the function&#39;s entry.
Definition: Local.cpp:2200
Value * CreateICmpSLT(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1870
This class represents a function call, abstracting a target machine&#39;s calling convention.
static cl::opt< bool > ClHandleAsmConservative("msan-handle-asm-conservative", cl::desc("conservative handling of inline assembly"), cl::Hidden, cl::init(true))
static PointerType * getInt32PtrTy(LLVMContext &C, unsigned AS=0)
Definition: Type.cpp:227
void setOrdering(AtomicOrdering Ordering)
Sets the ordering constraint of this load instruction.
Definition: Instructions.h:253
static PointerType * get(Type *ElementType, unsigned AddressSpace)
This constructs a pointer to an object of the specified type in a numbered address space...
Definition: Type.cpp:629
const Value * getTrueValue() const
The C convention as implemented on Windows/x86-64 and AArch64.
Definition: CallingConv.h:153
AtomicOrdering getOrdering() const
Returns the ordering constraint of this load instruction.
Definition: Instructions.h:247
Like Internal, but omit from symbol table.
Definition: GlobalValue.h:56
static VarArgHelper * CreateVarArgHelper(Function &Func, MemorySanitizer &Msan, MemorySanitizerVisitor &Visitor)
This instruction constructs a fixed permutation of two input vectors.
static cl::opt< bool > ClWithComdat("msan-with-comdat", cl::desc("Place MSan constructors in comdat sections"), cl::Hidden, cl::init(false))
Externally visible function.
Definition: GlobalValue.h:48
bool hasFnAttribute(Attribute::AttrKind Kind) const
Return true if the function has the attribute.
Definition: Function.h:320
A raw_ostream that writes to an SmallVector or SmallString.
Definition: raw_ostream.h:509
This class wraps the llvm.memset intrinsic.
static bool isEquality(Predicate P)
Return true if this predicate is either EQ or NE.
Value * CreateSExt(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1662
Metadata node.
Definition: Metadata.h:863
static unsigned TypeSizeToSizeIndex(unsigned TypeSize)
F(f)
This class represents a sign extension of integer types.
uint64_t alignTo(uint64_t Value, uint64_t Align, uint64_t Skew=0)
Returns the next integer (mod 2**64) that is greater than or equal to Value and is a multiple of Alig...
Definition: MathExtras.h:684
CallInst * CreateMemSet(Value *Ptr, Value *Val, uint64_t Size, unsigned Align, bool isVolatile=false, MDNode *TBAATag=nullptr, MDNode *ScopeTag=nullptr, MDNode *NoAliasTag=nullptr)
Create and insert a memset to the specified pointer and the specified value.
Definition: IRBuilder.h:403
An instruction for reading from memory.
Definition: Instructions.h:167
AttrBuilder & addAttribute(Attribute::AttrKind Val)
Add an attribute to the builder.
an instruction that atomically reads a memory location, combines it with another value, and then stores the result back.
Definition: Instructions.h:691
Hexagon Common GEP
bool isVectorTy() const
True if this is an instance of VectorType.
Definition: Type.h:229
#define op(i)
bool isMustTailCall() const
static Type * getX86_MMXTy(LLVMContext &C)
Definition: Type.cpp:170
static cl::opt< unsigned long long > ClXorMask("msan-xor-mask", cl::desc("Define custom MSan XorMask"), cl::Hidden, cl::init(0))
Use * op_iterator
Definition: User.h:224
bool isPPC_FP128Ty() const
Return true if this is powerpc long double.
Definition: Type.h:158
static const MemoryMapParams Linux_AArch64_MemoryMapParams
static cl::opt< bool > ClHandleICmp("msan-handle-icmp", cl::desc("propagate shadow through ICmpEQ and ICmpNE"), cl::Hidden, cl::init(true))
op_iterator op_begin()
Definition: User.h:229
static PointerType * getInt64PtrTy(LLVMContext &C, unsigned AS=0)
Definition: Type.cpp:231
static Constant * get(ArrayType *T, ArrayRef< Constant *> V)
Definition: Constants.cpp:982
unsigned getBitWidth() const
Return the number of bits in the APInt.
Definition: APInt.h:1508
static Constant * getNullValue(Type *Ty)
Constructor to create a &#39;0&#39; constant of arbitrary type.
Definition: Constants.cpp:264
Value * CreateNot(Value *V, const Twine &Name="")
Definition: IRBuilder.h:1333
StoreInst * CreateAlignedStore(Value *Val, Value *Ptr, unsigned Align, bool isVolatile=false)
Definition: IRBuilder.h:1429
IntegerType * getInt32Ty()
Fetch the type representing a 32-bit integer.
Definition: IRBuilder.h:346
Value * getArgOperand(unsigned i) const
Definition: InstrTypes.h:1134
unsigned countTrailingZeros() const
Count the number of trailing zero bits.
Definition: APInt.h:1631
static cl::opt< int > ClPoisonStackPattern("msan-poison-stack-pattern", cl::desc("poison uninitialized stack variables with the given pattern"), cl::Hidden, cl::init(0xff))
ArrayRef< unsigned > getIndices() const
AnalysisUsage & addRequired()
#define INITIALIZE_PASS_DEPENDENCY(depName)
Definition: PassSupport.h:50
amdgpu Simplify well known AMD library false Value Value const Twine & Name
bool isSigned() const
Definition: InstrTypes.h:815
static cl::opt< bool > ClDumpStrictInstructions("msan-dump-strict-instructions", cl::desc("print out instructions with default strict semantics"), cl::Hidden, cl::init(false))
This class represents the LLVM &#39;select&#39; instruction.
Type * getPointerElementType() const
Definition: Type.h:375
const DataLayout & getDataLayout() const
Get the data layout for the module&#39;s target platform.
Definition: Module.cpp:370
FunctionPass * createMemorySanitizerLegacyPassPass(int TrackOrigins=0, bool Recover=false, bool EnableKmsan=false)
unsigned getAlignment() const
Return the alignment of the memory that is being allocated by the instruction.
Definition: Instructions.h:112
IntegerType * getInt64Ty()
Fetch the type representing a 64-bit integer.
Definition: IRBuilder.h:351
This is the base class for all instructions that perform data casts.
Definition: InstrTypes.h:352
ArrayRef< T > makeArrayRef(const T &OneElt)
Construct an ArrayRef from a single element.
Definition: ArrayRef.h:450
&#39;undef&#39; values are things that do not have specified contents.
Definition: Constants.h:1285
This class wraps the llvm.memmove intrinsic.
Class to represent struct types.
Definition: DerivedTypes.h:200
LLVMContext & getContext() const
Get the global data context.
Definition: Module.h:243
static cl::opt< bool > ClCheckAccessAddress("msan-check-access-address", cl::desc("report accesses through a pointer which has poisoned shadow"), cl::Hidden, cl::init(true))
PointerType * getPointerTo(unsigned AddrSpace=0) const
Return a pointer to the current type.
Definition: Type.cpp:651
IterTy arg_end() const
Definition: CallSite.h:574
bool isUnsigned() const
Definition: InstrTypes.h:821
static cl::opt< unsigned long long > ClAndMask("msan-and-mask", cl::desc("Define custom MSan AndMask"), cl::Hidden, cl::init(0))
bool isIntegerTy() const
True if this is an instance of IntegerType.
Definition: Type.h:196
This provides a uniform API for creating instructions and inserting them into a basic block: either a...
Definition: IRBuilder.h:742
IntegerType * getIntPtrTy(const DataLayout &DL, unsigned AddrSpace=0)
Fetch the type representing a pointer to an integer value.
Definition: IRBuilder.h:389
This file contains the simple types necessary to represent the attributes associated with functions a...
static cl::opt< unsigned long long > ClOriginBase("msan-origin-base", cl::desc("Define custom MSan OriginBase"), cl::Hidden, cl::init(0))
InstrTy * getInstruction() const
Definition: CallSite.h:91
Value * CreateAdd(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:1013
void setName(const Twine &Name)
Change the name of the value.
Definition: Value.cpp:284
static StructType * get(LLVMContext &Context, ArrayRef< Type *> Elements, bool isPacked=false)
This static method is the primary way to create a literal StructType.
Definition: Type.cpp:341
This file implements a class to represent arbitrary precision integral constant values and operations...
Type * getVoidTy()
Fetch the type representing void.
Definition: IRBuilder.h:379
This class represents a cast from a pointer to an integer.
AtomicOrdering
Atomic ordering for LLVM&#39;s memory model.
StoreInst * CreateStore(Value *Val, Value *Ptr, bool isVolatile=false)
Definition: IRBuilder.h:1385
Value * CreateIntToPtr(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1726
bool isNullValue() const
Return true if this is the value that would be returned by getNullValue.
Definition: Constants.cpp:84
Class to represent function types.
Definition: DerivedTypes.h:102
Value * CreateBitCast(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1731
Type * getType() const
All values are typed, get the type of this value.
Definition: Value.h:244
This represents the llvm.va_start intrinsic.
static const char *const kMsanInitName
std::string itostr(int64_t X)
Definition: StringExtras.h:238
AtomicOrdering getSuccessOrdering() const
Returns the success ordering constraint of this cmpxchg instruction.
Definition: Instructions.h:582
#define T
static cl::opt< int > ClTrackOrigins("msan-track-origins", cl::desc("Track origins (allocation sites) of poisoned memory"), cl::Hidden, cl::init(0))
Track origins of uninitialized values.
Class to represent array types.
Definition: DerivedTypes.h:368
This instruction compares its operands according to the predicate given to the constructor.
static bool isStore(int Opcode)
void setComdat(Comdat *C)
Definition: GlobalObject.h:102
bool isVarArg() const
Definition: DerivedTypes.h:122
This class represents a no-op cast from one type to another.
bool paramHasAttr(unsigned ArgNo, Attribute::AttrKind Kind) const
Return true if the call or the callee has the given attribute.
Definition: CallSite.h:376
MDNode * getMetadata(unsigned KindID) const
Get the metadata of given kind attached to this Instruction.
Definition: Instruction.h:220
Value * getInsertedValueOperand()
SmallString - A SmallString is just a SmallVector with methods and accessors that make it work better...
Definition: SmallString.h:25
Value * CreateSub(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:1030
AttributeList getAttributes() const
Return the attribute list for this Function.
Definition: Function.h:223
An instruction for storing to memory.
Definition: Instructions.h:320
bool isIntOrIntVectorTy() const
Return true if this is an integer type or a vector of integer types.
Definition: Type.h:202
static const unsigned kParamTLSSize
Value * CreateZExt(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1658
ConstraintPrefix Type
Type - The basic type of the constraint: input/output/clobber.
Definition: InlineAsm.h:120
static cl::opt< bool > ClPoisonStack("msan-poison-stack", cl::desc("poison uninitialized stack variables"), cl::Hidden, cl::init(true))
Function * getDeclaration(Module *M, ID id, ArrayRef< Type *> Tys=None)
Create or insert an LLVM Function declaration for an intrinsic, and return it.
Definition: Function.cpp:1019
This class represents a truncation of integer types.
void SetInsertPoint(BasicBlock *TheBB)
This specifies that created instructions should be appended to the end of the specified block...
Definition: IRBuilder.h:126
Value * getOperand(unsigned i) const
Definition: User.h:169
Analysis containing CSE Info
Definition: CSEInfo.cpp:20
static PreservedAnalyses none()
Convenience factory function for the empty preserved set.
Definition: PassManager.h:156
static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams
bool isRelational() const
Return true if the predicate is relational (not EQ or NE).
bool isCall() const
Return true if a CallInst is enclosed.
Definition: CallSite.h:86
Value * CreateOr(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1181
static const unsigned kMinOriginAlignment
Constant * getAggregateElement(unsigned Elt) const
For aggregates (struct/array/vector) return the constant that corresponds to the specified element if...
Definition: Constants.cpp:334
Type * getScalarType() const
If this is a vector type, return the element type, otherwise return &#39;this&#39;.
Definition: Type.h:303
bool isZeroValue() const
Return true if the value is negative zero or null value.
Definition: Constants.cpp:64
bool isVoidTy() const
Return true if this is &#39;void&#39;.
Definition: Type.h:140
const BasicBlock & getEntryBlock() const
Definition: Function.h:639
an instruction for type-safe pointer arithmetic to access elements of arrays and structs ...
Definition: Instructions.h:853
static cl::opt< unsigned long long > ClShadowBase("msan-shadow-base", cl::desc("Define custom MSan ShadowBase"), cl::Hidden, cl::init(0))
static bool runOnFunction(Function &F, bool PostInlining)
initializer< Ty > init(const Ty &Val)
Definition: CommandLine.h:422
This instruction inserts a single (scalar) element into a VectorType value.
The landingpad instruction holds all of the information necessary to generate correct exception handl...
const Instruction * getFirstNonPHI() const
Returns a pointer to the first instruction in this block that is not a PHINode instruction.
Definition: BasicBlock.cpp:189
static GCRegistry::Add< OcamlGC > B("ocaml", "ocaml 3.10-compatible GC")
A set of analyses that are preserved following a run of a transformation pass.
Definition: PassManager.h:153
const_iterator getFirstInsertionPt() const
Returns an iterator to the first instruction in this block that is suitable for inserting a non-PHI i...
Definition: BasicBlock.cpp:216
const BasicBlock * getSinglePredecessor() const
Return the predecessor of this block if it has a single predecessor block.
Definition: BasicBlock.cpp:233
Value * getCalledValue() const
Definition: InstrTypes.h:1173
static Constant * getOrInsertGlobal(Module &M, StringRef Name, Type *Ty)
LLVM Basic Block Representation.
Definition: BasicBlock.h:57
The instances of the Type class are immutable: once they are created, they are never changed...
Definition: Type.h:45
This is an important class for using LLVM in a threaded context.
Definition: LLVMContext.h:68
const char * getOpcodeName() const
Definition: Instruction.h:127
This is an important base class in LLVM.
Definition: Constant.h:41
Resume the propagation of an exception.
This file contains the declarations for the subclasses of Constant, which represent the different fla...
Value * CreateSelect(Value *C, Value *True, Value *False, const Twine &Name="", Instruction *MDFrom=nullptr)
Definition: IRBuilder.h:2020
bool isPointerTy() const
True if this is an instance of PointerType.
Definition: Type.h:223
unsigned getNumParams() const
Return the number of fixed parameters this function type requires.
Definition: DerivedTypes.h:138
Represent the analysis usage information of a pass.
op_iterator op_end()
Definition: User.h:231
Value * CreateNeg(Value *V, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:1307
static const PlatformMemoryMapParams Linux_X86_MemoryMapParams
This instruction compares its operands according to the predicate given to the constructor.
Predicate
This enumeration lists the possible predicates for CmpInst subclasses.
Definition: InstrTypes.h:645
FunctionPass class - This class is used to implement most global optimizations.
Definition: Pass.h:284
static const unsigned kShadowTLSAlignment
static FunctionType * get(Type *Result, ArrayRef< Type *> Params, bool isVarArg)
This static method is the primary way of constructing a FunctionType.
Definition: Type.cpp:296
static Constant * get(StructType *T, ArrayRef< Constant *> V)
Definition: Constants.cpp:1043
Value * getPointerOperand()
Definition: Instructions.h:284
bool isX86_MMXTy() const
Return true if this is X86 MMX.
Definition: Type.h:181
Value * CreateICmpEQ(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1838
self_iterator getIterator()
Definition: ilist_node.h:81
Class to represent integer types.
Definition: DerivedTypes.h:39
IntegerType * getIntNTy(unsigned N)
Fetch the type representing an N-bit integer.
Definition: IRBuilder.h:359
Value * CreateExtractElement(Value *Vec, Value *Idx, const Twine &Name="")
Definition: IRBuilder.h:2040
This class represents a cast from an integer to a pointer.
const Value * getCondition() const
static Constant * getAllOnesValue(Type *Ty)
Definition: Constants.cpp:318
static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams
static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams
Comdat * getOrInsertComdat(StringRef Name)
Return the Comdat in the module with the specified name.
Definition: Module.cpp:483
INITIALIZE_PASS_BEGIN(MemorySanitizerLegacyPass, "msan", "MemorySanitizer: detects uninitialized reads.", false, false) INITIALIZE_PASS_END(MemorySanitizerLegacyPass
static PreservedAnalyses all()
Construct a special preserved set that preserves all passes.
Definition: PassManager.h:159
const Value * getArraySize() const
Get the number of elements allocated.
Definition: Instructions.h:92
Value * CreateExtractValue(Value *Agg, ArrayRef< unsigned > Idxs, const Twine &Name="")
Definition: IRBuilder.h:2082
size_t size() const
Definition: SmallVector.h:52
static wasm::ValType getType(const TargetRegisterClass *RC)
PointerType * getInt8PtrTy(unsigned AddrSpace=0)
Fetch the type representing a pointer to an 8-bit integer value.
Definition: IRBuilder.h:384
MDNode * createBranchWeights(uint32_t TrueWeight, uint32_t FalseWeight)
Return metadata containing two branch weights.
Definition: MDBuilder.cpp:37
AtomicOrdering getOrdering() const
Returns the ordering constraint of this rmw instruction.
Definition: Instructions.h:773
INITIALIZE_PASS_END(RegBankSelect, DEBUG_TYPE, "Assign register bank of generic virtual registers", false, false) RegBankSelect
#define llvm_unreachable(msg)
Marks that the current location is not supposed to be reachable.
Value * CreateMul(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:1047
Value * CreateTrunc(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1654
Type * getAllocatedType() const
Return the type that is being allocated by the instruction.
Definition: Instructions.h:105
Triple - Helper class for working with autoconf configuration names.
Definition: Triple.h:43
signed greater than
Definition: InstrTypes.h:672
std::vector< ConstraintInfo > ConstraintInfoVector
Definition: InlineAsm.h:115
unsigned first
bool isInvoke() const
Return true if a InvokeInst is enclosed.
Definition: CallSite.h:89
Intrinsic::ID getIntrinsicID() const
Return the intrinsic ID of this intrinsic.
Definition: IntrinsicInst.h:50
PHINode * CreatePHI(Type *Ty, unsigned NumReservedValues, const Twine &Name="")
Definition: IRBuilder.h:1968
Value * CreateGEP(Value *Ptr, ArrayRef< Value *> IdxList, const Twine &Name="")
Definition: IRBuilder.h:1457
bool isPtrOrPtrVectorTy() const
Return true if this is a pointer type or a vector of pointer types.
Definition: Type.h:226
static IntegerType * get(LLVMContext &C, unsigned NumBits)
This static method is the primary way of constructing an IntegerType.
Definition: Type.cpp:239
Type * getSequentialElementType() const
Definition: Type.h:357
Iterator for intrusive lists based on ilist_node.
unsigned getNumOperands() const
Definition: User.h:191
See the file comment.
Definition: ValueMap.h:85
This is the shared class of boolean and integer constants.
Definition: Constants.h:83
auto size(R &&Range, typename std::enable_if< std::is_same< typename std::iterator_traits< decltype(Range.begin())>::iterator_category, std::random_access_iterator_tag >::value, void >::type *=nullptr) -> decltype(std::distance(Range.begin(), Range.end()))
Get the size of a range.
Definition: STLExtras.h:1166
iterator end()
Definition: BasicBlock.h:270
unsigned getScalarSizeInBits() const LLVM_READONLY
If this is a vector type, return the getPrimitiveSizeInBits value for the element type...
Definition: Type.cpp:129
CallingConv::ID getCallingConv() const
getCallingConv()/setCallingConv(CC) - These method get and set the calling convention of this functio...
Definition: Function.h:212
IterTy arg_begin() const
Definition: CallSite.h:570
Value * CreateIntCast(Value *V, Type *DestTy, bool isSigned, const Twine &Name="")
Definition: IRBuilder.h:1800
This is a &#39;vector&#39; (really, a variable-sized array), optimized for the case when the array is small...
Definition: SmallVector.h:846
Module.h This file contains the declarations for the Module class.
Value * CreateInsertElement(Value *Vec, Value *NewElt, Value *Idx, const Twine &Name="")
Definition: IRBuilder.h:2053
Provides information about what library functions are available for the current target.
unsigned getABITypeAlignment(Type *Ty) const
Returns the minimum ABI-required alignment for the specified type.
Definition: DataLayout.cpp:729
bool isAggregateType() const
Return true if the type is an aggregate type.
Definition: Type.h:257
signed less than
Definition: InstrTypes.h:674
CallInst * CreateMaskedStore(Value *Val, Value *Ptr, unsigned Align, Value *Mask)
Create a call to Masked Store intrinsic.
Definition: IRBuilder.cpp:491
CHAIN = SC CHAIN, Imm128 - System call.
ConstantInt * getInt32(uint32_t C)
Get a constant 32-bit value.
Definition: IRBuilder.h:306
static IntegerType * getIntNTy(LLVMContext &C, unsigned N)
Definition: Type.cpp:179
StringRef str()
Return a StringRef for the vector contents.
Definition: raw_ostream.h:534
static GCRegistry::Add< StatepointGC > D("statepoint-example", "an example strategy for statepoint")
CallInst * CreateMemCpy(Value *Dst, unsigned DstAlign, Value *Src, unsigned SrcAlign, uint64_t Size, bool isVolatile=false, MDNode *TBAATag=nullptr, MDNode *TBAAStructTag=nullptr, MDNode *ScopeTag=nullptr, MDNode *NoAliasTag=nullptr)
Create and insert a memcpy between the specified pointers.
Definition: IRBuilder.h:445
This class wraps the llvm.memcpy intrinsic.
static Constant * get(Type *Ty, uint64_t V, bool isSigned=false)
If Ty is a vector type, return a Constant with a splat of the given value.
Definition: Constants.cpp:621
void appendToGlobalCtors(Module &M, Function *F, int Priority, Constant *Data=nullptr)
Append F to the list of global ctors of module M with the given Priority.
Definition: ModuleUtils.cpp:83
CallInst * CreateMaskedLoad(Value *Ptr, unsigned Align, Value *Mask, Value *PassThru=nullptr, const Twine &Name="")
Create a call to Masked Load intrinsic.
Definition: IRBuilder.cpp:470
unsigned getNumIncomingValues() const
Return the number of incoming edges.
static const size_t kNumberOfAccessSizes
static GlobalVariable * createPrivateNonConstGlobalForString(Module &M, StringRef Str)
Create a non-const global initialized with the given string.
static cl::opt< bool > ClKeepGoing("msan-keep-going", cl::desc("keep going after reporting a UMR"), cl::Hidden, cl::init(false))
raw_ostream & dbgs()
dbgs() - This returns a reference to a raw_ostream for debugging messages.
Definition: Debug.cpp:132
unsigned getVectorNumElements() const
Definition: DerivedTypes.h:461
signed less or equal
Definition: InstrTypes.h:675
Class to represent vector types.
Definition: DerivedTypes.h:392
const Module * getModule() const
Return the module owning the function this instruction belongs to or nullptr it the function does not...
Definition: Instruction.cpp:55
Class for arbitrary precision integers.
Definition: APInt.h:69
IntegerType * getInt8Ty()
Fetch the type representing an 8-bit integer.
Definition: IRBuilder.h:336
Value * CreateShl(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:1102
const Value * getFalseValue() const
Value * CreatePointerCast(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1777
Instruction * SplitBlockAndInsertIfThen(Value *Cond, Instruction *SplitBefore, bool Unreachable, MDNode *BranchWeights=nullptr, DominatorTree *DT=nullptr, LoopInfo *LI=nullptr)
Split the containing block at the specified instruction - everything before SplitBefore stays in the ...
uint64_t getTypeSizeInBits(Type *Ty) const
Size examples:
Definition: DataLayout.h:567
static cl::opt< ITMode > IT(cl::desc("IT block support"), cl::Hidden, cl::init(DefaultIT), cl::ZeroOrMore, cl::values(clEnumValN(DefaultIT, "arm-default-it", "Generate IT block based on arch"), clEnumValN(RestrictedIT, "arm-restrict-it", "Disallow deprecated IT based on ARMv8"), clEnumValN(NoRestrictedIT, "arm-no-restrict-it", "Allow IT blocks based on ARMv7")))
uint64_t getTypeAllocSize(Type *Ty) const
Returns the offset in bytes between successive objects of the specified type, including alignment pad...
Definition: DataLayout.h:435
void removeAttributes(unsigned i, const AttrBuilder &Attrs)
removes the attributes from the list of attributes.
Definition: Function.cpp:415
Predicate getPredicate() const
Return the predicate for this instruction.
Definition: InstrTypes.h:720
unsigned getNumArgOperands() const
Definition: InstrTypes.h:1132
static cl::opt< bool > ClEnableKmsan("msan-kernel", cl::desc("Enable KernelMemorySanitizer instrumentation"), cl::Hidden, cl::init(false))
bool isInlineAsm() const
Check if this call is an inline asm statement.
Constant * getOrInsertGlobal(StringRef Name, Type *Ty, function_ref< GlobalVariable *()> CreateGlobalCallback)
Look up the specified global in the module symbol table.
Definition: Module.cpp:205
static cl::opt< bool > ClHandleICmpExact("msan-handle-icmp-exact", cl::desc("exact handling of relational integer ICmp"), cl::Hidden, cl::init(false))
unsigned getAlignment() const
Return the alignment of the access that is being performed.
Definition: Instructions.h:240
LLVM_NODISCARD bool empty() const
Definition: SmallVector.h:55
StringRef getName() const
Return a constant reference to the value&#39;s name.
Definition: Value.cpp:213
Function * getCalledFunction() const
Returns the function called, or null if this is an indirect function invocation.
Definition: InstrTypes.h:1180
bool onlyReadsMemory(unsigned OpNo) const
Definition: InstrTypes.h:1434
#define I(x, y, z)
Definition: MD5.cpp:58
#define N
static const MemoryMapParams Linux_X86_64_MemoryMapParams
static ArrayType * get(Type *ElementType, uint64_t NumElements)
This static method is the primary way to construct an ArrayType.
Definition: Type.cpp:580
LLVM_NODISCARD std::enable_if<!is_simple_type< Y >::value, typename cast_retty< X, const Y >::ret_type >::type dyn_cast(const Y &Val)
Definition: Casting.h:322
This instruction extracts a single (scalar) element from a VectorType value.
uint32_t Size
Definition: Profile.cpp:46
void maybeMarkSanitizerLibraryCallNoBuiltin(CallInst *CI, const TargetLibraryInfo *TLI)
Given a CallInst, check if it calls a string function known to CodeGen, and mark it with NoBuiltin if...
Definition: Local.cpp:2822
CallInst * CreateCall(FunctionType *FTy, Value *Callee, ArrayRef< Value *> Args=None, const Twine &Name="", MDNode *FPMathTag=nullptr)
Definition: IRBuilder.h:1973
static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams
Value * getReturnValue() const
Convenience accessor. Returns null if there is no return value.
static InlineAsm * get(FunctionType *Ty, StringRef AsmString, StringRef Constraints, bool hasSideEffects, bool isAlignStack=false, AsmDialect asmDialect=AD_ATT)
InlineAsm::get - Return the specified uniqued inline asm string.
Definition: InlineAsm.cpp:42
PreservedAnalyses run(Function &F, FunctionAnalysisManager &FAM)
static cl::opt< bool > ClCheckConstantShadow("msan-check-constant-shadow", cl::desc("Insert checks for constant shadow values"), cl::Hidden, cl::init(false))
static const MemoryMapParams Linux_MIPS64_MemoryMapParams
Value * CreateAnd(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1163
Value * CreatePtrToInt(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1721
static const unsigned kOriginSize
const std::string to_string(const T &Value)
Definition: ScopedPrinter.h:61
static const MemoryMapParams FreeBSD_I386_MemoryMapParams
Analysis pass providing the TargetLibraryInfo.
bool isFPOrFPVectorTy() const
Return true if this is a FP type or a vector of FP.
Definition: Type.h:184
iterator_range< df_iterator< T > > depth_first(const T &G)
This represents the llvm.va_copy intrinsic.
bool isArrayAllocation() const
Return true if there is an allocation size parameter to the allocation instruction that is not 1...
size_type count(const KeyT &Val) const
Return 1 if the specified key is in the map, 0 otherwise.
Definition: ValueMap.h:157
assert(ImpDefSCC.getReg()==AMDGPU::SCC &&ImpDefSCC.isDef())
static cl::opt< int > ClInstrumentationWithCallThreshold("msan-instrumentation-with-call-threshold", cl::desc("If the function being instrumented requires more than " "this number of checks and origin stores, use callbacks instead of " "inline checks (-1 means never use callbacks)."), cl::Hidden, cl::init(3500))
unsigned getPrimitiveSizeInBits() const LLVM_READONLY
Return the basic size of this type if it is a primitive type.
Definition: Type.cpp:114
void setSuccessOrdering(AtomicOrdering Ordering)
Sets the success ordering constraint of this cmpxchg instruction.
Definition: Instructions.h:587
ArrayRef< unsigned > getIndices() const
Module * getParent()
Get the module that this global value is contained inside of...
Definition: GlobalValue.h:565
LLVM Value Representation.
Definition: Value.h:72
uint64_t getTypeStoreSize(Type *Ty) const
Returns the maximum number of bytes that may be overwritten by storing the specified type...
Definition: DataLayout.h:418
FunctionType * getFunctionType() const
Definition: CallSite.h:319
static VectorType * get(Type *ElementType, unsigned NumElements)
This static method is the primary way to construct an VectorType.
Definition: Type.cpp:605
#define LLVM_FALLTHROUGH
LLVM_FALLTHROUGH - Mark fallthrough cases in switch statements.
Definition: Compiler.h:250
std::underlying_type< E >::type Mask()
Get a bitmask with 1s in all places up to the high-order bit of E&#39;s largest value.
Definition: BitmaskEnum.h:80
unsigned getArgumentNo(Value::const_user_iterator I) const
Given a value use iterator, returns the argument that corresponds to it.
Definition: CallSite.h:198
AttributeSet getFnAttributes() const
The function attributes are returned.
BasicBlock::iterator GetInsertPoint() const
Definition: IRBuilder.h:121
BasicBlock * SplitBlock(BasicBlock *Old, Instruction *SplitPt, DominatorTree *DT=nullptr, LoopInfo *LI=nullptr, MemorySSAUpdater *MSSAU=nullptr)
Split the specified block at the specified instruction - everything before SplitPt stays in Old and e...
Value * CreateLShr(Value *LHS, Value *RHS, const Twine &Name="", bool isExact=false)
Definition: IRBuilder.h:1123
static cl::opt< bool > ClPoisonUndef("msan-poison-undef", cl::desc("poison undef temps"), cl::Hidden, cl::init(true))
ConstantInt * getInt8(uint8_t C)
Get a constant 8-bit value.
Definition: IRBuilder.h:296
StringRef - Represent a constant reference to a string, i.e.
Definition: StringRef.h:48
Predicate getSwappedPredicate() const
For example, EQ->EQ, SLE->SGE, ULT->UGT, OEQ->OEQ, ULE->UGE, OLT->OGT, etc.
Definition: InstrTypes.h:760
static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams
A container for analyses that lazily runs them and caches their results.
Type * getArrayElementType() const
Definition: Type.h:364
Value * CreateInsertValue(Value *Agg, Value *Val, ArrayRef< unsigned > Idxs, const Twine &Name="")
Definition: IRBuilder.h:2090
bool isBigEndian() const
Definition: DataLayout.h:221
#define LLVM_DEBUG(X)
Definition: Debug.h:122
Value * getPointerOperand()
Definition: Instructions.h:412
static IntegerType * getInt8Ty(LLVMContext &C)
Definition: Type.cpp:173
static const char *const kMsanModuleCtorName
static Constant * get(ArrayRef< Constant *> V)
Definition: Constants.cpp:1078
iterator_range< arg_iterator > args()
Definition: Function.h:688
std::pair< Function *, Function * > getOrCreateSanitizerCtorAndInitFunctions(Module &M, StringRef CtorName, StringRef InitName, ArrayRef< Type *> InitArgTypes, ArrayRef< Value *> InitArgs, function_ref< void(Function *, Function *)> FunctionsCreatedCallback, StringRef VersionCheckName=StringRef())
Creates sanitizer constructor function lazily.
signed greater or equal
Definition: InstrTypes.h:673
static ConstraintInfoVector ParseConstraints(StringRef ConstraintString)
ParseConstraints - Split up the constraint string into the specific constraints and their prefixes...
Definition: InlineAsm.cpp:207
bool isArrayTy() const
True if this is an instance of ArrayType.
Definition: Type.h:220
A wrapper class for inspecting calls to intrinsic functions.
Definition: IntrinsicInst.h:43
const BasicBlock * getParent() const
Definition: Instruction.h:66
an instruction to allocate memory on the stack
Definition: Instructions.h:59
This instruction inserts a struct field of array element value into an aggregate value.