LLVM  6.0.0svn
MemorySanitizer.cpp
Go to the documentation of this file.
1 //===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2 //
3 // The LLVM Compiler Infrastructure
4 //
5 // This file is distributed under the University of Illinois Open Source
6 // License. See LICENSE.TXT for details.
7 //
8 //===----------------------------------------------------------------------===//
9 //
10 /// \file
11 /// This file is a part of MemorySanitizer, a detector of uninitialized
12 /// reads.
13 ///
14 /// The algorithm of the tool is similar to Memcheck
15 /// (http://goo.gl/QKbem). We associate a few shadow bits with every
16 /// byte of the application memory, poison the shadow of the malloc-ed
17 /// or alloca-ed memory, load the shadow bits on every memory read,
18 /// propagate the shadow bits through some of the arithmetic
19 /// instruction (including MOV), store the shadow bits on every memory
20 /// write, report a bug on some other instructions (e.g. JMP) if the
21 /// associated shadow is poisoned.
22 ///
23 /// But there are differences too. The first and the major one:
24 /// compiler instrumentation instead of binary instrumentation. This
25 /// gives us much better register allocation, possible compiler
26 /// optimizations and a fast start-up. But this brings the major issue
27 /// as well: msan needs to see all program events, including system
28 /// calls and reads/writes in system libraries, so we either need to
29 /// compile *everything* with msan or use a binary translation
30 /// component (e.g. DynamoRIO) to instrument pre-built libraries.
31 /// Another difference from Memcheck is that we use 8 shadow bits per
32 /// byte of application memory and use a direct shadow mapping. This
33 /// greatly simplifies the instrumentation code and avoids races on
34 /// shadow updates (Memcheck is single-threaded so races are not a
35 /// concern there. Memcheck uses 2 shadow bits per byte with a slow
36 /// path storage that uses 8 bits per byte).
37 ///
38 /// The default value of shadow is 0, which means "clean" (not poisoned).
39 ///
40 /// Every module initializer should call __msan_init to ensure that the
41 /// shadow memory is ready. On error, __msan_warning is called. Since
42 /// parameters and return values may be passed via registers, we have a
43 /// specialized thread-local shadow for return values
44 /// (__msan_retval_tls) and parameters (__msan_param_tls).
45 ///
46 /// Origin tracking.
47 ///
48 /// MemorySanitizer can track origins (allocation points) of all uninitialized
49 /// values. This behavior is controlled with a flag (msan-track-origins) and is
50 /// disabled by default.
51 ///
52 /// Origins are 4-byte values created and interpreted by the runtime library.
53 /// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
54 /// of application memory. Propagation of origins is basically a bunch of
55 /// "select" instructions that pick the origin of a dirty argument, if an
56 /// instruction has one.
57 ///
58 /// Every 4 aligned, consecutive bytes of application memory have one origin
59 /// value associated with them. If these bytes contain uninitialized data
60 /// coming from 2 different allocations, the last store wins. Because of this,
61 /// MemorySanitizer reports can show unrelated origins, but this is unlikely in
62 /// practice.
63 ///
64 /// Origins are meaningless for fully initialized values, so MemorySanitizer
65 /// avoids storing origin to memory when a fully initialized value is stored.
66 /// This way it avoids needless overwritting origin of the 4-byte region on
67 /// a short (i.e. 1 byte) clean store, and it is also good for performance.
68 ///
69 /// Atomic handling.
70 ///
71 /// Ideally, every atomic store of application value should update the
72 /// corresponding shadow location in an atomic way. Unfortunately, atomic store
73 /// of two disjoint locations can not be done without severe slowdown.
74 ///
75 /// Therefore, we implement an approximation that may err on the safe side.
76 /// In this implementation, every atomically accessed location in the program
77 /// may only change from (partially) uninitialized to fully initialized, but
78 /// not the other way around. We load the shadow _after_ the application load,
79 /// and we store the shadow _before_ the app store. Also, we always store clean
80 /// shadow (if the application store is atomic). This way, if the store-load
81 /// pair constitutes a happens-before arc, shadow store and load are correctly
82 /// ordered such that the load will get either the value that was stored, or
83 /// some later value (which is always clean).
84 ///
85 /// This does not work very well with Compare-And-Swap (CAS) and
86 /// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
87 /// must store the new shadow before the app operation, and load the shadow
88 /// after the app operation. Computers don't work this way. Current
89 /// implementation ignores the load aspect of CAS/RMW, always returning a clean
90 /// value. It implements the store part as a simple atomic store by storing a
91 /// clean shadow.
92 //
93 //===----------------------------------------------------------------------===//
94 
95 #include "llvm/ADT/APInt.h"
96 #include "llvm/ADT/ArrayRef.h"
98 #include "llvm/ADT/SmallString.h"
99 #include "llvm/ADT/SmallVector.h"
100 #include "llvm/ADT/StringExtras.h"
101 #include "llvm/ADT/StringRef.h"
102 #include "llvm/ADT/Triple.h"
104 #include "llvm/IR/Argument.h"
105 #include "llvm/IR/Attributes.h"
106 #include "llvm/IR/BasicBlock.h"
107 #include "llvm/IR/CallSite.h"
108 #include "llvm/IR/CallingConv.h"
109 #include "llvm/IR/Constant.h"
110 #include "llvm/IR/Constants.h"
111 #include "llvm/IR/DataLayout.h"
112 #include "llvm/IR/DerivedTypes.h"
113 #include "llvm/IR/Function.h"
114 #include "llvm/IR/GlobalValue.h"
115 #include "llvm/IR/GlobalVariable.h"
116 #include "llvm/IR/IRBuilder.h"
117 #include "llvm/IR/InlineAsm.h"
118 #include "llvm/IR/InstVisitor.h"
119 #include "llvm/IR/InstrTypes.h"
120 #include "llvm/IR/Instruction.h"
121 #include "llvm/IR/Instructions.h"
122 #include "llvm/IR/IntrinsicInst.h"
123 #include "llvm/IR/Intrinsics.h"
124 #include "llvm/IR/LLVMContext.h"
125 #include "llvm/IR/MDBuilder.h"
126 #include "llvm/IR/Module.h"
127 #include "llvm/IR/Type.h"
128 #include "llvm/IR/Value.h"
129 #include "llvm/IR/ValueMap.h"
130 #include "llvm/Pass.h"
132 #include "llvm/Support/Casting.h"
134 #include "llvm/Support/Compiler.h"
135 #include "llvm/Support/Debug.h"
137 #include "llvm/Support/MathExtras.h"
143 #include <algorithm>
144 #include <cassert>
145 #include <cstddef>
146 #include <cstdint>
147 #include <memory>
148 #include <string>
149 #include <tuple>
150 
151 using namespace llvm;
152 
153 #define DEBUG_TYPE "msan"
154 
155 static const unsigned kOriginSize = 4;
156 static const unsigned kMinOriginAlignment = 4;
157 static const unsigned kShadowTLSAlignment = 8;
158 
159 // These constants must be kept in sync with the ones in msan.h.
160 static const unsigned kParamTLSSize = 800;
161 static const unsigned kRetvalTLSSize = 800;
162 
163 // Accesses sizes are powers of two: 1, 2, 4, 8.
164 static const size_t kNumberOfAccessSizes = 4;
165 
166 /// \brief Track origins of uninitialized values.
167 ///
168 /// Adds a section to MemorySanitizer report that points to the allocation
169 /// (stack or heap) the uninitialized bits came from originally.
170 static cl::opt<int> ClTrackOrigins("msan-track-origins",
171  cl::desc("Track origins (allocation sites) of poisoned memory"),
172  cl::Hidden, cl::init(0));
173 
174 static cl::opt<bool> ClKeepGoing("msan-keep-going",
175  cl::desc("keep going after reporting a UMR"),
176  cl::Hidden, cl::init(false));
177 
178 static cl::opt<bool> ClPoisonStack("msan-poison-stack",
179  cl::desc("poison uninitialized stack variables"),
180  cl::Hidden, cl::init(true));
181 
182 static cl::opt<bool> ClPoisonStackWithCall("msan-poison-stack-with-call",
183  cl::desc("poison uninitialized stack variables with a call"),
184  cl::Hidden, cl::init(false));
185 
186 static cl::opt<int> ClPoisonStackPattern("msan-poison-stack-pattern",
187  cl::desc("poison uninitialized stack variables with the given pattern"),
188  cl::Hidden, cl::init(0xff));
189 
190 static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
191  cl::desc("poison undef temps"),
192  cl::Hidden, cl::init(true));
193 
194 static cl::opt<bool> ClHandleICmp("msan-handle-icmp",
195  cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
196  cl::Hidden, cl::init(true));
197 
198 static cl::opt<bool> ClHandleICmpExact("msan-handle-icmp-exact",
199  cl::desc("exact handling of relational integer ICmp"),
200  cl::Hidden, cl::init(false));
201 
202 // This flag controls whether we check the shadow of the address
203 // operand of load or store. Such bugs are very rare, since load from
204 // a garbage address typically results in SEGV, but still happen
205 // (e.g. only lower bits of address are garbage, or the access happens
206 // early at program startup where malloc-ed memory is more likely to
207 // be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
208 static cl::opt<bool> ClCheckAccessAddress("msan-check-access-address",
209  cl::desc("report accesses through a pointer which has poisoned shadow"),
210  cl::Hidden, cl::init(true));
211 
212 static cl::opt<bool> ClDumpStrictInstructions("msan-dump-strict-instructions",
213  cl::desc("print out instructions with default strict semantics"),
214  cl::Hidden, cl::init(false));
215 
217  "msan-instrumentation-with-call-threshold",
218  cl::desc(
219  "If the function being instrumented requires more than "
220  "this number of checks and origin stores, use callbacks instead of "
221  "inline checks (-1 means never use callbacks)."),
222  cl::Hidden, cl::init(3500));
223 
224 // This is an experiment to enable handling of cases where shadow is a non-zero
225 // compile-time constant. For some unexplainable reason they were silently
226 // ignored in the instrumentation.
227 static cl::opt<bool> ClCheckConstantShadow("msan-check-constant-shadow",
228  cl::desc("Insert checks for constant shadow values"),
229  cl::Hidden, cl::init(false));
230 
231 // This is off by default because of a bug in gold:
232 // https://sourceware.org/bugzilla/show_bug.cgi?id=19002
233 static cl::opt<bool> ClWithComdat("msan-with-comdat",
234  cl::desc("Place MSan constructors in comdat sections"),
235  cl::Hidden, cl::init(false));
236 
237 static const char *const kMsanModuleCtorName = "msan.module_ctor";
238 static const char *const kMsanInitName = "__msan_init";
239 
240 namespace {
241 
242 // Memory map parameters used in application-to-shadow address calculation.
243 // Offset = (Addr & ~AndMask) ^ XorMask
244 // Shadow = ShadowBase + Offset
245 // Origin = OriginBase + Offset
246 struct MemoryMapParams {
247  uint64_t AndMask;
248  uint64_t XorMask;
249  uint64_t ShadowBase;
250  uint64_t OriginBase;
251 };
252 
253 struct PlatformMemoryMapParams {
254  const MemoryMapParams *bits32;
255  const MemoryMapParams *bits64;
256 };
257 
258 } // end anonymous namespace
259 
260 // i386 Linux
261 static const MemoryMapParams Linux_I386_MemoryMapParams = {
262  0x000080000000, // AndMask
263  0, // XorMask (not used)
264  0, // ShadowBase (not used)
265  0x000040000000, // OriginBase
266 };
267 
268 // x86_64 Linux
269 static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
270 #ifdef MSAN_LINUX_X86_64_OLD_MAPPING
271  0x400000000000, // AndMask
272  0, // XorMask (not used)
273  0, // ShadowBase (not used)
274  0x200000000000, // OriginBase
275 #else
276  0, // AndMask (not used)
277  0x500000000000, // XorMask
278  0, // ShadowBase (not used)
279  0x100000000000, // OriginBase
280 #endif
281 };
282 
283 // mips64 Linux
284 static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
285  0, // AndMask (not used)
286  0x008000000000, // XorMask
287  0, // ShadowBase (not used)
288  0x002000000000, // OriginBase
289 };
290 
291 // ppc64 Linux
292 static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
293  0xE00000000000, // AndMask
294  0x100000000000, // XorMask
295  0x080000000000, // ShadowBase
296  0x1C0000000000, // OriginBase
297 };
298 
299 // aarch64 Linux
300 static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
301  0, // AndMask (not used)
302  0x06000000000, // XorMask
303  0, // ShadowBase (not used)
304  0x01000000000, // OriginBase
305 };
306 
307 // i386 FreeBSD
308 static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
309  0x000180000000, // AndMask
310  0x000040000000, // XorMask
311  0x000020000000, // ShadowBase
312  0x000700000000, // OriginBase
313 };
314 
315 // x86_64 FreeBSD
316 static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
317  0xc00000000000, // AndMask
318  0x200000000000, // XorMask
319  0x100000000000, // ShadowBase
320  0x380000000000, // OriginBase
321 };
322 
323 static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
326 };
327 
328 static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
329  nullptr,
331 };
332 
333 static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
334  nullptr,
336 };
337 
338 static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
339  nullptr,
341 };
342 
343 static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
346 };
347 
348 namespace {
349 
350 /// \brief An instrumentation pass implementing detection of uninitialized
351 /// reads.
352 ///
353 /// MemorySanitizer: instrument the code in module to find
354 /// uninitialized reads.
355 class MemorySanitizer : public FunctionPass {
356 public:
357  // Pass identification, replacement for typeid.
358  static char ID;
359 
360  MemorySanitizer(int TrackOrigins = 0, bool Recover = false)
361  : FunctionPass(ID),
362  TrackOrigins(std::max(TrackOrigins, (int)ClTrackOrigins)),
363  Recover(Recover || ClKeepGoing) {}
364 
365  StringRef getPassName() const override { return "MemorySanitizer"; }
366 
367  void getAnalysisUsage(AnalysisUsage &AU) const override {
369  }
370 
371  bool runOnFunction(Function &F) override;
372  bool doInitialization(Module &M) override;
373 
374 private:
375  friend struct MemorySanitizerVisitor;
376  friend struct VarArgAMD64Helper;
377  friend struct VarArgMIPS64Helper;
378  friend struct VarArgAArch64Helper;
379  friend struct VarArgPowerPC64Helper;
380 
381  void initializeCallbacks(Module &M);
382 
383  /// \brief Track origins (allocation points) of uninitialized values.
384  int TrackOrigins;
385  bool Recover;
386 
387  LLVMContext *C;
388  Type *IntptrTy;
389  Type *OriginTy;
390 
391  /// \brief Thread-local shadow storage for function parameters.
392  GlobalVariable *ParamTLS;
393 
394  /// \brief Thread-local origin storage for function parameters.
395  GlobalVariable *ParamOriginTLS;
396 
397  /// \brief Thread-local shadow storage for function return value.
398  GlobalVariable *RetvalTLS;
399 
400  /// \brief Thread-local origin storage for function return value.
401  GlobalVariable *RetvalOriginTLS;
402 
403  /// \brief Thread-local shadow storage for in-register va_arg function
404  /// parameters (x86_64-specific).
405  GlobalVariable *VAArgTLS;
406 
407  /// \brief Thread-local shadow storage for va_arg overflow area
408  /// (x86_64-specific).
409  GlobalVariable *VAArgOverflowSizeTLS;
410 
411  /// \brief Thread-local space used to pass origin value to the UMR reporting
412  /// function.
413  GlobalVariable *OriginTLS;
414 
415  /// \brief The run-time callback to print a warning.
416  Value *WarningFn = nullptr;
417 
418  // These arrays are indexed by log2(AccessSize).
419  Value *MaybeWarningFn[kNumberOfAccessSizes];
420  Value *MaybeStoreOriginFn[kNumberOfAccessSizes];
421 
422  /// \brief Run-time helper that generates a new origin value for a stack
423  /// allocation.
424  Value *MsanSetAllocaOrigin4Fn;
425 
426  /// \brief Run-time helper that poisons stack on function entry.
427  Value *MsanPoisonStackFn;
428 
429  /// \brief Run-time helper that records a store (or any event) of an
430  /// uninitialized value and returns an updated origin id encoding this info.
431  Value *MsanChainOriginFn;
432 
433  /// \brief MSan runtime replacements for memmove, memcpy and memset.
434  Value *MemmoveFn, *MemcpyFn, *MemsetFn;
435 
436  /// \brief Memory map parameters used in application-to-shadow calculation.
437  const MemoryMapParams *MapParams;
438 
439  MDNode *ColdCallWeights;
440 
441  /// \brief Branch weights for origin store.
442  MDNode *OriginStoreWeights;
443 
444  /// \brief An empty volatile inline asm that prevents callback merge.
445  InlineAsm *EmptyAsm;
446 
447  Function *MsanCtorFunction;
448 };
449 
450 } // end anonymous namespace
451 
452 char MemorySanitizer::ID = 0;
453 
455  MemorySanitizer, "msan",
456  "MemorySanitizer: detects uninitialized reads.", false, false)
459  MemorySanitizer, "msan",
460  "MemorySanitizer: detects uninitialized reads.", false, false)
461 
462 FunctionPass *llvm::createMemorySanitizerPass(int TrackOrigins, bool Recover) {
463  return new MemorySanitizer(TrackOrigins, Recover);
464 }
465 
466 /// \brief Create a non-const global initialized with the given string.
467 ///
468 /// Creates a writable global for Str so that we can pass it to the
469 /// run-time lib. Runtime uses first 4 bytes of the string to store the
470 /// frame ID, so the string needs to be mutable.
472  StringRef Str) {
473  Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
474  return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/false,
475  GlobalValue::PrivateLinkage, StrConst, "");
476 }
477 
478 /// \brief Insert extern declaration of runtime-provided functions and globals.
479 void MemorySanitizer::initializeCallbacks(Module &M) {
480  // Only do this once.
481  if (WarningFn)
482  return;
483 
484  IRBuilder<> IRB(*C);
485  // Create the callback.
486  // FIXME: this function should have "Cold" calling conv,
487  // which is not yet implemented.
488  StringRef WarningFnName = Recover ? "__msan_warning"
489  : "__msan_warning_noreturn";
490  WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy());
491 
492  for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
493  AccessSizeIndex++) {
494  unsigned AccessSize = 1 << AccessSizeIndex;
495  std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
496  MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
497  FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8),
498  IRB.getInt32Ty());
499 
500  FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
501  MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
502  FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8),
503  IRB.getInt8PtrTy(), IRB.getInt32Ty());
504  }
505 
506  MsanSetAllocaOrigin4Fn = M.getOrInsertFunction(
507  "__msan_set_alloca_origin4", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy,
508  IRB.getInt8PtrTy(), IntptrTy);
509  MsanPoisonStackFn =
510  M.getOrInsertFunction("__msan_poison_stack", IRB.getVoidTy(),
511  IRB.getInt8PtrTy(), IntptrTy);
512  MsanChainOriginFn = M.getOrInsertFunction(
513  "__msan_chain_origin", IRB.getInt32Ty(), IRB.getInt32Ty());
514  MemmoveFn = M.getOrInsertFunction(
515  "__msan_memmove", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
516  IRB.getInt8PtrTy(), IntptrTy);
517  MemcpyFn = M.getOrInsertFunction(
518  "__msan_memcpy", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
519  IntptrTy);
520  MemsetFn = M.getOrInsertFunction(
521  "__msan_memset", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(),
522  IntptrTy);
523 
524  // Create globals.
525  RetvalTLS = new GlobalVariable(
526  M, ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8), false,
527  GlobalVariable::ExternalLinkage, nullptr, "__msan_retval_tls", nullptr,
529  RetvalOriginTLS = new GlobalVariable(
530  M, OriginTy, false, GlobalVariable::ExternalLinkage, nullptr,
531  "__msan_retval_origin_tls", nullptr, GlobalVariable::InitialExecTLSModel);
532 
533  ParamTLS = new GlobalVariable(
534  M, ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), false,
535  GlobalVariable::ExternalLinkage, nullptr, "__msan_param_tls", nullptr,
537  ParamOriginTLS = new GlobalVariable(
538  M, ArrayType::get(OriginTy, kParamTLSSize / 4), false,
539  GlobalVariable::ExternalLinkage, nullptr, "__msan_param_origin_tls",
541 
542  VAArgTLS = new GlobalVariable(
543  M, ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), false,
544  GlobalVariable::ExternalLinkage, nullptr, "__msan_va_arg_tls", nullptr,
546  VAArgOverflowSizeTLS = new GlobalVariable(
547  M, IRB.getInt64Ty(), false, GlobalVariable::ExternalLinkage, nullptr,
548  "__msan_va_arg_overflow_size_tls", nullptr,
550  OriginTLS = new GlobalVariable(
551  M, IRB.getInt32Ty(), false, GlobalVariable::ExternalLinkage, nullptr,
552  "__msan_origin_tls", nullptr, GlobalVariable::InitialExecTLSModel);
553 
554  // We insert an empty inline asm after __msan_report* to avoid callback merge.
555  EmptyAsm = InlineAsm::get(FunctionType::get(IRB.getVoidTy(), false),
556  StringRef(""), StringRef(""),
557  /*hasSideEffects=*/true);
558 }
559 
560 /// \brief Module-level initialization.
561 ///
562 /// inserts a call to __msan_init to the module's constructor list.
563 bool MemorySanitizer::doInitialization(Module &M) {
564  auto &DL = M.getDataLayout();
565 
566  Triple TargetTriple(M.getTargetTriple());
567  switch (TargetTriple.getOS()) {
568  case Triple::FreeBSD:
569  switch (TargetTriple.getArch()) {
570  case Triple::x86_64:
571  MapParams = FreeBSD_X86_MemoryMapParams.bits64;
572  break;
573  case Triple::x86:
574  MapParams = FreeBSD_X86_MemoryMapParams.bits32;
575  break;
576  default:
577  report_fatal_error("unsupported architecture");
578  }
579  break;
580  case Triple::Linux:
581  switch (TargetTriple.getArch()) {
582  case Triple::x86_64:
583  MapParams = Linux_X86_MemoryMapParams.bits64;
584  break;
585  case Triple::x86:
586  MapParams = Linux_X86_MemoryMapParams.bits32;
587  break;
588  case Triple::mips64:
589  case Triple::mips64el:
590  MapParams = Linux_MIPS_MemoryMapParams.bits64;
591  break;
592  case Triple::ppc64:
593  case Triple::ppc64le:
594  MapParams = Linux_PowerPC_MemoryMapParams.bits64;
595  break;
596  case Triple::aarch64:
597  case Triple::aarch64_be:
598  MapParams = Linux_ARM_MemoryMapParams.bits64;
599  break;
600  default:
601  report_fatal_error("unsupported architecture");
602  }
603  break;
604  default:
605  report_fatal_error("unsupported operating system");
606  }
607 
608  C = &(M.getContext());
609  IRBuilder<> IRB(*C);
610  IntptrTy = IRB.getIntPtrTy(DL);
611  OriginTy = IRB.getInt32Ty();
612 
613  ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000);
614  OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000);
615 
616  std::tie(MsanCtorFunction, std::ignore) =
618  /*InitArgTypes=*/{},
619  /*InitArgs=*/{});
620  if (ClWithComdat) {
621  Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
622  MsanCtorFunction->setComdat(MsanCtorComdat);
623  appendToGlobalCtors(M, MsanCtorFunction, 0, MsanCtorFunction);
624  } else {
625  appendToGlobalCtors(M, MsanCtorFunction, 0);
626  }
627 
628 
629  if (TrackOrigins)
631  IRB.getInt32(TrackOrigins), "__msan_track_origins");
632 
633  if (Recover)
635  IRB.getInt32(Recover), "__msan_keep_going");
636 
637  return true;
638 }
639 
640 namespace {
641 
642 /// \brief A helper class that handles instrumentation of VarArg
643 /// functions on a particular platform.
644 ///
645 /// Implementations are expected to insert the instrumentation
646 /// necessary to propagate argument shadow through VarArg function
647 /// calls. Visit* methods are called during an InstVisitor pass over
648 /// the function, and should avoid creating new basic blocks. A new
649 /// instance of this class is created for each instrumented function.
650 struct VarArgHelper {
651  virtual ~VarArgHelper() = default;
652 
653  /// \brief Visit a CallSite.
654  virtual void visitCallSite(CallSite &CS, IRBuilder<> &IRB) = 0;
655 
656  /// \brief Visit a va_start call.
657  virtual void visitVAStartInst(VAStartInst &I) = 0;
658 
659  /// \brief Visit a va_copy call.
660  virtual void visitVACopyInst(VACopyInst &I) = 0;
661 
662  /// \brief Finalize function instrumentation.
663  ///
664  /// This method is called after visiting all interesting (see above)
665  /// instructions in a function.
666  virtual void finalizeInstrumentation() = 0;
667 };
668 
669 struct MemorySanitizerVisitor;
670 
671 } // end anonymous namespace
672 
673 static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
674  MemorySanitizerVisitor &Visitor);
675 
676 static unsigned TypeSizeToSizeIndex(unsigned TypeSize) {
677  if (TypeSize <= 8) return 0;
678  return Log2_32_Ceil((TypeSize + 7) / 8);
679 }
680 
681 namespace {
682 
683 /// This class does all the work for a given function. Store and Load
684 /// instructions store and load corresponding shadow and origin
685 /// values. Most instructions propagate shadow from arguments to their
686 /// return values. Certain instructions (most importantly, BranchInst)
687 /// test their argument shadow and print reports (with a runtime call) if it's
688 /// non-zero.
689 struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
690  Function &F;
691  MemorySanitizer &MS;
692  SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
693  ValueMap<Value*, Value*> ShadowMap, OriginMap;
694  std::unique_ptr<VarArgHelper> VAHelper;
695  const TargetLibraryInfo *TLI;
696 
697  // The following flags disable parts of MSan instrumentation based on
698  // blacklist contents and command-line options.
699  bool InsertChecks;
700  bool PropagateShadow;
701  bool PoisonStack;
702  bool PoisonUndef;
703  bool CheckReturnValue;
704 
705  struct ShadowOriginAndInsertPoint {
706  Value *Shadow;
707  Value *Origin;
708  Instruction *OrigIns;
709 
710  ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
711  : Shadow(S), Origin(O), OrigIns(I) {}
712  };
715 
716  MemorySanitizerVisitor(Function &F, MemorySanitizer &MS)
717  : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)) {
718  bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeMemory);
719  InsertChecks = SanitizeFunction;
720  PropagateShadow = SanitizeFunction;
721  PoisonStack = SanitizeFunction && ClPoisonStack;
722  PoisonUndef = SanitizeFunction && ClPoisonUndef;
723  // FIXME: Consider using SpecialCaseList to specify a list of functions that
724  // must always return fully initialized values. For now, we hardcode "main".
725  CheckReturnValue = SanitizeFunction && (F.getName() == "main");
726  TLI = &MS.getAnalysis<TargetLibraryInfoWrapperPass>().getTLI();
727 
728  DEBUG(if (!InsertChecks)
729  dbgs() << "MemorySanitizer is not inserting checks into '"
730  << F.getName() << "'\n");
731  }
732 
733  Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
734  if (MS.TrackOrigins <= 1) return V;
735  return IRB.CreateCall(MS.MsanChainOriginFn, V);
736  }
737 
738  Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
739  const DataLayout &DL = F.getParent()->getDataLayout();
740  unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
741  if (IntptrSize == kOriginSize) return Origin;
742  assert(IntptrSize == kOriginSize * 2);
743  Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
744  return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
745  }
746 
747  /// \brief Fill memory range with the given origin value.
748  void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
749  unsigned Size, unsigned Alignment) {
750  const DataLayout &DL = F.getParent()->getDataLayout();
751  unsigned IntptrAlignment = DL.getABITypeAlignment(MS.IntptrTy);
752  unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
753  assert(IntptrAlignment >= kMinOriginAlignment);
754  assert(IntptrSize >= kOriginSize);
755 
756  unsigned Ofs = 0;
757  unsigned CurrentAlignment = Alignment;
758  if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
759  Value *IntptrOrigin = originToIntptr(IRB, Origin);
760  Value *IntptrOriginPtr =
761  IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
762  for (unsigned i = 0; i < Size / IntptrSize; ++i) {
763  Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
764  : IntptrOriginPtr;
765  IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
766  Ofs += IntptrSize / kOriginSize;
767  CurrentAlignment = IntptrAlignment;
768  }
769  }
770 
771  for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) {
772  Value *GEP =
773  i ? IRB.CreateConstGEP1_32(nullptr, OriginPtr, i) : OriginPtr;
774  IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
775  CurrentAlignment = kMinOriginAlignment;
776  }
777  }
778 
779  void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
780  unsigned Alignment, bool AsCall) {
781  const DataLayout &DL = F.getParent()->getDataLayout();
782  unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
783  unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
784  if (Shadow->getType()->isAggregateType()) {
785  paintOrigin(IRB, updateOrigin(Origin, IRB),
786  getOriginPtr(Addr, IRB, Alignment), StoreSize,
787  OriginAlignment);
788  } else {
789  Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB);
790  Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow);
791  if (ConstantShadow) {
792  if (ClCheckConstantShadow && !ConstantShadow->isZeroValue())
793  paintOrigin(IRB, updateOrigin(Origin, IRB),
794  getOriginPtr(Addr, IRB, Alignment), StoreSize,
795  OriginAlignment);
796  return;
797  }
798 
799  unsigned TypeSizeInBits =
800  DL.getTypeSizeInBits(ConvertedShadow->getType());
801  unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
802  if (AsCall && SizeIndex < kNumberOfAccessSizes) {
803  Value *Fn = MS.MaybeStoreOriginFn[SizeIndex];
804  Value *ConvertedShadow2 = IRB.CreateZExt(
805  ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
806  IRB.CreateCall(Fn, {ConvertedShadow2,
807  IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()),
808  Origin});
809  } else {
810  Value *Cmp = IRB.CreateICmpNE(
811  ConvertedShadow, getCleanShadow(ConvertedShadow), "_mscmp");
813  Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
814  IRBuilder<> IRBNew(CheckTerm);
815  paintOrigin(IRBNew, updateOrigin(Origin, IRBNew),
816  getOriginPtr(Addr, IRBNew, Alignment), StoreSize,
817  OriginAlignment);
818  }
819  }
820  }
821 
822  void materializeStores(bool InstrumentWithCalls) {
823  for (StoreInst *SI : StoreList) {
824  IRBuilder<> IRB(SI);
825  Value *Val = SI->getValueOperand();
826  Value *Addr = SI->getPointerOperand();
827  Value *Shadow = SI->isAtomic() ? getCleanShadow(Val) : getShadow(Val);
828  Value *ShadowPtr = getShadowPtr(Addr, Shadow->getType(), IRB);
829 
830  StoreInst *NewSI =
831  IRB.CreateAlignedStore(Shadow, ShadowPtr, SI->getAlignment());
832  DEBUG(dbgs() << " STORE: " << *NewSI << "\n");
833  (void)NewSI;
834 
836  insertShadowCheck(Addr, SI);
837 
838  if (SI->isAtomic())
839  SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
840 
841  if (MS.TrackOrigins && !SI->isAtomic())
842  storeOrigin(IRB, Addr, Shadow, getOrigin(Val), SI->getAlignment(),
843  InstrumentWithCalls);
844  }
845  }
846 
847  void materializeOneCheck(Instruction *OrigIns, Value *Shadow, Value *Origin,
848  bool AsCall) {
849  IRBuilder<> IRB(OrigIns);
850  DEBUG(dbgs() << " SHAD0 : " << *Shadow << "\n");
851  Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB);
852  DEBUG(dbgs() << " SHAD1 : " << *ConvertedShadow << "\n");
853 
854  Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow);
855  if (ConstantShadow) {
856  if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) {
857  if (MS.TrackOrigins) {
858  IRB.CreateStore(Origin ? (Value *)Origin : (Value *)IRB.getInt32(0),
859  MS.OriginTLS);
860  }
861  IRB.CreateCall(MS.WarningFn, {});
862  IRB.CreateCall(MS.EmptyAsm, {});
863  // FIXME: Insert UnreachableInst if !MS.Recover?
864  // This may invalidate some of the following checks and needs to be done
865  // at the very end.
866  }
867  return;
868  }
869 
870  const DataLayout &DL = OrigIns->getModule()->getDataLayout();
871 
872  unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
873  unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
874  if (AsCall && SizeIndex < kNumberOfAccessSizes) {
875  Value *Fn = MS.MaybeWarningFn[SizeIndex];
876  Value *ConvertedShadow2 =
877  IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
878  IRB.CreateCall(Fn, {ConvertedShadow2, MS.TrackOrigins && Origin
879  ? Origin
880  : (Value *)IRB.getInt32(0)});
881  } else {
882  Value *Cmp = IRB.CreateICmpNE(ConvertedShadow,
883  getCleanShadow(ConvertedShadow), "_mscmp");
885  Cmp, OrigIns,
886  /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
887 
888  IRB.SetInsertPoint(CheckTerm);
889  if (MS.TrackOrigins) {
890  IRB.CreateStore(Origin ? (Value *)Origin : (Value *)IRB.getInt32(0),
891  MS.OriginTLS);
892  }
893  IRB.CreateCall(MS.WarningFn, {});
894  IRB.CreateCall(MS.EmptyAsm, {});
895  DEBUG(dbgs() << " CHECK: " << *Cmp << "\n");
896  }
897  }
898 
899  void materializeChecks(bool InstrumentWithCalls) {
900  for (const auto &ShadowData : InstrumentationList) {
901  Instruction *OrigIns = ShadowData.OrigIns;
902  Value *Shadow = ShadowData.Shadow;
903  Value *Origin = ShadowData.Origin;
904  materializeOneCheck(OrigIns, Shadow, Origin, InstrumentWithCalls);
905  }
906  DEBUG(dbgs() << "DONE:\n" << F);
907  }
908 
909  /// \brief Add MemorySanitizer instrumentation to a function.
910  bool runOnFunction() {
911  MS.initializeCallbacks(*F.getParent());
912 
913  // In the presence of unreachable blocks, we may see Phi nodes with
914  // incoming nodes from such blocks. Since InstVisitor skips unreachable
915  // blocks, such nodes will not have any shadow value associated with them.
916  // It's easier to remove unreachable blocks than deal with missing shadow.
918 
919  // Iterate all BBs in depth-first order and create shadow instructions
920  // for all instructions (where applicable).
921  // For PHI nodes we create dummy shadow PHIs which will be finalized later.
922  for (BasicBlock *BB : depth_first(&F.getEntryBlock()))
923  visit(*BB);
924 
925  // Finalize PHI nodes.
926  for (PHINode *PN : ShadowPHINodes) {
927  PHINode *PNS = cast<PHINode>(getShadow(PN));
928  PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
929  size_t NumValues = PN->getNumIncomingValues();
930  for (size_t v = 0; v < NumValues; v++) {
931  PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
932  if (PNO) PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v));
933  }
934  }
935 
936  VAHelper->finalizeInstrumentation();
937 
938  bool InstrumentWithCalls = ClInstrumentationWithCallThreshold >= 0 &&
939  InstrumentationList.size() + StoreList.size() >
941 
942  // Delayed instrumentation of StoreInst.
943  // This may add new checks to be inserted later.
944  materializeStores(InstrumentWithCalls);
945 
946  // Insert shadow value checks.
947  materializeChecks(InstrumentWithCalls);
948 
949  return true;
950  }
951 
952  /// \brief Compute the shadow type that corresponds to a given Value.
953  Type *getShadowTy(Value *V) {
954  return getShadowTy(V->getType());
955  }
956 
957  /// \brief Compute the shadow type that corresponds to a given Type.
958  Type *getShadowTy(Type *OrigTy) {
959  if (!OrigTy->isSized()) {
960  return nullptr;
961  }
962  // For integer type, shadow is the same as the original type.
963  // This may return weird-sized types like i1.
964  if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
965  return IT;
966  const DataLayout &DL = F.getParent()->getDataLayout();
967  if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
968  uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
969  return VectorType::get(IntegerType::get(*MS.C, EltSize),
970  VT->getNumElements());
971  }
972  if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
973  return ArrayType::get(getShadowTy(AT->getElementType()),
974  AT->getNumElements());
975  }
976  if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
977  SmallVector<Type*, 4> Elements;
978  for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
979  Elements.push_back(getShadowTy(ST->getElementType(i)));
980  StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
981  DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
982  return Res;
983  }
984  uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
985  return IntegerType::get(*MS.C, TypeSize);
986  }
987 
988  /// \brief Flatten a vector type.
989  Type *getShadowTyNoVec(Type *ty) {
990  if (VectorType *vt = dyn_cast<VectorType>(ty))
991  return IntegerType::get(*MS.C, vt->getBitWidth());
992  return ty;
993  }
994 
995  /// \brief Convert a shadow value to it's flattened variant.
996  Value *convertToShadowTyNoVec(Value *V, IRBuilder<> &IRB) {
997  Type *Ty = V->getType();
998  Type *NoVecTy = getShadowTyNoVec(Ty);
999  if (Ty == NoVecTy) return V;
1000  return IRB.CreateBitCast(V, NoVecTy);
1001  }
1002 
1003  /// \brief Compute the integer shadow offset that corresponds to a given
1004  /// application address.
1005  ///
1006  /// Offset = (Addr & ~AndMask) ^ XorMask
1007  Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1008  Value *OffsetLong = IRB.CreatePointerCast(Addr, MS.IntptrTy);
1009 
1010  uint64_t AndMask = MS.MapParams->AndMask;
1011  if (AndMask)
1012  OffsetLong =
1013  IRB.CreateAnd(OffsetLong, ConstantInt::get(MS.IntptrTy, ~AndMask));
1014 
1015  uint64_t XorMask = MS.MapParams->XorMask;
1016  if (XorMask)
1017  OffsetLong =
1018  IRB.CreateXor(OffsetLong, ConstantInt::get(MS.IntptrTy, XorMask));
1019  return OffsetLong;
1020  }
1021 
1022  /// \brief Compute the shadow address that corresponds to a given application
1023  /// address.
1024  ///
1025  /// Shadow = ShadowBase + Offset
1026  Value *getShadowPtr(Value *Addr, Type *ShadowTy,
1027  IRBuilder<> &IRB) {
1028  Value *ShadowLong = getShadowPtrOffset(Addr, IRB);
1029  uint64_t ShadowBase = MS.MapParams->ShadowBase;
1030  if (ShadowBase != 0)
1031  ShadowLong =
1032  IRB.CreateAdd(ShadowLong,
1033  ConstantInt::get(MS.IntptrTy, ShadowBase));
1034  return IRB.CreateIntToPtr(ShadowLong, PointerType::get(ShadowTy, 0));
1035  }
1036 
1037  /// \brief Compute the origin address that corresponds to a given application
1038  /// address.
1039  ///
1040  /// OriginAddr = (OriginBase + Offset) & ~3ULL
1041  Value *getOriginPtr(Value *Addr, IRBuilder<> &IRB, unsigned Alignment) {
1042  Value *OriginLong = getShadowPtrOffset(Addr, IRB);
1043  uint64_t OriginBase = MS.MapParams->OriginBase;
1044  if (OriginBase != 0)
1045  OriginLong =
1046  IRB.CreateAdd(OriginLong,
1047  ConstantInt::get(MS.IntptrTy, OriginBase));
1048  if (Alignment < kMinOriginAlignment) {
1049  uint64_t Mask = kMinOriginAlignment - 1;
1050  OriginLong = IRB.CreateAnd(OriginLong,
1051  ConstantInt::get(MS.IntptrTy, ~Mask));
1052  }
1053  return IRB.CreateIntToPtr(OriginLong,
1054  PointerType::get(IRB.getInt32Ty(), 0));
1055  }
1056 
1057  /// \brief Compute the shadow address for a given function argument.
1058  ///
1059  /// Shadow = ParamTLS+ArgOffset.
1060  Value *getShadowPtrForArgument(Value *A, IRBuilder<> &IRB,
1061  int ArgOffset) {
1062  Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1063  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1064  return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0),
1065  "_msarg");
1066  }
1067 
1068  /// \brief Compute the origin address for a given function argument.
1069  Value *getOriginPtrForArgument(Value *A, IRBuilder<> &IRB,
1070  int ArgOffset) {
1071  if (!MS.TrackOrigins) return nullptr;
1072  Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1073  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1074  return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
1075  "_msarg_o");
1076  }
1077 
1078  /// \brief Compute the shadow address for a retval.
1079  Value *getShadowPtrForRetval(Value *A, IRBuilder<> &IRB) {
1080  Value *Base = IRB.CreatePointerCast(MS.RetvalTLS, MS.IntptrTy);
1081  return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0),
1082  "_msret");
1083  }
1084 
1085  /// \brief Compute the origin address for a retval.
1086  Value *getOriginPtrForRetval(IRBuilder<> &IRB) {
1087  // We keep a single origin for the entire retval. Might be too optimistic.
1088  return MS.RetvalOriginTLS;
1089  }
1090 
1091  /// \brief Set SV to be the shadow value for V.
1092  void setShadow(Value *V, Value *SV) {
1093  assert(!ShadowMap.count(V) && "Values may only have one shadow");
1094  ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V);
1095  }
1096 
1097  /// \brief Set Origin to be the origin value for V.
1098  void setOrigin(Value *V, Value *Origin) {
1099  if (!MS.TrackOrigins) return;
1100  assert(!OriginMap.count(V) && "Values may only have one origin");
1101  DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n");
1102  OriginMap[V] = Origin;
1103  }
1104 
1105  Constant *getCleanShadow(Type *OrigTy) {
1106  Type *ShadowTy = getShadowTy(OrigTy);
1107  if (!ShadowTy)
1108  return nullptr;
1109  return Constant::getNullValue(ShadowTy);
1110  }
1111 
1112  /// \brief Create a clean shadow value for a given value.
1113  ///
1114  /// Clean shadow (all zeroes) means all bits of the value are defined
1115  /// (initialized).
1116  Constant *getCleanShadow(Value *V) {
1117  return getCleanShadow(V->getType());
1118  }
1119 
1120  /// \brief Create a dirty shadow of a given shadow type.
1121  Constant *getPoisonedShadow(Type *ShadowTy) {
1122  assert(ShadowTy);
1123  if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy))
1124  return Constant::getAllOnesValue(ShadowTy);
1125  if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1126  SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1127  getPoisonedShadow(AT->getElementType()));
1128  return ConstantArray::get(AT, Vals);
1129  }
1130  if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1132  for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1133  Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1134  return ConstantStruct::get(ST, Vals);
1135  }
1136  llvm_unreachable("Unexpected shadow type");
1137  }
1138 
1139  /// \brief Create a dirty shadow for a given value.
1140  Constant *getPoisonedShadow(Value *V) {
1141  Type *ShadowTy = getShadowTy(V);
1142  if (!ShadowTy)
1143  return nullptr;
1144  return getPoisonedShadow(ShadowTy);
1145  }
1146 
1147  /// \brief Create a clean (zero) origin.
1148  Value *getCleanOrigin() {
1149  return Constant::getNullValue(MS.OriginTy);
1150  }
1151 
1152  /// \brief Get the shadow value for a given Value.
1153  ///
1154  /// This function either returns the value set earlier with setShadow,
1155  /// or extracts if from ParamTLS (for function arguments).
1156  Value *getShadow(Value *V) {
1157  if (!PropagateShadow) return getCleanShadow(V);
1158  if (Instruction *I = dyn_cast<Instruction>(V)) {
1159  // For instructions the shadow is already stored in the map.
1160  Value *Shadow = ShadowMap[V];
1161  if (!Shadow) {
1162  DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1163  (void)I;
1164  assert(Shadow && "No shadow for a value");
1165  }
1166  return Shadow;
1167  }
1168  if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1169  Value *AllOnes = PoisonUndef ? getPoisonedShadow(V) : getCleanShadow(V);
1170  DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1171  (void)U;
1172  return AllOnes;
1173  }
1174  if (Argument *A = dyn_cast<Argument>(V)) {
1175  // For arguments we compute the shadow on demand and store it in the map.
1176  Value **ShadowPtr = &ShadowMap[V];
1177  if (*ShadowPtr)
1178  return *ShadowPtr;
1179  Function *F = A->getParent();
1180  IRBuilder<> EntryIRB(F->getEntryBlock().getFirstNonPHI());
1181  unsigned ArgOffset = 0;
1182  const DataLayout &DL = F->getParent()->getDataLayout();
1183  for (auto &FArg : F->args()) {
1184  if (!FArg.getType()->isSized()) {
1185  DEBUG(dbgs() << "Arg is not sized\n");
1186  continue;
1187  }
1188  unsigned Size =
1189  FArg.hasByValAttr()
1190  ? DL.getTypeAllocSize(FArg.getType()->getPointerElementType())
1191  : DL.getTypeAllocSize(FArg.getType());
1192  if (A == &FArg) {
1193  bool Overflow = ArgOffset + Size > kParamTLSSize;
1194  Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset);
1195  if (FArg.hasByValAttr()) {
1196  // ByVal pointer itself has clean shadow. We copy the actual
1197  // argument shadow to the underlying memory.
1198  // Figure out maximal valid memcpy alignment.
1199  unsigned ArgAlign = FArg.getParamAlignment();
1200  if (ArgAlign == 0) {
1201  Type *EltType = A->getType()->getPointerElementType();
1202  ArgAlign = DL.getABITypeAlignment(EltType);
1203  }
1204  if (Overflow) {
1205  // ParamTLS overflow.
1206  EntryIRB.CreateMemSet(
1207  getShadowPtr(V, EntryIRB.getInt8Ty(), EntryIRB),
1208  Constant::getNullValue(EntryIRB.getInt8Ty()), Size, ArgAlign);
1209  } else {
1210  unsigned CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
1211  Value *Cpy = EntryIRB.CreateMemCpy(
1212  getShadowPtr(V, EntryIRB.getInt8Ty(), EntryIRB), Base, Size,
1213  CopyAlign);
1214  DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n");
1215  (void)Cpy;
1216  }
1217  *ShadowPtr = getCleanShadow(V);
1218  } else {
1219  if (Overflow) {
1220  // ParamTLS overflow.
1221  *ShadowPtr = getCleanShadow(V);
1222  } else {
1223  *ShadowPtr =
1224  EntryIRB.CreateAlignedLoad(Base, kShadowTLSAlignment);
1225  }
1226  }
1227  DEBUG(dbgs() << " ARG: " << FArg << " ==> " <<
1228  **ShadowPtr << "\n");
1229  if (MS.TrackOrigins && !Overflow) {
1230  Value *OriginPtr =
1231  getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset);
1232  setOrigin(A, EntryIRB.CreateLoad(OriginPtr));
1233  } else {
1234  setOrigin(A, getCleanOrigin());
1235  }
1236  }
1237  ArgOffset += alignTo(Size, kShadowTLSAlignment);
1238  }
1239  assert(*ShadowPtr && "Could not find shadow for an argument");
1240  return *ShadowPtr;
1241  }
1242  // For everything else the shadow is zero.
1243  return getCleanShadow(V);
1244  }
1245 
1246  /// \brief Get the shadow for i-th argument of the instruction I.
1247  Value *getShadow(Instruction *I, int i) {
1248  return getShadow(I->getOperand(i));
1249  }
1250 
1251  /// \brief Get the origin for a value.
1252  Value *getOrigin(Value *V) {
1253  if (!MS.TrackOrigins) return nullptr;
1254  if (!PropagateShadow) return getCleanOrigin();
1255  if (isa<Constant>(V)) return getCleanOrigin();
1256  assert((isa<Instruction>(V) || isa<Argument>(V)) &&
1257  "Unexpected value type in getOrigin()");
1258  Value *Origin = OriginMap[V];
1259  assert(Origin && "Missing origin");
1260  return Origin;
1261  }
1262 
1263  /// \brief Get the origin for i-th argument of the instruction I.
1264  Value *getOrigin(Instruction *I, int i) {
1265  return getOrigin(I->getOperand(i));
1266  }
1267 
1268  /// \brief Remember the place where a shadow check should be inserted.
1269  ///
1270  /// This location will be later instrumented with a check that will print a
1271  /// UMR warning in runtime if the shadow value is not 0.
1272  void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
1273  assert(Shadow);
1274  if (!InsertChecks) return;
1275 #ifndef NDEBUG
1276  Type *ShadowTy = Shadow->getType();
1277  assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy)) &&
1278  "Can only insert checks for integer and vector shadow types");
1279 #endif
1280  InstrumentationList.push_back(
1281  ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
1282  }
1283 
1284  /// \brief Remember the place where a shadow check should be inserted.
1285  ///
1286  /// This location will be later instrumented with a check that will print a
1287  /// UMR warning in runtime if the value is not fully defined.
1288  void insertShadowCheck(Value *Val, Instruction *OrigIns) {
1289  assert(Val);
1290  Value *Shadow, *Origin;
1291  if (ClCheckConstantShadow) {
1292  Shadow = getShadow(Val);
1293  if (!Shadow) return;
1294  Origin = getOrigin(Val);
1295  } else {
1296  Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
1297  if (!Shadow) return;
1298  Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
1299  }
1300  insertShadowCheck(Shadow, Origin, OrigIns);
1301  }
1302 
1303  AtomicOrdering addReleaseOrdering(AtomicOrdering a) {
1304  switch (a) {
1310  return AtomicOrdering::Release;
1316  }
1317  llvm_unreachable("Unknown ordering");
1318  }
1319 
1320  AtomicOrdering addAcquireOrdering(AtomicOrdering a) {
1321  switch (a) {
1327  return AtomicOrdering::Acquire;
1333  }
1334  llvm_unreachable("Unknown ordering");
1335  }
1336 
1337  // ------------------- Visitors.
1338 
1339  /// \brief Instrument LoadInst
1340  ///
1341  /// Loads the corresponding shadow and (optionally) origin.
1342  /// Optionally, checks that the load address is fully defined.
1343  void visitLoadInst(LoadInst &I) {
1344  assert(I.getType()->isSized() && "Load type must have size");
1345  IRBuilder<> IRB(I.getNextNode());
1346  Type *ShadowTy = getShadowTy(&I);
1347  Value *Addr = I.getPointerOperand();
1348  if (PropagateShadow && !I.getMetadata("nosanitize")) {
1349  Value *ShadowPtr = getShadowPtr(Addr, ShadowTy, IRB);
1350  setShadow(&I,
1351  IRB.CreateAlignedLoad(ShadowPtr, I.getAlignment(), "_msld"));
1352  } else {
1353  setShadow(&I, getCleanShadow(&I));
1354  }
1355 
1357  insertShadowCheck(I.getPointerOperand(), &I);
1358 
1359  if (I.isAtomic())
1360  I.setOrdering(addAcquireOrdering(I.getOrdering()));
1361 
1362  if (MS.TrackOrigins) {
1363  if (PropagateShadow) {
1364  unsigned Alignment = I.getAlignment();
1365  unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1366  setOrigin(&I, IRB.CreateAlignedLoad(getOriginPtr(Addr, IRB, Alignment),
1367  OriginAlignment));
1368  } else {
1369  setOrigin(&I, getCleanOrigin());
1370  }
1371  }
1372  }
1373 
1374  /// \brief Instrument StoreInst
1375  ///
1376  /// Stores the corresponding shadow and (optionally) origin.
1377  /// Optionally, checks that the store address is fully defined.
1378  void visitStoreInst(StoreInst &I) {
1379  StoreList.push_back(&I);
1380  }
1381 
1382  void handleCASOrRMW(Instruction &I) {
1383  assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
1384 
1385  IRBuilder<> IRB(&I);
1386  Value *Addr = I.getOperand(0);
1387  Value *ShadowPtr = getShadowPtr(Addr, I.getType(), IRB);
1388 
1390  insertShadowCheck(Addr, &I);
1391 
1392  // Only test the conditional argument of cmpxchg instruction.
1393  // The other argument can potentially be uninitialized, but we can not
1394  // detect this situation reliably without possible false positives.
1395  if (isa<AtomicCmpXchgInst>(I))
1396  insertShadowCheck(I.getOperand(1), &I);
1397 
1398  IRB.CreateStore(getCleanShadow(&I), ShadowPtr);
1399 
1400  setShadow(&I, getCleanShadow(&I));
1401  setOrigin(&I, getCleanOrigin());
1402  }
1403 
1404  void visitAtomicRMWInst(AtomicRMWInst &I) {
1405  handleCASOrRMW(I);
1406  I.setOrdering(addReleaseOrdering(I.getOrdering()));
1407  }
1408 
1409  void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
1410  handleCASOrRMW(I);
1411  I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
1412  }
1413 
1414  // Vector manipulation.
1415  void visitExtractElementInst(ExtractElementInst &I) {
1416  insertShadowCheck(I.getOperand(1), &I);
1417  IRBuilder<> IRB(&I);
1418  setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
1419  "_msprop"));
1420  setOrigin(&I, getOrigin(&I, 0));
1421  }
1422 
1423  void visitInsertElementInst(InsertElementInst &I) {
1424  insertShadowCheck(I.getOperand(2), &I);
1425  IRBuilder<> IRB(&I);
1426  setShadow(&I, IRB.CreateInsertElement(getShadow(&I, 0), getShadow(&I, 1),
1427  I.getOperand(2), "_msprop"));
1428  setOriginForNaryOp(I);
1429  }
1430 
1431  void visitShuffleVectorInst(ShuffleVectorInst &I) {
1432  insertShadowCheck(I.getOperand(2), &I);
1433  IRBuilder<> IRB(&I);
1434  setShadow(&I, IRB.CreateShuffleVector(getShadow(&I, 0), getShadow(&I, 1),
1435  I.getOperand(2), "_msprop"));
1436  setOriginForNaryOp(I);
1437  }
1438 
1439  // Casts.
1440  void visitSExtInst(SExtInst &I) {
1441  IRBuilder<> IRB(&I);
1442  setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
1443  setOrigin(&I, getOrigin(&I, 0));
1444  }
1445 
1446  void visitZExtInst(ZExtInst &I) {
1447  IRBuilder<> IRB(&I);
1448  setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
1449  setOrigin(&I, getOrigin(&I, 0));
1450  }
1451 
1452  void visitTruncInst(TruncInst &I) {
1453  IRBuilder<> IRB(&I);
1454  setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
1455  setOrigin(&I, getOrigin(&I, 0));
1456  }
1457 
1458  void visitBitCastInst(BitCastInst &I) {
1459  // Special case: if this is the bitcast (there is exactly 1 allowed) between
1460  // a musttail call and a ret, don't instrument. New instructions are not
1461  // allowed after a musttail call.
1462  if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
1463  if (CI->isMustTailCall())
1464  return;
1465  IRBuilder<> IRB(&I);
1466  setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
1467  setOrigin(&I, getOrigin(&I, 0));
1468  }
1469 
1470  void visitPtrToIntInst(PtrToIntInst &I) {
1471  IRBuilder<> IRB(&I);
1472  setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
1473  "_msprop_ptrtoint"));
1474  setOrigin(&I, getOrigin(&I, 0));
1475  }
1476 
1477  void visitIntToPtrInst(IntToPtrInst &I) {
1478  IRBuilder<> IRB(&I);
1479  setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
1480  "_msprop_inttoptr"));
1481  setOrigin(&I, getOrigin(&I, 0));
1482  }
1483 
1484  void visitFPToSIInst(CastInst& I) { handleShadowOr(I); }
1485  void visitFPToUIInst(CastInst& I) { handleShadowOr(I); }
1486  void visitSIToFPInst(CastInst& I) { handleShadowOr(I); }
1487  void visitUIToFPInst(CastInst& I) { handleShadowOr(I); }
1488  void visitFPExtInst(CastInst& I) { handleShadowOr(I); }
1489  void visitFPTruncInst(CastInst& I) { handleShadowOr(I); }
1490 
1491  /// \brief Propagate shadow for bitwise AND.
1492  ///
1493  /// This code is exact, i.e. if, for example, a bit in the left argument
1494  /// is defined and 0, then neither the value not definedness of the
1495  /// corresponding bit in B don't affect the resulting shadow.
1496  void visitAnd(BinaryOperator &I) {
1497  IRBuilder<> IRB(&I);
1498  // "And" of 0 and a poisoned value results in unpoisoned value.
1499  // 1&1 => 1; 0&1 => 0; p&1 => p;
1500  // 1&0 => 0; 0&0 => 0; p&0 => 0;
1501  // 1&p => p; 0&p => 0; p&p => p;
1502  // S = (S1 & S2) | (V1 & S2) | (S1 & V2)
1503  Value *S1 = getShadow(&I, 0);
1504  Value *S2 = getShadow(&I, 1);
1505  Value *V1 = I.getOperand(0);
1506  Value *V2 = I.getOperand(1);
1507  if (V1->getType() != S1->getType()) {
1508  V1 = IRB.CreateIntCast(V1, S1->getType(), false);
1509  V2 = IRB.CreateIntCast(V2, S2->getType(), false);
1510  }
1511  Value *S1S2 = IRB.CreateAnd(S1, S2);
1512  Value *V1S2 = IRB.CreateAnd(V1, S2);
1513  Value *S1V2 = IRB.CreateAnd(S1, V2);
1514  setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2)));
1515  setOriginForNaryOp(I);
1516  }
1517 
1518  void visitOr(BinaryOperator &I) {
1519  IRBuilder<> IRB(&I);
1520  // "Or" of 1 and a poisoned value results in unpoisoned value.
1521  // 1|1 => 1; 0|1 => 1; p|1 => 1;
1522  // 1|0 => 1; 0|0 => 0; p|0 => p;
1523  // 1|p => 1; 0|p => p; p|p => p;
1524  // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
1525  Value *S1 = getShadow(&I, 0);
1526  Value *S2 = getShadow(&I, 1);
1527  Value *V1 = IRB.CreateNot(I.getOperand(0));
1528  Value *V2 = IRB.CreateNot(I.getOperand(1));
1529  if (V1->getType() != S1->getType()) {
1530  V1 = IRB.CreateIntCast(V1, S1->getType(), false);
1531  V2 = IRB.CreateIntCast(V2, S2->getType(), false);
1532  }
1533  Value *S1S2 = IRB.CreateAnd(S1, S2);
1534  Value *V1S2 = IRB.CreateAnd(V1, S2);
1535  Value *S1V2 = IRB.CreateAnd(S1, V2);
1536  setShadow(&I, IRB.CreateOr(S1S2, IRB.CreateOr(V1S2, S1V2)));
1537  setOriginForNaryOp(I);
1538  }
1539 
1540  /// \brief Default propagation of shadow and/or origin.
1541  ///
1542  /// This class implements the general case of shadow propagation, used in all
1543  /// cases where we don't know and/or don't care about what the operation
1544  /// actually does. It converts all input shadow values to a common type
1545  /// (extending or truncating as necessary), and bitwise OR's them.
1546  ///
1547  /// This is much cheaper than inserting checks (i.e. requiring inputs to be
1548  /// fully initialized), and less prone to false positives.
1549  ///
1550  /// This class also implements the general case of origin propagation. For a
1551  /// Nary operation, result origin is set to the origin of an argument that is
1552  /// not entirely initialized. If there is more than one such arguments, the
1553  /// rightmost of them is picked. It does not matter which one is picked if all
1554  /// arguments are initialized.
1555  template <bool CombineShadow>
1556  class Combiner {
1557  Value *Shadow = nullptr;
1558  Value *Origin = nullptr;
1559  IRBuilder<> &IRB;
1560  MemorySanitizerVisitor *MSV;
1561 
1562  public:
1563  Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
1564  : IRB(IRB), MSV(MSV) {}
1565 
1566  /// \brief Add a pair of shadow and origin values to the mix.
1567  Combiner &Add(Value *OpShadow, Value *OpOrigin) {
1568  if (CombineShadow) {
1569  assert(OpShadow);
1570  if (!Shadow)
1571  Shadow = OpShadow;
1572  else {
1573  OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
1574  Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
1575  }
1576  }
1577 
1578  if (MSV->MS.TrackOrigins) {
1579  assert(OpOrigin);
1580  if (!Origin) {
1581  Origin = OpOrigin;
1582  } else {
1583  Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
1584  // No point in adding something that might result in 0 origin value.
1585  if (!ConstOrigin || !ConstOrigin->isNullValue()) {
1586  Value *FlatShadow = MSV->convertToShadowTyNoVec(OpShadow, IRB);
1587  Value *Cond =
1588  IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow));
1589  Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
1590  }
1591  }
1592  }
1593  return *this;
1594  }
1595 
1596  /// \brief Add an application value to the mix.
1597  Combiner &Add(Value *V) {
1598  Value *OpShadow = MSV->getShadow(V);
1599  Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr;
1600  return Add(OpShadow, OpOrigin);
1601  }
1602 
1603  /// \brief Set the current combined values as the given instruction's shadow
1604  /// and origin.
1605  void Done(Instruction *I) {
1606  if (CombineShadow) {
1607  assert(Shadow);
1608  Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
1609  MSV->setShadow(I, Shadow);
1610  }
1611  if (MSV->MS.TrackOrigins) {
1612  assert(Origin);
1613  MSV->setOrigin(I, Origin);
1614  }
1615  }
1616  };
1617 
1618  using ShadowAndOriginCombiner = Combiner<true>;
1619  using OriginCombiner = Combiner<false>;
1620 
1621  /// \brief Propagate origin for arbitrary operation.
1622  void setOriginForNaryOp(Instruction &I) {
1623  if (!MS.TrackOrigins) return;
1624  IRBuilder<> IRB(&I);
1625  OriginCombiner OC(this, IRB);
1626  for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI)
1627  OC.Add(OI->get());
1628  OC.Done(&I);
1629  }
1630 
1631  size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
1632  assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
1633  "Vector of pointers is not a valid shadow type");
1634  return Ty->isVectorTy() ?
1636  Ty->getPrimitiveSizeInBits();
1637  }
1638 
1639  /// \brief Cast between two shadow types, extending or truncating as
1640  /// necessary.
1641  Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
1642  bool Signed = false) {
1643  Type *srcTy = V->getType();
1644  size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
1645  size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
1646  if (srcSizeInBits > 1 && dstSizeInBits == 1)
1647  return IRB.CreateICmpNE(V, getCleanShadow(V));
1648 
1649  if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
1650  return IRB.CreateIntCast(V, dstTy, Signed);
1651  if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
1652  dstTy->getVectorNumElements() == srcTy->getVectorNumElements())
1653  return IRB.CreateIntCast(V, dstTy, Signed);
1654  Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
1655  Value *V2 =
1656  IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
1657  return IRB.CreateBitCast(V2, dstTy);
1658  // TODO: handle struct types.
1659  }
1660 
1661  /// \brief Cast an application value to the type of its own shadow.
1662  Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
1663  Type *ShadowTy = getShadowTy(V);
1664  if (V->getType() == ShadowTy)
1665  return V;
1666  if (V->getType()->isPtrOrPtrVectorTy())
1667  return IRB.CreatePtrToInt(V, ShadowTy);
1668  else
1669  return IRB.CreateBitCast(V, ShadowTy);
1670  }
1671 
1672  /// \brief Propagate shadow for arbitrary operation.
1673  void handleShadowOr(Instruction &I) {
1674  IRBuilder<> IRB(&I);
1675  ShadowAndOriginCombiner SC(this, IRB);
1676  for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); ++OI)
1677  SC.Add(OI->get());
1678  SC.Done(&I);
1679  }
1680 
1681  // \brief Handle multiplication by constant.
1682  //
1683  // Handle a special case of multiplication by constant that may have one or
1684  // more zeros in the lower bits. This makes corresponding number of lower bits
1685  // of the result zero as well. We model it by shifting the other operand
1686  // shadow left by the required number of bits. Effectively, we transform
1687  // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
1688  // We use multiplication by 2**N instead of shift to cover the case of
1689  // multiplication by 0, which may occur in some elements of a vector operand.
1690  void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
1691  Value *OtherArg) {
1692  Constant *ShadowMul;
1693  Type *Ty = ConstArg->getType();
1694  if (Ty->isVectorTy()) {
1695  unsigned NumElements = Ty->getVectorNumElements();
1696  Type *EltTy = Ty->getSequentialElementType();
1697  SmallVector<Constant *, 16> Elements;
1698  for (unsigned Idx = 0; Idx < NumElements; ++Idx) {
1699  if (ConstantInt *Elt =
1700  dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
1701  const APInt &V = Elt->getValue();
1702  APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
1703  Elements.push_back(ConstantInt::get(EltTy, V2));
1704  } else {
1705  Elements.push_back(ConstantInt::get(EltTy, 1));
1706  }
1707  }
1708  ShadowMul = ConstantVector::get(Elements);
1709  } else {
1710  if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
1711  const APInt &V = Elt->getValue();
1712  APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
1713  ShadowMul = ConstantInt::get(Ty, V2);
1714  } else {
1715  ShadowMul = ConstantInt::get(Ty, 1);
1716  }
1717  }
1718 
1719  IRBuilder<> IRB(&I);
1720  setShadow(&I,
1721  IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
1722  setOrigin(&I, getOrigin(OtherArg));
1723  }
1724 
1725  void visitMul(BinaryOperator &I) {
1726  Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
1727  Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
1728  if (constOp0 && !constOp1)
1729  handleMulByConstant(I, constOp0, I.getOperand(1));
1730  else if (constOp1 && !constOp0)
1731  handleMulByConstant(I, constOp1, I.getOperand(0));
1732  else
1733  handleShadowOr(I);
1734  }
1735 
1736  void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
1737  void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
1738  void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
1739  void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
1740  void visitSub(BinaryOperator &I) { handleShadowOr(I); }
1741  void visitXor(BinaryOperator &I) { handleShadowOr(I); }
1742 
1743  void handleDiv(Instruction &I) {
1744  IRBuilder<> IRB(&I);
1745  // Strict on the second argument.
1746  insertShadowCheck(I.getOperand(1), &I);
1747  setShadow(&I, getShadow(&I, 0));
1748  setOrigin(&I, getOrigin(&I, 0));
1749  }
1750 
1751  void visitUDiv(BinaryOperator &I) { handleDiv(I); }
1752  void visitSDiv(BinaryOperator &I) { handleDiv(I); }
1753  void visitFDiv(BinaryOperator &I) { handleDiv(I); }
1754  void visitURem(BinaryOperator &I) { handleDiv(I); }
1755  void visitSRem(BinaryOperator &I) { handleDiv(I); }
1756  void visitFRem(BinaryOperator &I) { handleDiv(I); }
1757 
1758  /// \brief Instrument == and != comparisons.
1759  ///
1760  /// Sometimes the comparison result is known even if some of the bits of the
1761  /// arguments are not.
1762  void handleEqualityComparison(ICmpInst &I) {
1763  IRBuilder<> IRB(&I);
1764  Value *A = I.getOperand(0);
1765  Value *B = I.getOperand(1);
1766  Value *Sa = getShadow(A);
1767  Value *Sb = getShadow(B);
1768 
1769  // Get rid of pointers and vectors of pointers.
1770  // For ints (and vectors of ints), types of A and Sa match,
1771  // and this is a no-op.
1772  A = IRB.CreatePointerCast(A, Sa->getType());
1773  B = IRB.CreatePointerCast(B, Sb->getType());
1774 
1775  // A == B <==> (C = A^B) == 0
1776  // A != B <==> (C = A^B) != 0
1777  // Sc = Sa | Sb
1778  Value *C = IRB.CreateXor(A, B);
1779  Value *Sc = IRB.CreateOr(Sa, Sb);
1780  // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
1781  // Result is defined if one of the following is true
1782  // * there is a defined 1 bit in C
1783  // * C is fully defined
1784  // Si = !(C & ~Sc) && Sc
1785  Value *Zero = Constant::getNullValue(Sc->getType());
1786  Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
1787  Value *Si =
1788  IRB.CreateAnd(IRB.CreateICmpNE(Sc, Zero),
1789  IRB.CreateICmpEQ(
1790  IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero));
1791  Si->setName("_msprop_icmp");
1792  setShadow(&I, Si);
1793  setOriginForNaryOp(I);
1794  }
1795 
1796  /// \brief Build the lowest possible value of V, taking into account V's
1797  /// uninitialized bits.
1798  Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
1799  bool isSigned) {
1800  if (isSigned) {
1801  // Split shadow into sign bit and other bits.
1802  Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
1803  Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
1804  // Maximise the undefined shadow bit, minimize other undefined bits.
1805  return
1806  IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)), SaSignBit);
1807  } else {
1808  // Minimize undefined bits.
1809  return IRB.CreateAnd(A, IRB.CreateNot(Sa));
1810  }
1811  }
1812 
1813  /// \brief Build the highest possible value of V, taking into account V's
1814  /// uninitialized bits.
1815  Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
1816  bool isSigned) {
1817  if (isSigned) {
1818  // Split shadow into sign bit and other bits.
1819  Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
1820  Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
1821  // Minimise the undefined shadow bit, maximise other undefined bits.
1822  return
1823  IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)), SaOtherBits);
1824  } else {
1825  // Maximize undefined bits.
1826  return IRB.CreateOr(A, Sa);
1827  }
1828  }
1829 
1830  /// \brief Instrument relational comparisons.
1831  ///
1832  /// This function does exact shadow propagation for all relational
1833  /// comparisons of integers, pointers and vectors of those.
1834  /// FIXME: output seems suboptimal when one of the operands is a constant
1835  void handleRelationalComparisonExact(ICmpInst &I) {
1836  IRBuilder<> IRB(&I);
1837  Value *A = I.getOperand(0);
1838  Value *B = I.getOperand(1);
1839  Value *Sa = getShadow(A);
1840  Value *Sb = getShadow(B);
1841 
1842  // Get rid of pointers and vectors of pointers.
1843  // For ints (and vectors of ints), types of A and Sa match,
1844  // and this is a no-op.
1845  A = IRB.CreatePointerCast(A, Sa->getType());
1846  B = IRB.CreatePointerCast(B, Sb->getType());
1847 
1848  // Let [a0, a1] be the interval of possible values of A, taking into account
1849  // its undefined bits. Let [b0, b1] be the interval of possible values of B.
1850  // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
1851  bool IsSigned = I.isSigned();
1852  Value *S1 = IRB.CreateICmp(I.getPredicate(),
1853  getLowestPossibleValue(IRB, A, Sa, IsSigned),
1854  getHighestPossibleValue(IRB, B, Sb, IsSigned));
1855  Value *S2 = IRB.CreateICmp(I.getPredicate(),
1856  getHighestPossibleValue(IRB, A, Sa, IsSigned),
1857  getLowestPossibleValue(IRB, B, Sb, IsSigned));
1858  Value *Si = IRB.CreateXor(S1, S2);
1859  setShadow(&I, Si);
1860  setOriginForNaryOp(I);
1861  }
1862 
1863  /// \brief Instrument signed relational comparisons.
1864  ///
1865  /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
1866  /// bit of the shadow. Everything else is delegated to handleShadowOr().
1867  void handleSignedRelationalComparison(ICmpInst &I) {
1868  Constant *constOp;
1869  Value *op = nullptr;
1870  CmpInst::Predicate pre;
1871  if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
1872  op = I.getOperand(0);
1873  pre = I.getPredicate();
1874  } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) {
1875  op = I.getOperand(1);
1876  pre = I.getSwappedPredicate();
1877  } else {
1878  handleShadowOr(I);
1879  return;
1880  }
1881 
1882  if ((constOp->isNullValue() &&
1883  (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) ||
1884  (constOp->isAllOnesValue() &&
1885  (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) {
1886  IRBuilder<> IRB(&I);
1887  Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
1888  "_msprop_icmp_s");
1889  setShadow(&I, Shadow);
1890  setOrigin(&I, getOrigin(op));
1891  } else {
1892  handleShadowOr(I);
1893  }
1894  }
1895 
1896  void visitICmpInst(ICmpInst &I) {
1897  if (!ClHandleICmp) {
1898  handleShadowOr(I);
1899  return;
1900  }
1901  if (I.isEquality()) {
1902  handleEqualityComparison(I);
1903  return;
1904  }
1905 
1906  assert(I.isRelational());
1907  if (ClHandleICmpExact) {
1908  handleRelationalComparisonExact(I);
1909  return;
1910  }
1911  if (I.isSigned()) {
1912  handleSignedRelationalComparison(I);
1913  return;
1914  }
1915 
1916  assert(I.isUnsigned());
1917  if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
1918  handleRelationalComparisonExact(I);
1919  return;
1920  }
1921 
1922  handleShadowOr(I);
1923  }
1924 
1925  void visitFCmpInst(FCmpInst &I) {
1926  handleShadowOr(I);
1927  }
1928 
1929  void handleShift(BinaryOperator &I) {
1930  IRBuilder<> IRB(&I);
1931  // If any of the S2 bits are poisoned, the whole thing is poisoned.
1932  // Otherwise perform the same shift on S1.
1933  Value *S1 = getShadow(&I, 0);
1934  Value *S2 = getShadow(&I, 1);
1935  Value *S2Conv = IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)),
1936  S2->getType());
1937  Value *V2 = I.getOperand(1);
1938  Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
1939  setShadow(&I, IRB.CreateOr(Shift, S2Conv));
1940  setOriginForNaryOp(I);
1941  }
1942 
1943  void visitShl(BinaryOperator &I) { handleShift(I); }
1944  void visitAShr(BinaryOperator &I) { handleShift(I); }
1945  void visitLShr(BinaryOperator &I) { handleShift(I); }
1946 
1947  /// \brief Instrument llvm.memmove
1948  ///
1949  /// At this point we don't know if llvm.memmove will be inlined or not.
1950  /// If we don't instrument it and it gets inlined,
1951  /// our interceptor will not kick in and we will lose the memmove.
1952  /// If we instrument the call here, but it does not get inlined,
1953  /// we will memove the shadow twice: which is bad in case
1954  /// of overlapping regions. So, we simply lower the intrinsic to a call.
1955  ///
1956  /// Similar situation exists for memcpy and memset.
1957  void visitMemMoveInst(MemMoveInst &I) {
1958  IRBuilder<> IRB(&I);
1959  IRB.CreateCall(
1960  MS.MemmoveFn,
1961  {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
1962  IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
1963  IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
1964  I.eraseFromParent();
1965  }
1966 
1967  // Similar to memmove: avoid copying shadow twice.
1968  // This is somewhat unfortunate as it may slowdown small constant memcpys.
1969  // FIXME: consider doing manual inline for small constant sizes and proper
1970  // alignment.
1971  void visitMemCpyInst(MemCpyInst &I) {
1972  IRBuilder<> IRB(&I);
1973  IRB.CreateCall(
1974  MS.MemcpyFn,
1975  {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
1976  IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
1977  IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
1978  I.eraseFromParent();
1979  }
1980 
1981  // Same as memcpy.
1982  void visitMemSetInst(MemSetInst &I) {
1983  IRBuilder<> IRB(&I);
1984  IRB.CreateCall(
1985  MS.MemsetFn,
1986  {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
1987  IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
1988  IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
1989  I.eraseFromParent();
1990  }
1991 
1992  void visitVAStartInst(VAStartInst &I) {
1993  VAHelper->visitVAStartInst(I);
1994  }
1995 
1996  void visitVACopyInst(VACopyInst &I) {
1997  VAHelper->visitVACopyInst(I);
1998  }
1999 
2000  /// \brief Handle vector store-like intrinsics.
2001  ///
2002  /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2003  /// has 1 pointer argument and 1 vector argument, returns void.
2004  bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2005  IRBuilder<> IRB(&I);
2006  Value* Addr = I.getArgOperand(0);
2007  Value *Shadow = getShadow(&I, 1);
2008  Value *ShadowPtr = getShadowPtr(Addr, Shadow->getType(), IRB);
2009 
2010  // We don't know the pointer alignment (could be unaligned SSE store!).
2011  // Have to assume to worst case.
2012  IRB.CreateAlignedStore(Shadow, ShadowPtr, 1);
2013 
2015  insertShadowCheck(Addr, &I);
2016 
2017  // FIXME: factor out common code from materializeStores
2018  if (MS.TrackOrigins)
2019  IRB.CreateStore(getOrigin(&I, 1), getOriginPtr(Addr, IRB, 1));
2020  return true;
2021  }
2022 
2023  /// \brief Handle vector load-like intrinsics.
2024  ///
2025  /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2026  /// has 1 pointer argument, returns a vector.
2027  bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2028  IRBuilder<> IRB(&I);
2029  Value *Addr = I.getArgOperand(0);
2030 
2031  Type *ShadowTy = getShadowTy(&I);
2032  if (PropagateShadow) {
2033  Value *ShadowPtr = getShadowPtr(Addr, ShadowTy, IRB);
2034  // We don't know the pointer alignment (could be unaligned SSE load!).
2035  // Have to assume to worst case.
2036  setShadow(&I, IRB.CreateAlignedLoad(ShadowPtr, 1, "_msld"));
2037  } else {
2038  setShadow(&I, getCleanShadow(&I));
2039  }
2040 
2042  insertShadowCheck(Addr, &I);
2043 
2044  if (MS.TrackOrigins) {
2045  if (PropagateShadow)
2046  setOrigin(&I, IRB.CreateLoad(getOriginPtr(Addr, IRB, 1)));
2047  else
2048  setOrigin(&I, getCleanOrigin());
2049  }
2050  return true;
2051  }
2052 
2053  /// \brief Handle (SIMD arithmetic)-like intrinsics.
2054  ///
2055  /// Instrument intrinsics with any number of arguments of the same type,
2056  /// equal to the return type. The type should be simple (no aggregates or
2057  /// pointers; vectors are fine).
2058  /// Caller guarantees that this intrinsic does not access memory.
2059  bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2060  Type *RetTy = I.getType();
2061  if (!(RetTy->isIntOrIntVectorTy() ||
2062  RetTy->isFPOrFPVectorTy() ||
2063  RetTy->isX86_MMXTy()))
2064  return false;
2065 
2066  unsigned NumArgOperands = I.getNumArgOperands();
2067 
2068  for (unsigned i = 0; i < NumArgOperands; ++i) {
2069  Type *Ty = I.getArgOperand(i)->getType();
2070  if (Ty != RetTy)
2071  return false;
2072  }
2073 
2074  IRBuilder<> IRB(&I);
2075  ShadowAndOriginCombiner SC(this, IRB);
2076  for (unsigned i = 0; i < NumArgOperands; ++i)
2077  SC.Add(I.getArgOperand(i));
2078  SC.Done(&I);
2079 
2080  return true;
2081  }
2082 
2083  /// \brief Heuristically instrument unknown intrinsics.
2084  ///
2085  /// The main purpose of this code is to do something reasonable with all
2086  /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
2087  /// We recognize several classes of intrinsics by their argument types and
2088  /// ModRefBehaviour and apply special intrumentation when we are reasonably
2089  /// sure that we know what the intrinsic does.
2090  ///
2091  /// We special-case intrinsics where this approach fails. See llvm.bswap
2092  /// handling as an example of that.
2093  bool handleUnknownIntrinsic(IntrinsicInst &I) {
2094  unsigned NumArgOperands = I.getNumArgOperands();
2095  if (NumArgOperands == 0)
2096  return false;
2097 
2098  if (NumArgOperands == 2 &&
2099  I.getArgOperand(0)->getType()->isPointerTy() &&
2100  I.getArgOperand(1)->getType()->isVectorTy() &&
2101  I.getType()->isVoidTy() &&
2102  !I.onlyReadsMemory()) {
2103  // This looks like a vector store.
2104  return handleVectorStoreIntrinsic(I);
2105  }
2106 
2107  if (NumArgOperands == 1 &&
2108  I.getArgOperand(0)->getType()->isPointerTy() &&
2109  I.getType()->isVectorTy() &&
2110  I.onlyReadsMemory()) {
2111  // This looks like a vector load.
2112  return handleVectorLoadIntrinsic(I);
2113  }
2114 
2115  if (I.doesNotAccessMemory())
2116  if (maybeHandleSimpleNomemIntrinsic(I))
2117  return true;
2118 
2119  // FIXME: detect and handle SSE maskstore/maskload
2120  return false;
2121  }
2122 
2123  void handleBswap(IntrinsicInst &I) {
2124  IRBuilder<> IRB(&I);
2125  Value *Op = I.getArgOperand(0);
2126  Type *OpType = Op->getType();
2127  Function *BswapFunc = Intrinsic::getDeclaration(
2128  F.getParent(), Intrinsic::bswap, makeArrayRef(&OpType, 1));
2129  setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
2130  setOrigin(&I, getOrigin(Op));
2131  }
2132 
2133  // \brief Instrument vector convert instrinsic.
2134  //
2135  // This function instruments intrinsics like cvtsi2ss:
2136  // %Out = int_xxx_cvtyyy(%ConvertOp)
2137  // or
2138  // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
2139  // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
2140  // number \p Out elements, and (if has 2 arguments) copies the rest of the
2141  // elements from \p CopyOp.
2142  // In most cases conversion involves floating-point value which may trigger a
2143  // hardware exception when not fully initialized. For this reason we require
2144  // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
2145  // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
2146  // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
2147  // return a fully initialized value.
2148  void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements) {
2149  IRBuilder<> IRB(&I);
2150  Value *CopyOp, *ConvertOp;
2151 
2152  switch (I.getNumArgOperands()) {
2153  case 3:
2154  assert(isa<ConstantInt>(I.getArgOperand(2)) && "Invalid rounding mode");
2156  case 2:
2157  CopyOp = I.getArgOperand(0);
2158  ConvertOp = I.getArgOperand(1);
2159  break;
2160  case 1:
2161  ConvertOp = I.getArgOperand(0);
2162  CopyOp = nullptr;
2163  break;
2164  default:
2165  llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
2166  }
2167 
2168  // The first *NumUsedElements* elements of ConvertOp are converted to the
2169  // same number of output elements. The rest of the output is copied from
2170  // CopyOp, or (if not available) filled with zeroes.
2171  // Combine shadow for elements of ConvertOp that are used in this operation,
2172  // and insert a check.
2173  // FIXME: consider propagating shadow of ConvertOp, at least in the case of
2174  // int->any conversion.
2175  Value *ConvertShadow = getShadow(ConvertOp);
2176  Value *AggShadow = nullptr;
2177  if (ConvertOp->getType()->isVectorTy()) {
2178  AggShadow = IRB.CreateExtractElement(
2179  ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
2180  for (int i = 1; i < NumUsedElements; ++i) {
2181  Value *MoreShadow = IRB.CreateExtractElement(
2182  ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
2183  AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
2184  }
2185  } else {
2186  AggShadow = ConvertShadow;
2187  }
2188  assert(AggShadow->getType()->isIntegerTy());
2189  insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
2190 
2191  // Build result shadow by zero-filling parts of CopyOp shadow that come from
2192  // ConvertOp.
2193  if (CopyOp) {
2194  assert(CopyOp->getType() == I.getType());
2195  assert(CopyOp->getType()->isVectorTy());
2196  Value *ResultShadow = getShadow(CopyOp);
2197  Type *EltTy = ResultShadow->getType()->getVectorElementType();
2198  for (int i = 0; i < NumUsedElements; ++i) {
2199  ResultShadow = IRB.CreateInsertElement(
2200  ResultShadow, ConstantInt::getNullValue(EltTy),
2201  ConstantInt::get(IRB.getInt32Ty(), i));
2202  }
2203  setShadow(&I, ResultShadow);
2204  setOrigin(&I, getOrigin(CopyOp));
2205  } else {
2206  setShadow(&I, getCleanShadow(&I));
2207  setOrigin(&I, getCleanOrigin());
2208  }
2209  }
2210 
2211  // Given a scalar or vector, extract lower 64 bits (or less), and return all
2212  // zeroes if it is zero, and all ones otherwise.
2213  Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2214  if (S->getType()->isVectorTy())
2215  S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
2216  assert(S->getType()->getPrimitiveSizeInBits() <= 64);
2217  Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2218  return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2219  }
2220 
2221  // Given a vector, extract its first element, and return all
2222  // zeroes if it is zero, and all ones otherwise.
2223  Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2224  Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
2225  Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
2226  return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2227  }
2228 
2229  Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
2230  Type *T = S->getType();
2231  assert(T->isVectorTy());
2232  Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2233  return IRB.CreateSExt(S2, T);
2234  }
2235 
2236  // \brief Instrument vector shift instrinsic.
2237  //
2238  // This function instruments intrinsics like int_x86_avx2_psll_w.
2239  // Intrinsic shifts %In by %ShiftSize bits.
2240  // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
2241  // size, and the rest is ignored. Behavior is defined even if shift size is
2242  // greater than register (or field) width.
2243  void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
2244  assert(I.getNumArgOperands() == 2);
2245  IRBuilder<> IRB(&I);
2246  // If any of the S2 bits are poisoned, the whole thing is poisoned.
2247  // Otherwise perform the same shift on S1.
2248  Value *S1 = getShadow(&I, 0);
2249  Value *S2 = getShadow(&I, 1);
2250  Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2)
2251  : Lower64ShadowExtend(IRB, S2, getShadowTy(&I));
2252  Value *V1 = I.getOperand(0);
2253  Value *V2 = I.getOperand(1);
2254  Value *Shift = IRB.CreateCall(I.getCalledValue(),
2255  {IRB.CreateBitCast(S1, V1->getType()), V2});
2256  Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
2257  setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2258  setOriginForNaryOp(I);
2259  }
2260 
2261  // \brief Get an X86_MMX-sized vector type.
2262  Type *getMMXVectorTy(unsigned EltSizeInBits) {
2263  const unsigned X86_MMXSizeInBits = 64;
2264  return VectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
2265  X86_MMXSizeInBits / EltSizeInBits);
2266  }
2267 
2268  // \brief Returns a signed counterpart for an (un)signed-saturate-and-pack
2269  // intrinsic.
2270  Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
2271  switch (id) {
2272  case Intrinsic::x86_sse2_packsswb_128:
2273  case Intrinsic::x86_sse2_packuswb_128:
2274  return Intrinsic::x86_sse2_packsswb_128;
2275 
2276  case Intrinsic::x86_sse2_packssdw_128:
2277  case Intrinsic::x86_sse41_packusdw:
2278  return Intrinsic::x86_sse2_packssdw_128;
2279 
2280  case Intrinsic::x86_avx2_packsswb:
2281  case Intrinsic::x86_avx2_packuswb:
2282  return Intrinsic::x86_avx2_packsswb;
2283 
2284  case Intrinsic::x86_avx2_packssdw:
2285  case Intrinsic::x86_avx2_packusdw:
2286  return Intrinsic::x86_avx2_packssdw;
2287 
2288  case Intrinsic::x86_mmx_packsswb:
2289  case Intrinsic::x86_mmx_packuswb:
2290  return Intrinsic::x86_mmx_packsswb;
2291 
2292  case Intrinsic::x86_mmx_packssdw:
2293  return Intrinsic::x86_mmx_packssdw;
2294  default:
2295  llvm_unreachable("unexpected intrinsic id");
2296  }
2297  }
2298 
2299  // \brief Instrument vector pack instrinsic.
2300  //
2301  // This function instruments intrinsics like x86_mmx_packsswb, that
2302  // packs elements of 2 input vectors into half as many bits with saturation.
2303  // Shadow is propagated with the signed variant of the same intrinsic applied
2304  // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
2305  // EltSizeInBits is used only for x86mmx arguments.
2306  void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
2307  assert(I.getNumArgOperands() == 2);
2308  bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2309  IRBuilder<> IRB(&I);
2310  Value *S1 = getShadow(&I, 0);
2311  Value *S2 = getShadow(&I, 1);
2312  assert(isX86_MMX || S1->getType()->isVectorTy());
2313 
2314  // SExt and ICmpNE below must apply to individual elements of input vectors.
2315  // In case of x86mmx arguments, cast them to appropriate vector types and
2316  // back.
2317  Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType();
2318  if (isX86_MMX) {
2319  S1 = IRB.CreateBitCast(S1, T);
2320  S2 = IRB.CreateBitCast(S2, T);
2321  }
2322  Value *S1_ext = IRB.CreateSExt(
2323  IRB.CreateICmpNE(S1, Constant::getNullValue(T)), T);
2324  Value *S2_ext = IRB.CreateSExt(
2325  IRB.CreateICmpNE(S2, Constant::getNullValue(T)), T);
2326  if (isX86_MMX) {
2327  Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
2328  S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
2329  S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
2330  }
2331 
2332  Function *ShadowFn = Intrinsic::getDeclaration(
2333  F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
2334 
2335  Value *S =
2336  IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
2337  if (isX86_MMX) S = IRB.CreateBitCast(S, getShadowTy(&I));
2338  setShadow(&I, S);
2339  setOriginForNaryOp(I);
2340  }
2341 
2342  // \brief Instrument sum-of-absolute-differencies intrinsic.
2343  void handleVectorSadIntrinsic(IntrinsicInst &I) {
2344  const unsigned SignificantBitsPerResultElement = 16;
2345  bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2346  Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType();
2347  unsigned ZeroBitsPerResultElement =
2348  ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
2349 
2350  IRBuilder<> IRB(&I);
2351  Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2352  S = IRB.CreateBitCast(S, ResTy);
2353  S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2354  ResTy);
2355  S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
2356  S = IRB.CreateBitCast(S, getShadowTy(&I));
2357  setShadow(&I, S);
2358  setOriginForNaryOp(I);
2359  }
2360 
2361  // \brief Instrument multiply-add intrinsic.
2362  void handleVectorPmaddIntrinsic(IntrinsicInst &I,
2363  unsigned EltSizeInBits = 0) {
2364  bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2365  Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType();
2366  IRBuilder<> IRB(&I);
2367  Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2368  S = IRB.CreateBitCast(S, ResTy);
2369  S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2370  ResTy);
2371  S = IRB.CreateBitCast(S, getShadowTy(&I));
2372  setShadow(&I, S);
2373  setOriginForNaryOp(I);
2374  }
2375 
2376  // \brief Instrument compare-packed intrinsic.
2377  // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
2378  // all-ones shadow.
2379  void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
2380  IRBuilder<> IRB(&I);
2381  Type *ResTy = getShadowTy(&I);
2382  Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2383  Value *S = IRB.CreateSExt(
2384  IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
2385  setShadow(&I, S);
2386  setOriginForNaryOp(I);
2387  }
2388 
2389  // \brief Instrument compare-scalar intrinsic.
2390  // This handles both cmp* intrinsics which return the result in the first
2391  // element of a vector, and comi* which return the result as i32.
2392  void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
2393  IRBuilder<> IRB(&I);
2394  Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2395  Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
2396  setShadow(&I, S);
2397  setOriginForNaryOp(I);
2398  }
2399 
2400  void handleStmxcsr(IntrinsicInst &I) {
2401  IRBuilder<> IRB(&I);
2402  Value* Addr = I.getArgOperand(0);
2403  Type *Ty = IRB.getInt32Ty();
2404  Value *ShadowPtr = getShadowPtr(Addr, Ty, IRB);
2405 
2406  IRB.CreateStore(getCleanShadow(Ty),
2407  IRB.CreatePointerCast(ShadowPtr, Ty->getPointerTo()));
2408 
2410  insertShadowCheck(Addr, &I);
2411  }
2412 
2413  void handleLdmxcsr(IntrinsicInst &I) {
2414  if (!InsertChecks) return;
2415 
2416  IRBuilder<> IRB(&I);
2417  Value *Addr = I.getArgOperand(0);
2418  Type *Ty = IRB.getInt32Ty();
2419  unsigned Alignment = 1;
2420 
2422  insertShadowCheck(Addr, &I);
2423 
2424  Value *Shadow = IRB.CreateAlignedLoad(getShadowPtr(Addr, Ty, IRB),
2425  Alignment, "_ldmxcsr");
2426  Value *Origin = MS.TrackOrigins
2427  ? IRB.CreateLoad(getOriginPtr(Addr, IRB, Alignment))
2428  : getCleanOrigin();
2429  insertShadowCheck(Shadow, Origin, &I);
2430  }
2431 
2432  void visitIntrinsicInst(IntrinsicInst &I) {
2433  switch (I.getIntrinsicID()) {
2434  case Intrinsic::bswap:
2435  handleBswap(I);
2436  break;
2437  case Intrinsic::x86_sse_stmxcsr:
2438  handleStmxcsr(I);
2439  break;
2440  case Intrinsic::x86_sse_ldmxcsr:
2441  handleLdmxcsr(I);
2442  break;
2443  case Intrinsic::x86_avx512_vcvtsd2usi64:
2444  case Intrinsic::x86_avx512_vcvtsd2usi32:
2445  case Intrinsic::x86_avx512_vcvtss2usi64:
2446  case Intrinsic::x86_avx512_vcvtss2usi32:
2447  case Intrinsic::x86_avx512_cvttss2usi64:
2448  case Intrinsic::x86_avx512_cvttss2usi:
2449  case Intrinsic::x86_avx512_cvttsd2usi64:
2450  case Intrinsic::x86_avx512_cvttsd2usi:
2451  case Intrinsic::x86_avx512_cvtusi2sd:
2452  case Intrinsic::x86_avx512_cvtusi2ss:
2453  case Intrinsic::x86_avx512_cvtusi642sd:
2454  case Intrinsic::x86_avx512_cvtusi642ss:
2455  case Intrinsic::x86_sse2_cvtsd2si64:
2456  case Intrinsic::x86_sse2_cvtsd2si:
2457  case Intrinsic::x86_sse2_cvtsd2ss:
2458  case Intrinsic::x86_sse2_cvtsi2sd:
2459  case Intrinsic::x86_sse2_cvtsi642sd:
2460  case Intrinsic::x86_sse2_cvtss2sd:
2461  case Intrinsic::x86_sse2_cvttsd2si64:
2462  case Intrinsic::x86_sse2_cvttsd2si:
2463  case Intrinsic::x86_sse_cvtsi2ss:
2464  case Intrinsic::x86_sse_cvtsi642ss:
2465  case Intrinsic::x86_sse_cvtss2si64:
2466  case Intrinsic::x86_sse_cvtss2si:
2467  case Intrinsic::x86_sse_cvttss2si64:
2468  case Intrinsic::x86_sse_cvttss2si:
2469  handleVectorConvertIntrinsic(I, 1);
2470  break;
2471  case Intrinsic::x86_sse_cvtps2pi:
2472  case Intrinsic::x86_sse_cvttps2pi:
2473  handleVectorConvertIntrinsic(I, 2);
2474  break;
2475 
2476  case Intrinsic::x86_avx512_psll_w_512:
2477  case Intrinsic::x86_avx512_psll_d_512:
2478  case Intrinsic::x86_avx512_psll_q_512:
2479  case Intrinsic::x86_avx512_pslli_w_512:
2480  case Intrinsic::x86_avx512_pslli_d_512:
2481  case Intrinsic::x86_avx512_pslli_q_512:
2482  case Intrinsic::x86_avx512_psrl_w_512:
2483  case Intrinsic::x86_avx512_psrl_d_512:
2484  case Intrinsic::x86_avx512_psrl_q_512:
2485  case Intrinsic::x86_avx512_psra_w_512:
2486  case Intrinsic::x86_avx512_psra_d_512:
2487  case Intrinsic::x86_avx512_psra_q_512:
2488  case Intrinsic::x86_avx512_psrli_w_512:
2489  case Intrinsic::x86_avx512_psrli_d_512:
2490  case Intrinsic::x86_avx512_psrli_q_512:
2491  case Intrinsic::x86_avx512_psrai_w_512:
2492  case Intrinsic::x86_avx512_psrai_d_512:
2493  case Intrinsic::x86_avx512_psrai_q_512:
2494  case Intrinsic::x86_avx512_psra_q_256:
2495  case Intrinsic::x86_avx512_psra_q_128:
2496  case Intrinsic::x86_avx512_psrai_q_256:
2497  case Intrinsic::x86_avx512_psrai_q_128:
2498  case Intrinsic::x86_avx2_psll_w:
2499  case Intrinsic::x86_avx2_psll_d:
2500  case Intrinsic::x86_avx2_psll_q:
2501  case Intrinsic::x86_avx2_pslli_w:
2502  case Intrinsic::x86_avx2_pslli_d:
2503  case Intrinsic::x86_avx2_pslli_q:
2504  case Intrinsic::x86_avx2_psrl_w:
2505  case Intrinsic::x86_avx2_psrl_d:
2506  case Intrinsic::x86_avx2_psrl_q:
2507  case Intrinsic::x86_avx2_psra_w:
2508  case Intrinsic::x86_avx2_psra_d:
2509  case Intrinsic::x86_avx2_psrli_w:
2510  case Intrinsic::x86_avx2_psrli_d:
2511  case Intrinsic::x86_avx2_psrli_q:
2512  case Intrinsic::x86_avx2_psrai_w:
2513  case Intrinsic::x86_avx2_psrai_d:
2514  case Intrinsic::x86_sse2_psll_w:
2515  case Intrinsic::x86_sse2_psll_d:
2516  case Intrinsic::x86_sse2_psll_q:
2517  case Intrinsic::x86_sse2_pslli_w:
2518  case Intrinsic::x86_sse2_pslli_d:
2519  case Intrinsic::x86_sse2_pslli_q:
2520  case Intrinsic::x86_sse2_psrl_w:
2521  case Intrinsic::x86_sse2_psrl_d:
2522  case Intrinsic::x86_sse2_psrl_q:
2523  case Intrinsic::x86_sse2_psra_w:
2524  case Intrinsic::x86_sse2_psra_d:
2525  case Intrinsic::x86_sse2_psrli_w:
2526  case Intrinsic::x86_sse2_psrli_d:
2527  case Intrinsic::x86_sse2_psrli_q:
2528  case Intrinsic::x86_sse2_psrai_w:
2529  case Intrinsic::x86_sse2_psrai_d:
2530  case Intrinsic::x86_mmx_psll_w:
2531  case Intrinsic::x86_mmx_psll_d:
2532  case Intrinsic::x86_mmx_psll_q:
2533  case Intrinsic::x86_mmx_pslli_w:
2534  case Intrinsic::x86_mmx_pslli_d:
2535  case Intrinsic::x86_mmx_pslli_q:
2536  case Intrinsic::x86_mmx_psrl_w:
2537  case Intrinsic::x86_mmx_psrl_d:
2538  case Intrinsic::x86_mmx_psrl_q:
2539  case Intrinsic::x86_mmx_psra_w:
2540  case Intrinsic::x86_mmx_psra_d:
2541  case Intrinsic::x86_mmx_psrli_w:
2542  case Intrinsic::x86_mmx_psrli_d:
2543  case Intrinsic::x86_mmx_psrli_q:
2544  case Intrinsic::x86_mmx_psrai_w:
2545  case Intrinsic::x86_mmx_psrai_d:
2546  handleVectorShiftIntrinsic(I, /* Variable */ false);
2547  break;
2548  case Intrinsic::x86_avx2_psllv_d:
2549  case Intrinsic::x86_avx2_psllv_d_256:
2550  case Intrinsic::x86_avx512_psllv_d_512:
2551  case Intrinsic::x86_avx2_psllv_q:
2552  case Intrinsic::x86_avx2_psllv_q_256:
2553  case Intrinsic::x86_avx512_psllv_q_512:
2554  case Intrinsic::x86_avx2_psrlv_d:
2555  case Intrinsic::x86_avx2_psrlv_d_256:
2556  case Intrinsic::x86_avx512_psrlv_d_512:
2557  case Intrinsic::x86_avx2_psrlv_q:
2558  case Intrinsic::x86_avx2_psrlv_q_256:
2559  case Intrinsic::x86_avx512_psrlv_q_512:
2560  case Intrinsic::x86_avx2_psrav_d:
2561  case Intrinsic::x86_avx2_psrav_d_256:
2562  case Intrinsic::x86_avx512_psrav_d_512:
2563  case Intrinsic::x86_avx512_psrav_q_128:
2564  case Intrinsic::x86_avx512_psrav_q_256:
2565  case Intrinsic::x86_avx512_psrav_q_512:
2566  handleVectorShiftIntrinsic(I, /* Variable */ true);
2567  break;
2568 
2569  case Intrinsic::x86_sse2_packsswb_128:
2570  case Intrinsic::x86_sse2_packssdw_128:
2571  case Intrinsic::x86_sse2_packuswb_128:
2572  case Intrinsic::x86_sse41_packusdw:
2573  case Intrinsic::x86_avx2_packsswb:
2574  case Intrinsic::x86_avx2_packssdw:
2575  case Intrinsic::x86_avx2_packuswb:
2576  case Intrinsic::x86_avx2_packusdw:
2577  handleVectorPackIntrinsic(I);
2578  break;
2579 
2580  case Intrinsic::x86_mmx_packsswb:
2581  case Intrinsic::x86_mmx_packuswb:
2582  handleVectorPackIntrinsic(I, 16);
2583  break;
2584 
2585  case Intrinsic::x86_mmx_packssdw:
2586  handleVectorPackIntrinsic(I, 32);
2587  break;
2588 
2589  case Intrinsic::x86_mmx_psad_bw:
2590  case Intrinsic::x86_sse2_psad_bw:
2591  case Intrinsic::x86_avx2_psad_bw:
2592  handleVectorSadIntrinsic(I);
2593  break;
2594 
2595  case Intrinsic::x86_sse2_pmadd_wd:
2596  case Intrinsic::x86_avx2_pmadd_wd:
2597  case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
2598  case Intrinsic::x86_avx2_pmadd_ub_sw:
2599  handleVectorPmaddIntrinsic(I);
2600  break;
2601 
2602  case Intrinsic::x86_ssse3_pmadd_ub_sw:
2603  handleVectorPmaddIntrinsic(I, 8);
2604  break;
2605 
2606  case Intrinsic::x86_mmx_pmadd_wd:
2607  handleVectorPmaddIntrinsic(I, 16);
2608  break;
2609 
2610  case Intrinsic::x86_sse_cmp_ss:
2611  case Intrinsic::x86_sse2_cmp_sd:
2612  case Intrinsic::x86_sse_comieq_ss:
2613  case Intrinsic::x86_sse_comilt_ss:
2614  case Intrinsic::x86_sse_comile_ss:
2615  case Intrinsic::x86_sse_comigt_ss:
2616  case Intrinsic::x86_sse_comige_ss:
2617  case Intrinsic::x86_sse_comineq_ss:
2618  case Intrinsic::x86_sse_ucomieq_ss:
2619  case Intrinsic::x86_sse_ucomilt_ss:
2620  case Intrinsic::x86_sse_ucomile_ss:
2621  case Intrinsic::x86_sse_ucomigt_ss:
2622  case Intrinsic::x86_sse_ucomige_ss:
2623  case Intrinsic::x86_sse_ucomineq_ss:
2624  case Intrinsic::x86_sse2_comieq_sd:
2625  case Intrinsic::x86_sse2_comilt_sd:
2626  case Intrinsic::x86_sse2_comile_sd:
2627  case Intrinsic::x86_sse2_comigt_sd:
2628  case Intrinsic::x86_sse2_comige_sd:
2629  case Intrinsic::x86_sse2_comineq_sd:
2630  case Intrinsic::x86_sse2_ucomieq_sd:
2631  case Intrinsic::x86_sse2_ucomilt_sd:
2632  case Intrinsic::x86_sse2_ucomile_sd:
2633  case Intrinsic::x86_sse2_ucomigt_sd:
2634  case Intrinsic::x86_sse2_ucomige_sd:
2635  case Intrinsic::x86_sse2_ucomineq_sd:
2636  handleVectorCompareScalarIntrinsic(I);
2637  break;
2638 
2639  case Intrinsic::x86_sse_cmp_ps:
2640  case Intrinsic::x86_sse2_cmp_pd:
2641  // FIXME: For x86_avx_cmp_pd_256 and x86_avx_cmp_ps_256 this function
2642  // generates reasonably looking IR that fails in the backend with "Do not
2643  // know how to split the result of this operator!".
2644  handleVectorComparePackedIntrinsic(I);
2645  break;
2646 
2647  default:
2648  if (!handleUnknownIntrinsic(I))
2649  visitInstruction(I);
2650  break;
2651  }
2652  }
2653 
2654  void visitCallSite(CallSite CS) {
2655  Instruction &I = *CS.getInstruction();
2656  if (I.getMetadata("nosanitize")) return;
2657  assert((CS.isCall() || CS.isInvoke()) && "Unknown type of CallSite");
2658  if (CS.isCall()) {
2659  CallInst *Call = cast<CallInst>(&I);
2660 
2661  // For inline asm, do the usual thing: check argument shadow and mark all
2662  // outputs as clean. Note that any side effects of the inline asm that are
2663  // not immediately visible in its constraints are not handled.
2664  if (Call->isInlineAsm()) {
2665  visitInstruction(I);
2666  return;
2667  }
2668 
2669  assert(!isa<IntrinsicInst>(&I) && "intrinsics are handled elsewhere");
2670 
2671  // We are going to insert code that relies on the fact that the callee
2672  // will become a non-readonly function after it is instrumented by us. To
2673  // prevent this code from being optimized out, mark that function
2674  // non-readonly in advance.
2675  if (Function *Func = Call->getCalledFunction()) {
2676  // Clear out readonly/readnone attributes.
2677  AttrBuilder B;
2678  B.addAttribute(Attribute::ReadOnly)
2679  .addAttribute(Attribute::ReadNone);
2681  }
2682 
2684  }
2685  IRBuilder<> IRB(&I);
2686 
2687  unsigned ArgOffset = 0;
2688  DEBUG(dbgs() << " CallSite: " << I << "\n");
2689  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
2690  ArgIt != End; ++ArgIt) {
2691  Value *A = *ArgIt;
2692  unsigned i = ArgIt - CS.arg_begin();
2693  if (!A->getType()->isSized()) {
2694  DEBUG(dbgs() << "Arg " << i << " is not sized: " << I << "\n");
2695  continue;
2696  }
2697  unsigned Size = 0;
2698  Value *Store = nullptr;
2699  // Compute the Shadow for arg even if it is ByVal, because
2700  // in that case getShadow() will copy the actual arg shadow to
2701  // __msan_param_tls.
2702  Value *ArgShadow = getShadow(A);
2703  Value *ArgShadowBase = getShadowPtrForArgument(A, IRB, ArgOffset);
2704  DEBUG(dbgs() << " Arg#" << i << ": " << *A <<
2705  " Shadow: " << *ArgShadow << "\n");
2706  bool ArgIsInitialized = false;
2707  const DataLayout &DL = F.getParent()->getDataLayout();
2708  if (CS.paramHasAttr(i, Attribute::ByVal)) {
2709  assert(A->getType()->isPointerTy() &&
2710  "ByVal argument is not a pointer!");
2711  Size = DL.getTypeAllocSize(A->getType()->getPointerElementType());
2712  if (ArgOffset + Size > kParamTLSSize) break;
2713  unsigned ParamAlignment = CS.getParamAlignment(i);
2714  unsigned Alignment = std::min(ParamAlignment, kShadowTLSAlignment);
2715  Store = IRB.CreateMemCpy(ArgShadowBase,
2716  getShadowPtr(A, Type::getInt8Ty(*MS.C), IRB),
2717  Size, Alignment);
2718  } else {
2719  Size = DL.getTypeAllocSize(A->getType());
2720  if (ArgOffset + Size > kParamTLSSize) break;
2721  Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
2722  kShadowTLSAlignment);
2723  Constant *Cst = dyn_cast<Constant>(ArgShadow);
2724  if (Cst && Cst->isNullValue()) ArgIsInitialized = true;
2725  }
2726  if (MS.TrackOrigins && !ArgIsInitialized)
2727  IRB.CreateStore(getOrigin(A),
2728  getOriginPtrForArgument(A, IRB, ArgOffset));
2729  (void)Store;
2730  assert(Size != 0 && Store != nullptr);
2731  DEBUG(dbgs() << " Param:" << *Store << "\n");
2732  ArgOffset += alignTo(Size, 8);
2733  }
2734  DEBUG(dbgs() << " done with call args\n");
2735 
2736  FunctionType *FT =
2737  cast<FunctionType>(CS.getCalledValue()->getType()->getContainedType(0));
2738  if (FT->isVarArg()) {
2739  VAHelper->visitCallSite(CS, IRB);
2740  }
2741 
2742  // Now, get the shadow for the RetVal.
2743  if (!I.getType()->isSized()) return;
2744  // Don't emit the epilogue for musttail call returns.
2745  if (CS.isCall() && cast<CallInst>(&I)->isMustTailCall()) return;
2746  IRBuilder<> IRBBefore(&I);
2747  // Until we have full dynamic coverage, make sure the retval shadow is 0.
2748  Value *Base = getShadowPtrForRetval(&I, IRBBefore);
2749  IRBBefore.CreateAlignedStore(getCleanShadow(&I), Base, kShadowTLSAlignment);
2750  BasicBlock::iterator NextInsn;
2751  if (CS.isCall()) {
2752  NextInsn = ++I.getIterator();
2753  assert(NextInsn != I.getParent()->end());
2754  } else {
2755  BasicBlock *NormalDest = cast<InvokeInst>(&I)->getNormalDest();
2756  if (!NormalDest->getSinglePredecessor()) {
2757  // FIXME: this case is tricky, so we are just conservative here.
2758  // Perhaps we need to split the edge between this BB and NormalDest,
2759  // but a naive attempt to use SplitEdge leads to a crash.
2760  setShadow(&I, getCleanShadow(&I));
2761  setOrigin(&I, getCleanOrigin());
2762  return;
2763  }
2764  NextInsn = NormalDest->getFirstInsertionPt();
2765  assert(NextInsn != NormalDest->end() &&
2766  "Could not find insertion point for retval shadow load");
2767  }
2768  IRBuilder<> IRBAfter(&*NextInsn);
2769  Value *RetvalShadow =
2770  IRBAfter.CreateAlignedLoad(getShadowPtrForRetval(&I, IRBAfter),
2771  kShadowTLSAlignment, "_msret");
2772  setShadow(&I, RetvalShadow);
2773  if (MS.TrackOrigins)
2774  setOrigin(&I, IRBAfter.CreateLoad(getOriginPtrForRetval(IRBAfter)));
2775  }
2776 
2777  bool isAMustTailRetVal(Value *RetVal) {
2778  if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
2779  RetVal = I->getOperand(0);
2780  }
2781  if (auto *I = dyn_cast<CallInst>(RetVal)) {
2782  return I->isMustTailCall();
2783  }
2784  return false;
2785  }
2786 
2787  void visitReturnInst(ReturnInst &I) {
2788  IRBuilder<> IRB(&I);
2789  Value *RetVal = I.getReturnValue();
2790  if (!RetVal) return;
2791  // Don't emit the epilogue for musttail call returns.
2792  if (isAMustTailRetVal(RetVal)) return;
2793  Value *ShadowPtr = getShadowPtrForRetval(RetVal, IRB);
2794  if (CheckReturnValue) {
2795  insertShadowCheck(RetVal, &I);
2796  Value *Shadow = getCleanShadow(RetVal);
2797  IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
2798  } else {
2799  Value *Shadow = getShadow(RetVal);
2800  IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
2801  if (MS.TrackOrigins)
2802  IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB));
2803  }
2804  }
2805 
2806  void visitPHINode(PHINode &I) {
2807  IRBuilder<> IRB(&I);
2808  if (!PropagateShadow) {
2809  setShadow(&I, getCleanShadow(&I));
2810  setOrigin(&I, getCleanOrigin());
2811  return;
2812  }
2813 
2814  ShadowPHINodes.push_back(&I);
2815  setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
2816  "_msphi_s"));
2817  if (MS.TrackOrigins)
2818  setOrigin(&I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(),
2819  "_msphi_o"));
2820  }
2821 
2822  void visitAllocaInst(AllocaInst &I) {
2823  setShadow(&I, getCleanShadow(&I));
2824  setOrigin(&I, getCleanOrigin());
2825  IRBuilder<> IRB(I.getNextNode());
2826  const DataLayout &DL = F.getParent()->getDataLayout();
2827  uint64_t TypeSize = DL.getTypeAllocSize(I.getAllocatedType());
2828  Value *Len = ConstantInt::get(MS.IntptrTy, TypeSize);
2829  if (I.isArrayAllocation())
2830  Len = IRB.CreateMul(Len, I.getArraySize());
2831  if (PoisonStack && ClPoisonStackWithCall) {
2832  IRB.CreateCall(MS.MsanPoisonStackFn,
2833  {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
2834  } else {
2835  Value *ShadowBase = getShadowPtr(&I, Type::getInt8PtrTy(*MS.C), IRB);
2836  Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0);
2837  IRB.CreateMemSet(ShadowBase, PoisonValue, Len, I.getAlignment());
2838  }
2839 
2840  if (PoisonStack && MS.TrackOrigins) {
2841  SmallString<2048> StackDescriptionStorage;
2842  raw_svector_ostream StackDescription(StackDescriptionStorage);
2843  // We create a string with a description of the stack allocation and
2844  // pass it into __msan_set_alloca_origin.
2845  // It will be printed by the run-time if stack-originated UMR is found.
2846  // The first 4 bytes of the string are set to '----' and will be replaced
2847  // by __msan_va_arg_overflow_size_tls at the first call.
2848  StackDescription << "----" << I.getName() << "@" << F.getName();
2849  Value *Descr =
2851  StackDescription.str());
2852 
2853  IRB.CreateCall(MS.MsanSetAllocaOrigin4Fn,
2854  {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
2855  IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy()),
2856  IRB.CreatePointerCast(&F, MS.IntptrTy)});
2857  }
2858  }
2859 
2860  void visitSelectInst(SelectInst& I) {
2861  IRBuilder<> IRB(&I);
2862  // a = select b, c, d
2863  Value *B = I.getCondition();
2864  Value *C = I.getTrueValue();
2865  Value *D = I.getFalseValue();
2866  Value *Sb = getShadow(B);
2867  Value *Sc = getShadow(C);
2868  Value *Sd = getShadow(D);
2869 
2870  // Result shadow if condition shadow is 0.
2871  Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
2872  Value *Sa1;
2873  if (I.getType()->isAggregateType()) {
2874  // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
2875  // an extra "select". This results in much more compact IR.
2876  // Sa = select Sb, poisoned, (select b, Sc, Sd)
2877  Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
2878  } else {
2879  // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
2880  // If Sb (condition is poisoned), look for bits in c and d that are equal
2881  // and both unpoisoned.
2882  // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
2883 
2884  // Cast arguments to shadow-compatible type.
2885  C = CreateAppToShadowCast(IRB, C);
2886  D = CreateAppToShadowCast(IRB, D);
2887 
2888  // Result shadow if condition shadow is 1.
2889  Sa1 = IRB.CreateOr(IRB.CreateXor(C, D), IRB.CreateOr(Sc, Sd));
2890  }
2891  Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
2892  setShadow(&I, Sa);
2893  if (MS.TrackOrigins) {
2894  // Origins are always i32, so any vector conditions must be flattened.
2895  // FIXME: consider tracking vector origins for app vectors?
2896  if (B->getType()->isVectorTy()) {
2897  Type *FlatTy = getShadowTyNoVec(B->getType());
2898  B = IRB.CreateICmpNE(IRB.CreateBitCast(B, FlatTy),
2899  ConstantInt::getNullValue(FlatTy));
2900  Sb = IRB.CreateICmpNE(IRB.CreateBitCast(Sb, FlatTy),
2901  ConstantInt::getNullValue(FlatTy));
2902  }
2903  // a = select b, c, d
2904  // Oa = Sb ? Ob : (b ? Oc : Od)
2905  setOrigin(
2906  &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()),
2907  IRB.CreateSelect(B, getOrigin(I.getTrueValue()),
2908  getOrigin(I.getFalseValue()))));
2909  }
2910  }
2911 
2912  void visitLandingPadInst(LandingPadInst &I) {
2913  // Do nothing.
2914  // See https://github.com/google/sanitizers/issues/504
2915  setShadow(&I, getCleanShadow(&I));
2916  setOrigin(&I, getCleanOrigin());
2917  }
2918 
2919  void visitCatchSwitchInst(CatchSwitchInst &I) {
2920  setShadow(&I, getCleanShadow(&I));
2921  setOrigin(&I, getCleanOrigin());
2922  }
2923 
2924  void visitFuncletPadInst(FuncletPadInst &I) {
2925  setShadow(&I, getCleanShadow(&I));
2926  setOrigin(&I, getCleanOrigin());
2927  }
2928 
2929  void visitGetElementPtrInst(GetElementPtrInst &I) {
2930  handleShadowOr(I);
2931  }
2932 
2933  void visitExtractValueInst(ExtractValueInst &I) {
2934  IRBuilder<> IRB(&I);
2935  Value *Agg = I.getAggregateOperand();
2936  DEBUG(dbgs() << "ExtractValue: " << I << "\n");
2937  Value *AggShadow = getShadow(Agg);
2938  DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
2939  Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
2940  DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n");
2941  setShadow(&I, ResShadow);
2942  setOriginForNaryOp(I);
2943  }
2944 
2945  void visitInsertValueInst(InsertValueInst &I) {
2946  IRBuilder<> IRB(&I);
2947  DEBUG(dbgs() << "InsertValue: " << I << "\n");
2948  Value *AggShadow = getShadow(I.getAggregateOperand());
2949  Value *InsShadow = getShadow(I.getInsertedValueOperand());
2950  DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
2951  DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n");
2952  Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
2953  DEBUG(dbgs() << " Res: " << *Res << "\n");
2954  setShadow(&I, Res);
2955  setOriginForNaryOp(I);
2956  }
2957 
2958  void dumpInst(Instruction &I) {
2959  if (CallInst *CI = dyn_cast<CallInst>(&I)) {
2960  errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
2961  } else {
2962  errs() << "ZZZ " << I.getOpcodeName() << "\n";
2963  }
2964  errs() << "QQQ " << I << "\n";
2965  }
2966 
2967  void visitResumeInst(ResumeInst &I) {
2968  DEBUG(dbgs() << "Resume: " << I << "\n");
2969  // Nothing to do here.
2970  }
2971 
2972  void visitCleanupReturnInst(CleanupReturnInst &CRI) {
2973  DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
2974  // Nothing to do here.
2975  }
2976 
2977  void visitCatchReturnInst(CatchReturnInst &CRI) {
2978  DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
2979  // Nothing to do here.
2980  }
2981 
2982  void visitInstruction(Instruction &I) {
2983  // Everything else: stop propagating and check for poisoned shadow.
2985  dumpInst(I);
2986  DEBUG(dbgs() << "DEFAULT: " << I << "\n");
2987  for (size_t i = 0, n = I.getNumOperands(); i < n; i++) {
2988  Value *Operand = I.getOperand(i);
2989  if (Operand->getType()->isSized())
2990  insertShadowCheck(Operand, &I);
2991  }
2992  setShadow(&I, getCleanShadow(&I));
2993  setOrigin(&I, getCleanOrigin());
2994  }
2995 };
2996 
2997 /// \brief AMD64-specific implementation of VarArgHelper.
2998 struct VarArgAMD64Helper : public VarArgHelper {
2999  // An unfortunate workaround for asymmetric lowering of va_arg stuff.
3000  // See a comment in visitCallSite for more details.
3001  static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7
3002  static const unsigned AMD64FpEndOffset = 176;
3003 
3004  Function &F;
3005  MemorySanitizer &MS;
3006  MemorySanitizerVisitor &MSV;
3007  Value *VAArgTLSCopy = nullptr;
3008  Value *VAArgOverflowSize = nullptr;
3009 
3010  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3011 
3012  enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
3013 
3014  VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
3015  MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
3016 
3017  ArgKind classifyArgument(Value* arg) {
3018  // A very rough approximation of X86_64 argument classification rules.
3019  Type *T = arg->getType();
3020  if (T->isFPOrFPVectorTy() || T->isX86_MMXTy())
3021  return AK_FloatingPoint;
3022  if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
3023  return AK_GeneralPurpose;
3024  if (T->isPointerTy())
3025  return AK_GeneralPurpose;
3026  return AK_Memory;
3027  }
3028 
3029  // For VarArg functions, store the argument shadow in an ABI-specific format
3030  // that corresponds to va_list layout.
3031  // We do this because Clang lowers va_arg in the frontend, and this pass
3032  // only sees the low level code that deals with va_list internals.
3033  // A much easier alternative (provided that Clang emits va_arg instructions)
3034  // would have been to associate each live instance of va_list with a copy of
3035  // MSanParamTLS, and extract shadow on va_arg() call in the argument list
3036  // order.
3037  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3038  unsigned GpOffset = 0;
3039  unsigned FpOffset = AMD64GpEndOffset;
3040  unsigned OverflowOffset = AMD64FpEndOffset;
3041  const DataLayout &DL = F.getParent()->getDataLayout();
3042  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
3043  ArgIt != End; ++ArgIt) {
3044  Value *A = *ArgIt;
3045  unsigned ArgNo = CS.getArgumentNo(ArgIt);
3046  bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
3047  bool IsByVal = CS.paramHasAttr(ArgNo, Attribute::ByVal);
3048  if (IsByVal) {
3049  // ByVal arguments always go to the overflow area.
3050  // Fixed arguments passed through the overflow area will be stepped
3051  // over by va_start, so don't count them towards the offset.
3052  if (IsFixed)
3053  continue;
3054  assert(A->getType()->isPointerTy());
3055  Type *RealTy = A->getType()->getPointerElementType();
3056  uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
3057  Value *Base = getShadowPtrForVAArgument(RealTy, IRB, OverflowOffset);
3058  OverflowOffset += alignTo(ArgSize, 8);
3059  IRB.CreateMemCpy(Base, MSV.getShadowPtr(A, IRB.getInt8Ty(), IRB),
3060  ArgSize, kShadowTLSAlignment);
3061  } else {
3062  ArgKind AK = classifyArgument(A);
3063  if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset)
3064  AK = AK_Memory;
3065  if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset)
3066  AK = AK_Memory;
3067  Value *Base;
3068  switch (AK) {
3069  case AK_GeneralPurpose:
3070  Base = getShadowPtrForVAArgument(A->getType(), IRB, GpOffset);
3071  GpOffset += 8;
3072  break;
3073  case AK_FloatingPoint:
3074  Base = getShadowPtrForVAArgument(A->getType(), IRB, FpOffset);
3075  FpOffset += 16;
3076  break;
3077  case AK_Memory:
3078  if (IsFixed)
3079  continue;
3080  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
3081  Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset);
3082  OverflowOffset += alignTo(ArgSize, 8);
3083  }
3084  // Take fixed arguments into account for GpOffset and FpOffset,
3085  // but don't actually store shadows for them.
3086  if (IsFixed)
3087  continue;
3088  IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
3089  }
3090  }
3091  Constant *OverflowSize =
3092  ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
3093  IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
3094  }
3095 
3096  /// \brief Compute the shadow address for a given va_arg.
3097  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
3098  int ArgOffset) {
3099  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
3100  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3101  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
3102  "_msarg");
3103  }
3104 
3105  void visitVAStartInst(VAStartInst &I) override {
3107  return;
3108  IRBuilder<> IRB(&I);
3109  VAStartInstrumentationList.push_back(&I);
3110  Value *VAListTag = I.getArgOperand(0);
3111  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3112 
3113  // Unpoison the whole __va_list_tag.
3114  // FIXME: magic ABI constants.
3115  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3116  /* size */24, /* alignment */8, false);
3117  }
3118 
3119  void visitVACopyInst(VACopyInst &I) override {
3121  return;
3122  IRBuilder<> IRB(&I);
3123  Value *VAListTag = I.getArgOperand(0);
3124  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3125 
3126  // Unpoison the whole __va_list_tag.
3127  // FIXME: magic ABI constants.
3128  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3129  /* size */24, /* alignment */8, false);
3130  }
3131 
3132  void finalizeInstrumentation() override {
3133  assert(!VAArgOverflowSize && !VAArgTLSCopy &&
3134  "finalizeInstrumentation called twice");
3135  if (!VAStartInstrumentationList.empty()) {
3136  // If there is a va_start in this function, make a backup copy of
3137  // va_arg_tls somewhere in the function entry block.
3139  VAArgOverflowSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
3140  Value *CopySize =
3141  IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset),
3142  VAArgOverflowSize);
3143  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3144  IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8);
3145  }
3146 
3147  // Instrument va_start.
3148  // Copy va_list shadow from the backup copy of the TLS contents.
3149  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
3150  CallInst *OrigInst = VAStartInstrumentationList[i];
3151  IRBuilder<> IRB(OrigInst->getNextNode());
3152  Value *VAListTag = OrigInst->getArgOperand(0);
3153 
3154  Value *RegSaveAreaPtrPtr =
3155  IRB.CreateIntToPtr(
3156  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3157  ConstantInt::get(MS.IntptrTy, 16)),
3158  Type::getInt64PtrTy(*MS.C));
3159  Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr);
3160  Value *RegSaveAreaShadowPtr =
3161  MSV.getShadowPtr(RegSaveAreaPtr, IRB.getInt8Ty(), IRB);
3162  IRB.CreateMemCpy(RegSaveAreaShadowPtr, VAArgTLSCopy,
3163  AMD64FpEndOffset, 16);
3164 
3165  Value *OverflowArgAreaPtrPtr =
3166  IRB.CreateIntToPtr(
3167  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3168  ConstantInt::get(MS.IntptrTy, 8)),
3169  Type::getInt64PtrTy(*MS.C));
3170  Value *OverflowArgAreaPtr = IRB.CreateLoad(OverflowArgAreaPtrPtr);
3171  Value *OverflowArgAreaShadowPtr =
3172  MSV.getShadowPtr(OverflowArgAreaPtr, IRB.getInt8Ty(), IRB);
3173  Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy,
3174  AMD64FpEndOffset);
3175  IRB.CreateMemCpy(OverflowArgAreaShadowPtr, SrcPtr, VAArgOverflowSize, 16);
3176  }
3177  }
3178 };
3179 
3180 /// \brief MIPS64-specific implementation of VarArgHelper.
3181 struct VarArgMIPS64Helper : public VarArgHelper {
3182  Function &F;
3183  MemorySanitizer &MS;
3184  MemorySanitizerVisitor &MSV;
3185  Value *VAArgTLSCopy = nullptr;
3186  Value *VAArgSize = nullptr;
3187 
3188  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3189 
3190  VarArgMIPS64Helper(Function &F, MemorySanitizer &MS,
3191  MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
3192 
3193  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3194  unsigned VAArgOffset = 0;
3195  const DataLayout &DL = F.getParent()->getDataLayout();
3196  for (CallSite::arg_iterator ArgIt = CS.arg_begin() +
3197  CS.getFunctionType()->getNumParams(), End = CS.arg_end();
3198  ArgIt != End; ++ArgIt) {
3199  Triple TargetTriple(F.getParent()->getTargetTriple());
3200  Value *A = *ArgIt;
3201  Value *Base;
3202  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
3203  if (TargetTriple.getArch() == Triple::mips64) {
3204  // Adjusting the shadow for argument with size < 8 to match the placement
3205  // of bits in big endian system
3206  if (ArgSize < 8)
3207  VAArgOffset += (8 - ArgSize);
3208  }
3209  Base = getShadowPtrForVAArgument(A->getType(), IRB, VAArgOffset);
3210  VAArgOffset += ArgSize;
3211  VAArgOffset = alignTo(VAArgOffset, 8);
3212  IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
3213  }
3214 
3215  Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(), VAArgOffset);
3216  // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
3217  // a new class member i.e. it is the total size of all VarArgs.
3218  IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
3219  }
3220 
3221  /// \brief Compute the shadow address for a given va_arg.
3222  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
3223  int ArgOffset) {
3224  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
3225  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3226  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
3227  "_msarg");
3228  }
3229 
3230  void visitVAStartInst(VAStartInst &I) override {
3231  IRBuilder<> IRB(&I);
3232  VAStartInstrumentationList.push_back(&I);
3233  Value *VAListTag = I.getArgOperand(0);
3234  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3235  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3236  /* size */8, /* alignment */8, false);
3237  }
3238 
3239  void visitVACopyInst(VACopyInst &I) override {
3240  IRBuilder<> IRB(&I);
3241  Value *VAListTag = I.getArgOperand(0);
3242  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3243  // Unpoison the whole __va_list_tag.
3244  // FIXME: magic ABI constants.
3245  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3246  /* size */8, /* alignment */8, false);
3247  }
3248 
3249  void finalizeInstrumentation() override {
3250  assert(!VAArgSize && !VAArgTLSCopy &&
3251  "finalizeInstrumentation called twice");
3253  VAArgSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
3254  Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
3255  VAArgSize);
3256 
3257  if (!VAStartInstrumentationList.empty()) {
3258  // If there is a va_start in this function, make a backup copy of
3259  // va_arg_tls somewhere in the function entry block.
3260  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3261  IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8);
3262  }
3263 
3264  // Instrument va_start.
3265  // Copy va_list shadow from the backup copy of the TLS contents.
3266  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
3267  CallInst *OrigInst = VAStartInstrumentationList[i];
3268  IRBuilder<> IRB(OrigInst->getNextNode());
3269  Value *VAListTag = OrigInst->getArgOperand(0);
3270  Value *RegSaveAreaPtrPtr =
3271  IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3272  Type::getInt64PtrTy(*MS.C));
3273  Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr);
3274  Value *RegSaveAreaShadowPtr =
3275  MSV.getShadowPtr(RegSaveAreaPtr, IRB.getInt8Ty(), IRB);
3276  IRB.CreateMemCpy(RegSaveAreaShadowPtr, VAArgTLSCopy, CopySize, 8);
3277  }
3278  }
3279 };
3280 
3281 /// \brief AArch64-specific implementation of VarArgHelper.
3282 struct VarArgAArch64Helper : public VarArgHelper {
3283  static const unsigned kAArch64GrArgSize = 64;
3284  static const unsigned kAArch64VrArgSize = 128;
3285 
3286  static const unsigned AArch64GrBegOffset = 0;
3287  static const unsigned AArch64GrEndOffset = kAArch64GrArgSize;
3288  // Make VR space aligned to 16 bytes.
3289  static const unsigned AArch64VrBegOffset = AArch64GrEndOffset;
3290  static const unsigned AArch64VrEndOffset = AArch64VrBegOffset
3291  + kAArch64VrArgSize;
3292  static const unsigned AArch64VAEndOffset = AArch64VrEndOffset;
3293 
3294  Function &F;
3295  MemorySanitizer &MS;
3296  MemorySanitizerVisitor &MSV;
3297  Value *VAArgTLSCopy = nullptr;
3298  Value *VAArgOverflowSize = nullptr;
3299 
3300  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3301 
3302  enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
3303 
3304  VarArgAArch64Helper(Function &F, MemorySanitizer &MS,
3305  MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
3306 
3307  ArgKind classifyArgument(Value* arg) {
3308  Type *T = arg->getType();
3309  if (T->isFPOrFPVectorTy())
3310  return AK_FloatingPoint;
3311  if ((T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
3312  || (T->isPointerTy()))
3313  return AK_GeneralPurpose;
3314  return AK_Memory;
3315  }
3316 
3317  // The instrumentation stores the argument shadow in a non ABI-specific
3318  // format because it does not know which argument is named (since Clang,
3319  // like x86_64 case, lowers the va_args in the frontend and this pass only
3320  // sees the low level code that deals with va_list internals).
3321  // The first seven GR registers are saved in the first 56 bytes of the
3322  // va_arg tls arra, followers by the first 8 FP/SIMD registers, and then
3323  // the remaining arguments.
3324  // Using constant offset within the va_arg TLS array allows fast copy
3325  // in the finalize instrumentation.
3326  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3327  unsigned GrOffset = AArch64GrBegOffset;
3328  unsigned VrOffset = AArch64VrBegOffset;
3329  unsigned OverflowOffset = AArch64VAEndOffset;
3330 
3331  const DataLayout &DL = F.getParent()->getDataLayout();
3332  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
3333  ArgIt != End; ++ArgIt) {
3334  Value *A = *ArgIt;
3335  unsigned ArgNo = CS.getArgumentNo(ArgIt);
3336  bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
3337  ArgKind AK = classifyArgument(A);
3338  if (AK == AK_GeneralPurpose && GrOffset >= AArch64GrEndOffset)
3339  AK = AK_Memory;
3340  if (AK == AK_FloatingPoint && VrOffset >= AArch64VrEndOffset)
3341  AK = AK_Memory;
3342  Value *Base;
3343  switch (AK) {
3344  case AK_GeneralPurpose:
3345  Base = getShadowPtrForVAArgument(A->getType(), IRB, GrOffset);
3346  GrOffset += 8;
3347  break;
3348  case AK_FloatingPoint:
3349  Base = getShadowPtrForVAArgument(A->getType(), IRB, VrOffset);
3350  VrOffset += 16;
3351  break;
3352  case AK_Memory:
3353  // Don't count fixed arguments in the overflow area - va_start will
3354  // skip right over them.
3355  if (IsFixed)
3356  continue;
3357  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
3358  Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset);
3359  OverflowOffset += alignTo(ArgSize, 8);
3360  break;
3361  }
3362  // Count Gp/Vr fixed arguments to their respective offsets, but don't
3363  // bother to actually store a shadow.
3364  if (IsFixed)
3365  continue;
3366  IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
3367  }
3368  Constant *OverflowSize =
3369  ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AArch64VAEndOffset);
3370  IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
3371  }
3372 
3373  /// Compute the shadow address for a given va_arg.
3374  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
3375  int ArgOffset) {
3376  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
3377  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3378  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
3379  "_msarg");
3380  }
3381 
3382  void visitVAStartInst(VAStartInst &I) override {
3383  IRBuilder<> IRB(&I);
3384  VAStartInstrumentationList.push_back(&I);
3385  Value *VAListTag = I.getArgOperand(0);
3386  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3387  // Unpoison the whole __va_list_tag.
3388  // FIXME: magic ABI constants (size of va_list).
3389  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3390  /* size */32, /* alignment */8, false);
3391  }
3392 
3393  void visitVACopyInst(VACopyInst &I) override {
3394  IRBuilder<> IRB(&I);
3395  Value *VAListTag = I.getArgOperand(0);
3396  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3397  // Unpoison the whole __va_list_tag.
3398  // FIXME: magic ABI constants (size of va_list).
3399  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3400  /* size */32, /* alignment */8, false);
3401  }
3402 
3403  // Retrieve a va_list field of 'void*' size.
3404  Value* getVAField64(IRBuilder<> &IRB, Value *VAListTag, int offset) {
3405  Value *SaveAreaPtrPtr =
3406  IRB.CreateIntToPtr(
3407  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3408  ConstantInt::get(MS.IntptrTy, offset)),
3409  Type::getInt64PtrTy(*MS.C));
3410  return IRB.CreateLoad(SaveAreaPtrPtr);
3411  }
3412 
3413  // Retrieve a va_list field of 'int' size.
3414  Value* getVAField32(IRBuilder<> &IRB, Value *VAListTag, int offset) {
3415  Value *SaveAreaPtr =
3416  IRB.CreateIntToPtr(
3417  IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3418  ConstantInt::get(MS.IntptrTy, offset)),
3419  Type::getInt32PtrTy(*MS.C));
3420  Value *SaveArea32 = IRB.CreateLoad(SaveAreaPtr);
3421  return IRB.CreateSExt(SaveArea32, MS.IntptrTy);
3422  }
3423 
3424  void finalizeInstrumentation() override {
3425  assert(!VAArgOverflowSize && !VAArgTLSCopy &&
3426  "finalizeInstrumentation called twice");
3427  if (!VAStartInstrumentationList.empty()) {
3428  // If there is a va_start in this function, make a backup copy of
3429  // va_arg_tls somewhere in the function entry block.
3431  VAArgOverflowSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
3432  Value *CopySize =
3433  IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AArch64VAEndOffset),
3434  VAArgOverflowSize);
3435  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3436  IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8);
3437  }
3438 
3439  Value *GrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64GrArgSize);
3440  Value *VrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64VrArgSize);
3441 
3442  // Instrument va_start, copy va_list shadow from the backup copy of
3443  // the TLS contents.
3444  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
3445  CallInst *OrigInst = VAStartInstrumentationList[i];
3446  IRBuilder<> IRB(OrigInst->getNextNode());
3447 
3448  Value *VAListTag = OrigInst->getArgOperand(0);
3449 
3450  // The variadic ABI for AArch64 creates two areas to save the incoming
3451  // argument registers (one for 64-bit general register xn-x7 and another
3452  // for 128-bit FP/SIMD vn-v7).
3453  // We need then to propagate the shadow arguments on both regions
3454  // 'va::__gr_top + va::__gr_offs' and 'va::__vr_top + va::__vr_offs'.
3455  // The remaning arguments are saved on shadow for 'va::stack'.
3456  // One caveat is it requires only to propagate the non-named arguments,
3457  // however on the call site instrumentation 'all' the arguments are
3458  // saved. So to copy the shadow values from the va_arg TLS array
3459  // we need to adjust the offset for both GR and VR fields based on
3460  // the __{gr,vr}_offs value (since they are stores based on incoming
3461  // named arguments).
3462 
3463  // Read the stack pointer from the va_list.
3464  Value *StackSaveAreaPtr = getVAField64(IRB, VAListTag, 0);
3465 
3466  // Read both the __gr_top and __gr_off and add them up.
3467  Value *GrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 8);
3468  Value *GrOffSaveArea = getVAField32(IRB, VAListTag, 24);
3469 
3470  Value *GrRegSaveAreaPtr = IRB.CreateAdd(GrTopSaveAreaPtr, GrOffSaveArea);
3471 
3472  // Read both the __vr_top and __vr_off and add them up.
3473  Value *VrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 16);
3474  Value *VrOffSaveArea = getVAField32(IRB, VAListTag, 28);
3475 
3476  Value *VrRegSaveAreaPtr = IRB.CreateAdd(VrTopSaveAreaPtr, VrOffSaveArea);
3477 
3478  // It does not know how many named arguments is being used and, on the
3479  // callsite all the arguments were saved. Since __gr_off is defined as
3480  // '0 - ((8 - named_gr) * 8)', the idea is to just propagate the variadic
3481  // argument by ignoring the bytes of shadow from named arguments.
3482  Value *GrRegSaveAreaShadowPtrOff =
3483  IRB.CreateAdd(GrArgSize, GrOffSaveArea);
3484 
3485  Value *GrRegSaveAreaShadowPtr =
3486  MSV.getShadowPtr(GrRegSaveAreaPtr, IRB.getInt8Ty(), IRB);
3487 
3488  Value *GrSrcPtr = IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
3489  GrRegSaveAreaShadowPtrOff);
3490  Value *GrCopySize = IRB.CreateSub(GrArgSize, GrRegSaveAreaShadowPtrOff);
3491 
3492  IRB.CreateMemCpy(GrRegSaveAreaShadowPtr, GrSrcPtr, GrCopySize, 8);
3493 
3494  // Again, but for FP/SIMD values.
3495  Value *VrRegSaveAreaShadowPtrOff =
3496  IRB.CreateAdd(VrArgSize, VrOffSaveArea);
3497 
3498  Value *VrRegSaveAreaShadowPtr =
3499  MSV.getShadowPtr(VrRegSaveAreaPtr, IRB.getInt8Ty(), IRB);
3500 
3501  Value *VrSrcPtr = IRB.CreateInBoundsGEP(
3502  IRB.getInt8Ty(),
3503  IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
3504  IRB.getInt32(AArch64VrBegOffset)),
3505  VrRegSaveAreaShadowPtrOff);
3506  Value *VrCopySize = IRB.CreateSub(VrArgSize, VrRegSaveAreaShadowPtrOff);
3507 
3508  IRB.CreateMemCpy(VrRegSaveAreaShadowPtr, VrSrcPtr, VrCopySize, 8);
3509 
3510  // And finally for remaining arguments.
3511  Value *StackSaveAreaShadowPtr =
3512  MSV.getShadowPtr(StackSaveAreaPtr, IRB.getInt8Ty(), IRB);
3513 
3514  Value *StackSrcPtr =
3515  IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
3516  IRB.getInt32(AArch64VAEndOffset));
3517 
3518  IRB.CreateMemCpy(StackSaveAreaShadowPtr, StackSrcPtr,
3519  VAArgOverflowSize, 16);
3520  }
3521  }
3522 };
3523 
3524 /// \brief PowerPC64-specific implementation of VarArgHelper.
3525 struct VarArgPowerPC64Helper : public VarArgHelper {
3526  Function &F;
3527  MemorySanitizer &MS;
3528  MemorySanitizerVisitor &MSV;
3529  Value *VAArgTLSCopy = nullptr;
3530  Value *VAArgSize = nullptr;
3531 
3532  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3533 
3534  VarArgPowerPC64Helper(Function &F, MemorySanitizer &MS,
3535  MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
3536 
3537  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3538  // For PowerPC, we need to deal with alignment of stack arguments -
3539  // they are mostly aligned to 8 bytes, but vectors and i128 arrays
3540  // are aligned to 16 bytes, byvals can be aligned to 8 or 16 bytes,
3541  // and QPX vectors are aligned to 32 bytes. For that reason, we
3542  // compute current offset from stack pointer (which is always properly
3543  // aligned), and offset for the first vararg, then subtract them.
3544  unsigned VAArgBase;
3545  Triple TargetTriple(F.getParent()->getTargetTriple());
3546  // Parameter save area starts at 48 bytes from frame pointer for ABIv1,
3547  // and 32 bytes for ABIv2. This is usually determined by target
3548  // endianness, but in theory could be overriden by function attribute.
3549  // For simplicity, we ignore it here (it'd only matter for QPX vectors).
3550  if (TargetTriple.getArch() == Triple::ppc64)
3551  VAArgBase = 48;
3552  else
3553  VAArgBase = 32;
3554  unsigned VAArgOffset = VAArgBase;
3555  const DataLayout &DL = F.getParent()->getDataLayout();
3556  for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
3557  ArgIt != End; ++ArgIt) {
3558  Value *A = *ArgIt;
3559  unsigned ArgNo = CS.getArgumentNo(ArgIt);
3560  bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
3561  bool IsByVal = CS.paramHasAttr(ArgNo, Attribute::ByVal);
3562  if (IsByVal) {
3563  assert(A->getType()->isPointerTy());
3564  Type *RealTy = A->getType()->getPointerElementType();
3565  uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
3566  uint64_t ArgAlign = CS.getParamAlignment(ArgNo);
3567  if (ArgAlign < 8)
3568  ArgAlign = 8;
3569  VAArgOffset = alignTo(VAArgOffset, ArgAlign);
3570  if (!IsFixed) {
3571  Value *Base = getShadowPtrForVAArgument(RealTy, IRB,
3572  VAArgOffset - VAArgBase);
3573  IRB.CreateMemCpy(Base, MSV.getShadowPtr(A, IRB.getInt8Ty(), IRB),
3574  ArgSize, kShadowTLSAlignment);
3575  }
3576  VAArgOffset += alignTo(ArgSize, 8);
3577  } else {
3578  Value *Base;
3579  uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
3580  uint64_t ArgAlign = 8;
3581  if (A->getType()->isArrayTy()) {
3582  // Arrays are aligned to element size, except for long double
3583  // arrays, which are aligned to 8 bytes.
3584  Type *ElementTy = A->getType()->getArrayElementType();
3585  if (!ElementTy->isPPC_FP128Ty())
3586  ArgAlign = DL.getTypeAllocSize(ElementTy);
3587  } else if (A->getType()->isVectorTy()) {
3588  // Vectors are naturally aligned.
3589  ArgAlign = DL.getTypeAllocSize(A->getType());
3590  }
3591  if (ArgAlign < 8)
3592  ArgAlign = 8;
3593  VAArgOffset = alignTo(VAArgOffset, ArgAlign);
3594  if (DL.isBigEndian()) {
3595  // Adjusting the shadow for argument with size < 8 to match the placement
3596  // of bits in big endian system
3597  if (ArgSize < 8)
3598  VAArgOffset += (8 - ArgSize);
3599  }
3600  if (!IsFixed) {
3601  Base = getShadowPtrForVAArgument(A->getType(), IRB,
3602  VAArgOffset - VAArgBase);
3603  IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
3604  }
3605  VAArgOffset += ArgSize;
3606  VAArgOffset = alignTo(VAArgOffset, 8);
3607  }
3608  if (IsFixed)
3609  VAArgBase = VAArgOffset;
3610  }
3611 
3612  Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(),
3613  VAArgOffset - VAArgBase);
3614  // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
3615  // a new class member i.e. it is the total size of all VarArgs.
3616  IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
3617  }
3618 
3619  /// \brief Compute the shadow address for a given va_arg.
3620  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
3621  int ArgOffset) {
3622  Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
3623  Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3624  return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
3625  "_msarg");
3626  }
3627 
3628  void visitVAStartInst(VAStartInst &I) override {
3629  IRBuilder<> IRB(&I);
3630  VAStartInstrumentationList.push_back(&I);
3631  Value *VAListTag = I.getArgOperand(0);
3632  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3633  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3634  /* size */8, /* alignment */8, false);
3635  }
3636 
3637  void visitVACopyInst(VACopyInst &I) override {
3638  IRBuilder<> IRB(&I);
3639  Value *VAListTag = I.getArgOperand(0);
3640  Value *ShadowPtr = MSV.getShadowPtr(VAListTag, IRB.getInt8Ty(), IRB);
3641  // Unpoison the whole __va_list_tag.
3642  // FIXME: magic ABI constants.
3643  IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3644  /* size */8, /* alignment */8, false);
3645  }
3646 
3647  void finalizeInstrumentation() override {
3648  assert(!VAArgSize && !VAArgTLSCopy &&
3649  "finalizeInstrumentation called twice");
3651  VAArgSize = IRB.CreateLoad(MS.VAArgOverflowSizeTLS);
3652  Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
3653  VAArgSize);
3654 
3655  if (!VAStartInstrumentationList.empty()) {
3656  // If there is a va_start in this function, make a backup copy of
3657  // va_arg_tls somewhere in the function entry block.
3658  VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3659  IRB.CreateMemCpy(VAArgTLSCopy, MS.VAArgTLS, CopySize, 8);
3660  }
3661 
3662  // Instrument va_start.
3663  // Copy va_list shadow from the backup copy of the TLS contents.
3664  for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
3665  CallInst *OrigInst = VAStartInstrumentationList[i];
3666  IRBuilder<> IRB(OrigInst->getNextNode());
3667  Value *VAListTag = OrigInst->getArgOperand(0);
3668  Value *RegSaveAreaPtrPtr =
3669  IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3670  Type::getInt64PtrTy(*MS.C));
3671  Value *RegSaveAreaPtr = IRB.CreateLoad(RegSaveAreaPtrPtr);
3672  Value *RegSaveAreaShadowPtr =
3673  MSV.getShadowPtr(RegSaveAreaPtr, IRB.getInt8Ty(), IRB);
3674  IRB.CreateMemCpy(RegSaveAreaShadowPtr, VAArgTLSCopy, CopySize, 8);
3675  }
3676  }
3677 };
3678 
3679 /// \brief A no-op implementation of VarArgHelper.
3680 struct VarArgNoOpHelper : public VarArgHelper {
3681  VarArgNoOpHelper(Function &F, MemorySanitizer &MS,
3682  MemorySanitizerVisitor &MSV) {}
3683 
3684  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {}
3685 
3686  void visitVAStartInst(VAStartInst &I) override {}
3687 
3688  void visitVACopyInst(VACopyInst &I) override {}
3689 
3690  void finalizeInstrumentation() override {}
3691 };
3692 
3693 } // end anonymous namespace
3694 
3695 static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
3696  MemorySanitizerVisitor &Visitor) {
3697  // VarArg handling is only implemented on AMD64. False positives are possible
3698  // on other platforms.
3699  Triple TargetTriple(Func.getParent()->getTargetTriple());
3700  if (TargetTriple.getArch() == Triple::x86_64)
3701  return new VarArgAMD64Helper(Func, Msan, Visitor);
3702  else if (TargetTriple.getArch() == Triple::mips64 ||
3703  TargetTriple.getArch() == Triple::mips64el)
3704  return new VarArgMIPS64Helper(Func, Msan, Visitor);
3705  else if (TargetTriple.getArch() == Triple::aarch64)
3706  return new VarArgAArch64Helper(Func, Msan, Visitor);
3707  else if (TargetTriple.getArch() == Triple::ppc64 ||
3708  TargetTriple.getArch() == Triple::ppc64le)
3709  return new VarArgPowerPC64Helper(Func, Msan, Visitor);
3710  else
3711  return new VarArgNoOpHelper(Func, Msan, Visitor);
3712 }
3713 
3715  if (&F == MsanCtorFunction)
3716  return false;
3717  MemorySanitizerVisitor Visitor(F, *this);
3718 
3719  // Clear out readonly/readnone attributes.
3720  AttrBuilder B;
3721  B.addAttribute(Attribute::ReadOnly)
3722  .addAttribute(Attribute::ReadNone);
3724 
3725  return Visitor.runOnFunction();
3726 }
Value * CreateInBoundsGEP(Value *Ptr, ArrayRef< Value *> IdxList, const Twine &Name="")
Definition: IRBuilder.h:1244
Type * getVectorElementType() const
Definition: Type.h:368
uint64_t CallInst * C
Return a value (possibly void), from a function.
User::op_iterator arg_iterator
The type of iterator to use when looping over actual arguments at this call site. ...
Definition: CallSite.h:213
SymbolTableList< Instruction >::iterator eraseFromParent()
This method unlinks &#39;this&#39; from the containing basic block and deletes it.
Definition: Instruction.cpp:69
unsigned Log2_32_Ceil(uint32_t Value)
Return the ceil log base 2 of the specified value, 32 if the value is zero.
Definition: MathExtras.h:544
A parsed version of the target data layout string in and methods for querying it. ...
Definition: DataLayout.h:109
const std::string & getTargetTriple() const
Get the target triple which is a string describing the target host.
Definition: Module.h:233
static const MemoryMapParams Linux_PowerPC64_MemoryMapParams
Value * CreateICmp(CmpInst::Predicate P, Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1638
Value * CreateConstGEP1_32(Value *Ptr, unsigned Idx0, const Twine &Name="")
Definition: IRBuilder.h:1278
static Constant * getString(LLVMContext &Context, StringRef Initializer, bool AddNull=true)
This method constructs a CDS and initializes it with a text string.
Definition: Constants.cpp:2451
bool isAllOnesValue() const
Return true if this is the value that would be returned by getAllOnesValue.
Definition: Constants.cpp:101
raw_ostream & errs()
This returns a reference to a raw_ostream for standard error.
void addIncoming(Value *V, BasicBlock *BB)
Add an incoming value to the end of the PHI list.
This instruction extracts a struct member or array element value from an aggregate value...
Value * CreateBinOp(Instruction::BinaryOps Opc, Value *LHS, Value *RHS, const Twine &Name="", MDNode *FPMathTag=nullptr)
Definition: IRBuilder.h:1112
GCNRegPressure max(const GCNRegPressure &P1, const GCNRegPressure &P2)
This class represents an incoming formal argument to a Function.
Definition: Argument.h:30
static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams
AllocaInst * CreateAlloca(Type *Ty, unsigned AddrSpace, Value *ArraySize=nullptr, const Twine &Name="")
Definition: IRBuilder.h:1156
Base class for instruction visitors.
Definition: InstVisitor.h:81
Value * getAggregateOperand()
Value * CreateICmpNE(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1553
Atomic ordering constants.
NodeTy * getNextNode()
Get the next node, or nullptr for the list tail.
Definition: ilist_node.h:289
LLVM_ATTRIBUTE_NORETURN void report_fatal_error(Error Err, bool gen_crash_diag=true)
Report a serious error, calling any installed error handler.
Definition: Error.cpp:115
Compute iterated dominance frontiers using a linear time algorithm.
Definition: AllocatorList.h:24
BinaryOps getOpcode() const
Definition: InstrTypes.h:523
unsigned getParamAlignment(unsigned ArgNo) const
Extract the alignment for a call or parameter (0=unknown).
Definition: CallSite.h:406
bool isAtomic() const
Return true if this instruction has an AtomicOrdering of unordered or higher.
Constant * getOrInsertFunction(StringRef Name, FunctionType *T, AttributeList AttributeList)
Look up the specified function in the module symbol table.
Definition: Module.cpp:142
Value * CreateXor(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1099
A Module instance is used to store all the information related to an LLVM module. ...
Definition: Module.h:63
bool isSized(SmallPtrSetImpl< Type *> *Visited=nullptr) const
Return true if it makes sense to take the size of this type.
Definition: Type.h:262
LLVM_ATTRIBUTE_ALWAYS_INLINE size_type size() const
Definition: SmallVector.h:136
void setOrdering(AtomicOrdering Ordering)
Sets the ordering constraint of this rmw instruction.
Definition: Instructions.h:762
static const MemoryMapParams Linux_I386_MemoryMapParams
Same, but only replaced by something equivalent.
Definition: GlobalValue.h:54
an instruction that atomically checks whether a specified value is in a memory location, and, if it is, stores a new value there.
Definition: Instructions.h:514
static cl::opt< bool > ClPoisonStackWithCall("msan-poison-stack-with-call", cl::desc("poison uninitialized stack variables with a call"), cl::Hidden, cl::init(false))
This class represents zero extension of integer types.
static const unsigned kRetvalTLSSize
Value * CreateICmpSLT(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1574
This class represents a function call, abstracting a target machine&#39;s calling convention.
static PointerType * getInt32PtrTy(LLVMContext &C, unsigned AS=0)
Definition: Type.cpp:228
void setOrdering(AtomicOrdering Ordering)
Sets the ordering constraint of this load instruction.
Definition: Instructions.h:239
static PointerType * get(Type *ElementType, unsigned AddressSpace)
This constructs a pointer to an object of the specified type in a numbered address space...
Definition: Type.cpp:617
const Value * getTrueValue() const
AtomicOrdering getOrdering() const
Returns the ordering constraint of this load instruction.
Definition: Instructions.h:233
Like Internal, but omit from symbol table.
Definition: GlobalValue.h:57
static VarArgHelper * CreateVarArgHelper(Function &Func, MemorySanitizer &Msan, MemorySanitizerVisitor &Visitor)
This instruction constructs a fixed permutation of two input vectors.
static cl::opt< bool > ClWithComdat("msan-with-comdat", cl::desc("Place MSan constructors in comdat sections"), cl::Hidden, cl::init(false))
Externally visible function.
Definition: GlobalValue.h:49
bool hasFnAttribute(Attribute::AttrKind Kind) const
Return true if the function has the attribute.
Definition: Function.h:262
A raw_ostream that writes to an SmallVector or SmallString.
Definition: raw_ostream.h:489
This class wraps the llvm.memset intrinsic.
static bool isEquality(Predicate P)
Return true if this predicate is either EQ or NE.
Value * CreateSExt(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1390
Metadata node.
Definition: Metadata.h:862
static unsigned TypeSizeToSizeIndex(unsigned TypeSize)
F(f)
This class represents a sign extension of integer types.
uint64_t alignTo(uint64_t Value, uint64_t Align, uint64_t Skew=0)
Returns the next integer (mod 2**64) that is greater than or equal to Value and is a multiple of Alig...
Definition: MathExtras.h:677
CallInst * CreateMemSet(Value *Ptr, Value *Val, uint64_t Size, unsigned Align, bool isVolatile=false, MDNode *TBAATag=nullptr, MDNode *ScopeTag=nullptr, MDNode *NoAliasTag=nullptr)
Create and insert a memset to the specified pointer and the specified value.
Definition: IRBuilder.h:405
An instruction for reading from memory.
Definition: Instructions.h:164
AttrBuilder & addAttribute(Attribute::AttrKind Val)
Add an attribute to the builder.
an instruction that atomically reads a memory location, combines it with another value, and then stores the result back.
Definition: Instructions.h:677
Hexagon Common GEP
bool isVectorTy() const
True if this is an instance of VectorType.
Definition: Type.h:227
#define op(i)
bool isMustTailCall() const
static Type * getX86_MMXTy(LLVMContext &C)
Definition: Type.cpp:171
Use * op_iterator
Definition: User.h:209
bool isPPC_FP128Ty() const
Return true if this is powerpc long double.
Definition: Type.h:159
static const MemoryMapParams Linux_AArch64_MemoryMapParams
static cl::opt< bool > ClHandleICmp("msan-handle-icmp", cl::desc("propagate shadow through ICmpEQ and ICmpNE"), cl::Hidden, cl::init(true))
op_iterator op_begin()
Definition: User.h:214
static PointerType * getInt64PtrTy(LLVMContext &C, unsigned AS=0)
Definition: Type.cpp:232
static Constant * get(ArrayType *T, ArrayRef< Constant *> V)
Definition: Constants.cpp:888
unsigned getBitWidth() const
Return the number of bits in the APInt.
Definition: APInt.h:1488
static Constant * getNullValue(Type *Ty)
Constructor to create a &#39;0&#39; constant of arbitrary type.
Definition: Constants.cpp:207
Value * CreateNot(Value *V, const Twine &Name="")
Definition: IRBuilder.h:1146
StoreInst * CreateAlignedStore(Value *Val, Value *Ptr, unsigned Align, bool isVolatile=false)
Definition: IRBuilder.h:1203
IntegerType * getInt32Ty()
Fetch the type representing a 32-bit integer.
Definition: IRBuilder.h:348
unsigned countTrailingZeros() const
Count the number of trailing zero bits.
Definition: APInt.h:1611
static cl::opt< int > ClPoisonStackPattern("msan-poison-stack-pattern", cl::desc("poison uninitialized stack variables with the given pattern"), cl::Hidden, cl::init(0xff))
ArrayRef< unsigned > getIndices() const
AnalysisUsage & addRequired()
#define INITIALIZE_PASS_DEPENDENCY(depName)
Definition: PassSupport.h:51
bool isSigned() const
Determine if this instruction is using a signed comparison.
Definition: InstrTypes.h:994
static cl::opt< bool > ClDumpStrictInstructions("msan-dump-strict-instructions", cl::desc("print out instructions with default strict semantics"), cl::Hidden, cl::init(false))
This class represents the LLVM &#39;select&#39; instruction.
Type * getPointerElementType() const
Definition: Type.h:373
const DataLayout & getDataLayout() const
Get the data layout for the module&#39;s target platform.
Definition: Module.cpp:361
unsigned getAlignment() const
Return the alignment of the memory that is being allocated by the instruction.
Definition: Instructions.h:109
unsigned getNumArgOperands() const
Return the number of call arguments.
IntegerType * getInt64Ty()
Fetch the type representing a 64-bit integer.
Definition: IRBuilder.h:353
This is the base class for all instructions that perform data casts.
Definition: InstrTypes.h:560
ArrayRef< T > makeArrayRef(const T &OneElt)
Construct an ArrayRef from a single element.
Definition: ArrayRef.h:451
&#39;undef&#39; values are things that do not have specified contents.
Definition: Constants.h:1247
This class wraps the llvm.memmove intrinsic.
Class to represent struct types.
Definition: DerivedTypes.h:201
LLVMContext & getContext() const
Get the global data context.
Definition: Module.h:237
static cl::opt< bool > ClCheckAccessAddress("msan-check-access-address", cl::desc("report accesses through a pointer which has poisoned shadow"), cl::Hidden, cl::init(true))
PointerType * getPointerTo(unsigned AddrSpace=0) const
Return a pointer to the current type.
Definition: Type.cpp:639
IterTy arg_end() const
Definition: CallSite.h:575
bool isUnsigned() const
Determine if this instruction is using an unsigned comparison.
Definition: InstrTypes.h:1000
bool isIntegerTy() const
True if this is an instance of IntegerType.
Definition: Type.h:197
This provides a uniform API for creating instructions and inserting them into a basic block: either a...
Definition: IRBuilder.h:668
IntegerType * getIntPtrTy(const DataLayout &DL, unsigned AddrSpace=0)
Fetch the type representing a pointer to an integer value.
Definition: IRBuilder.h:391
This file contains the simple types necessary to represent the attributes associated with functions a...
InstrTy * getInstruction() const
Definition: CallSite.h:92
Value * CreateAdd(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:893
void setName(const Twine &Name)
Change the name of the value.
Definition: Value.cpp:286
static StructType * get(LLVMContext &Context, ArrayRef< Type *> Elements, bool isPacked=false)
This static method is the primary way to create a literal StructType.
Definition: Type.cpp:336
This file implements a class to represent arbitrary precision integral constant values and operations...
Type * getVoidTy()
Fetch the type representing void.
Definition: IRBuilder.h:381
This class represents a cast from a pointer to an integer.
AtomicOrdering
Atomic ordering for LLVM&#39;s memory model.
StoreInst * CreateStore(Value *Val, Value *Ptr, bool isVolatile=false)
Definition: IRBuilder.h:1180
Value * CreateIntToPtr(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1444
ValTy * getCalledValue() const
Return the pointer to function that is being called.
Definition: CallSite.h:100
bool isNullValue() const
Return true if this is the value that would be returned by getNullValue.
Definition: Constants.cpp:86
Class to represent function types.
Definition: DerivedTypes.h:103
Value * CreateBitCast(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1448
Type * getType() const
All values are typed, get the type of this value.
Definition: Value.h:245
This represents the llvm.va_start intrinsic.
static const char *const kMsanInitName
std::string itostr(int64_t X)
Definition: StringExtras.h:189
AtomicOrdering getSuccessOrdering() const
Returns the success ordering constraint of this cmpxchg instruction.
Definition: Instructions.h:568
#define T
static cl::opt< int > ClTrackOrigins("msan-track-origins", cl::desc("Track origins (allocation sites) of poisoned memory"), cl::Hidden, cl::init(0))
Track origins of uninitialized values.
Class to represent array types.
Definition: DerivedTypes.h:369
This instruction compares its operands according to the predicate given to the constructor.
bool isVarArg() const
Definition: DerivedTypes.h:123
This class represents a no-op cast from one type to another.
bool paramHasAttr(unsigned ArgNo, Attribute::AttrKind Kind) const
Return true if the call or the callee has the given attribute.
Definition: CallSite.h:377
MDNode * getMetadata(unsigned KindID) const
Get the metadata of given kind attached to this Instruction.
Definition: Instruction.h:194
Value * getInsertedValueOperand()
SmallString - A SmallString is just a SmallVector with methods and accessors that make it work better...
Definition: SmallString.h:26
Value * CreateSub(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:915
An instruction for storing to memory.
Definition: Instructions.h:306
bool isIntOrIntVectorTy() const
Return true if this is an integer type or a vector of integer types.
Definition: Type.h:203
static const unsigned kParamTLSSize
Value * CreateZExt(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1387
static cl::opt< bool > ClPoisonStack("msan-poison-stack", cl::desc("poison uninitialized stack variables"), cl::Hidden, cl::init(true))
Function * getDeclaration(Module *M, ID id, ArrayRef< Type *> Tys=None)
Create or insert an LLVM Function declaration for an intrinsic, and return it.
Definition: Function.cpp:980
This class represents a truncation of integer types.
void SetInsertPoint(BasicBlock *TheBB)
This specifies that created instructions should be appended to the end of the specified block...
Definition: IRBuilder.h:128
bool onlyReadsMemory() const
Determine if the call does not access or only reads memory.
Value * getOperand(unsigned i) const
Definition: User.h:154
static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams
bool isRelational() const
Return true if the predicate is relational (not EQ or NE).
bool isCall() const
Return true if a CallInst is enclosed.
Definition: CallSite.h:87
Value * CreateOr(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1083
static const unsigned kMinOriginAlignment
Constant * getAggregateElement(unsigned Elt) const
For aggregates (struct/array/vector) return the constant that corresponds to the specified element if...
Definition: Constants.cpp:277
Type * getScalarType() const
If this is a vector type, return the element type, otherwise return &#39;this&#39;.
Definition: Type.h:301
bool isZeroValue() const
Return true if the value is negative zero or null value.
Definition: Constants.cpp:66
bool isVoidTy() const
Return true if this is &#39;void&#39;.
Definition: Type.h:141
const BasicBlock & getEntryBlock() const
Definition: Function.h:572
an instruction for type-safe pointer arithmetic to access elements of arrays and structs ...
Definition: Instructions.h:837
LoadInst * CreateLoad(Value *Ptr, const char *Name)
Definition: IRBuilder.h:1168
static bool runOnFunction(Function &F, bool PostInlining)
initializer< Ty > init(const Ty &Val)
Definition: CommandLine.h:406
This instruction inserts a single (scalar) element into a VectorType value.
The landingpad instruction holds all of the information necessary to generate correct exception handl...
const Instruction * getFirstNonPHI() const
Returns a pointer to the first instruction in this block that is not a PHINode instruction.
Definition: BasicBlock.cpp:171
const Value * getCalledValue() const
Get a pointer to the function that is invoked by this instruction.
static GCRegistry::Add< OcamlGC > B("ocaml", "ocaml 3.10-compatible GC")
const_iterator getFirstInsertionPt() const
Returns an iterator to the first instruction in this block that is suitable for inserting a non-PHI i...
Definition: BasicBlock.cpp:200
const BasicBlock * getSinglePredecessor() const
Return the predecessor of this block if it has a single predecessor block.
Definition: BasicBlock.cpp:217
INITIALIZE_PASS_BEGIN(MemorySanitizer, "msan", "MemorySanitizer: detects uninitialized reads.", false, false) INITIALIZE_PASS_END(MemorySanitizer
LLVM Basic Block Representation.
Definition: BasicBlock.h:59
The instances of the Type class are immutable: once they are created, they are never changed...
Definition: Type.h:46
std::pair< Function *, Function * > createSanitizerCtorAndInitFunctions(Module &M, StringRef CtorName, StringRef InitName, ArrayRef< Type *> InitArgTypes, ArrayRef< Value *> InitArgs, StringRef VersionCheckName=StringRef())
Creates sanitizer constructor function, and calls sanitizer&#39;s init function from it.
This is an important class for using LLVM in a threaded context.
Definition: LLVMContext.h:69
const char * getOpcodeName() const
Definition: Instruction.h:127
This is an important base class in LLVM.
Definition: Constant.h:42
Resume the propagation of an exception.
This file contains the declarations for the subclasses of Constant, which represent the different fla...
Value * CreateSelect(Value *C, Value *True, Value *False, const Twine &Name="", Instruction *MDFrom=nullptr)
Definition: IRBuilder.h:1693
bool isPointerTy() const
True if this is an instance of PointerType.
Definition: Type.h:221
unsigned getNumParams() const
Return the number of fixed parameters this function type requires.
Definition: DerivedTypes.h:139
Represent the analysis usage information of a pass.
op_iterator op_end()
Definition: User.h:216
static const PlatformMemoryMapParams Linux_X86_MemoryMapParams
This instruction compares its operands according to the predicate given to the constructor.
Predicate
This enumeration lists the possible predicates for CmpInst subclasses.
Definition: InstrTypes.h:853
static const unsigned End
FunctionPass class - This class is used to implement most global optimizations.
Definition: Pass.h:285
static const unsigned kShadowTLSAlignment
static FunctionType * get(Type *Result, ArrayRef< Type *> Params, bool isVarArg)
This static method is the primary way of constructing a FunctionType.
Definition: Type.cpp:297
static Constant * get(StructType *T, ArrayRef< Constant *> V)
Definition: Constants.cpp:949
Value * getPointerOperand()
Definition: Instructions.h:270
bool isX86_MMXTy() const
Return true if this is X86 MMX.
Definition: Type.h:182
Value * CreateICmpEQ(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1550
self_iterator getIterator()
Definition: ilist_node.h:82
Class to represent integer types.
Definition: DerivedTypes.h:40
IntegerType * getIntNTy(unsigned N)
Fetch the type representing an N-bit integer.
Definition: IRBuilder.h:361
Value * CreateExtractElement(Value *Vec, Value *Idx, const Twine &Name="")
Definition: IRBuilder.h:1713
This class represents a cast from an integer to a pointer.
const Value * getCondition() const
static Constant * getAllOnesValue(Type *Ty)
Get the all ones value.
Definition: Constants.cpp:261
static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams
static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams
Comdat * getOrInsertComdat(StringRef Name)
Return the Comdat in the module with the specified name.
Definition: Module.cpp:467
const Value * getArraySize() const
Get the number of elements allocated.
Definition: Instructions.h:93
Value * CreateExtractValue(Value *Agg, ArrayRef< unsigned > Idxs, const Twine &Name="")
Definition: IRBuilder.h:1755
static PointerType * getInt8PtrTy(LLVMContext &C, unsigned AS=0)
Definition: Type.cpp:220
PointerType * getInt8PtrTy(unsigned AddrSpace=0)
Fetch the type representing a pointer to an 8-bit integer value.
Definition: IRBuilder.h:386
MDNode * createBranchWeights(uint32_t TrueWeight, uint32_t FalseWeight)
Return metadata containing two branch weights.
Definition: MDBuilder.cpp:38
AtomicOrdering getOrdering() const
Returns the ordering constraint of this rmw instruction.
Definition: Instructions.h:757
INITIALIZE_PASS_END(RegBankSelect, DEBUG_TYPE, "Assign register bank of generic virtual registers", false, false) RegBankSelect
#define llvm_unreachable(msg)
Marks that the current location is not supposed to be reachable.
Value * CreateMul(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:937
Value * CreateTrunc(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1384
Type * getAllocatedType() const
Return the type that is being allocated by the instruction.
Definition: Instructions.h:102
Triple - Helper class for working with autoconf configuration names.
Definition: Triple.h:44
signed greater than
Definition: InstrTypes.h:880
bool isInvoke() const
Return true if a InvokeInst is enclosed.
Definition: CallSite.h:90
Intrinsic::ID getIntrinsicID() const
Return the intrinsic ID of this intrinsic.
Definition: IntrinsicInst.h:51
PHINode * CreatePHI(Type *Ty, unsigned NumReservedValues, const Twine &Name="")
Definition: IRBuilder.h:1658
bool isPtrOrPtrVectorTy() const
Return true if this is a pointer type or a vector of pointer types.
Definition: Type.h:224
static IntegerType * get(LLVMContext &C, unsigned NumBits)
This static method is the primary way of constructing an IntegerType.
Definition: Type.cpp:240
Type * getSequentialElementType() const
Definition: Type.h:355
Iterator for intrusive lists based on ilist_node.
unsigned getNumOperands() const
Definition: User.h:176
See the file comment.
Definition: ValueMap.h:86
This is the shared class of boolean and integer constants.
Definition: Constants.h:84
iterator end()
Definition: BasicBlock.h:254
unsigned getScalarSizeInBits() const LLVM_READONLY
If this is a vector type, return the getPrimitiveSizeInBits value for the element type...
Definition: Type.cpp:130
bool removeUnreachableBlocks(Function &F, LazyValueInfo *LVI=nullptr)
Remove all blocks that can not be reached from the function&#39;s entry.
Definition: Local.cpp:1730
CallingConv::ID getCallingConv() const
getCallingConv()/setCallingConv(CC) - These method get and set the calling convention of this functio...
Definition: Function.h:194
IterTy arg_begin() const
Definition: CallSite.h:571
Value * CreateIntCast(Value *V, Type *DestTy, bool isSigned, const Twine &Name="")
Definition: IRBuilder.h:1511
This is a &#39;vector&#39; (really, a variable-sized array), optimized for the case when the array is small...
Definition: SmallVector.h:864
Module.h This file contains the declarations for the Module class.
Value * CreateInsertElement(Value *Vec, Value *NewElt, Value *Idx, const Twine &Name="")
Definition: IRBuilder.h:1726
Provides information about what library functions are available for the current target.
unsigned getABITypeAlignment(Type *Ty) const
Returns the minimum ABI-required alignment for the specified type.
Definition: DataLayout.cpp:682
bool isAggregateType() const
Return true if the type is an aggregate type.
Definition: Type.h:255
signed less than
Definition: InstrTypes.h:882
TerminatorInst * SplitBlockAndInsertIfThen(Value *Cond, Instruction *SplitBefore, bool Unreachable, MDNode *BranchWeights=nullptr, DominatorTree *DT=nullptr, LoopInfo *LI=nullptr)
Split the containing block at the specified instruction - everything before SplitBefore stays in the ...
CHAIN = SC CHAIN, Imm128 - System call.
ConstantInt * getInt32(uint32_t C)
Get a constant 32-bit value.
Definition: IRBuilder.h:308
static IntegerType * getIntNTy(LLVMContext &C, unsigned N)
Definition: Type.cpp:180
StringRef str()
Return a StringRef for the vector contents.
Definition: raw_ostream.h:514
static GCRegistry::Add< StatepointGC > D("statepoint-example", "an example strategy for statepoint")
This class wraps the llvm.memcpy intrinsic.
static Constant * get(Type *Ty, uint64_t V, bool isSigned=false)
If Ty is a vector type, return a Constant with a splat of the given value.
Definition: Constants.cpp:560
void appendToGlobalCtors(Module &M, Function *F, int Priority, Constant *Data=nullptr)
Append F to the list of global ctors of module M with the given Priority.
Definition: ModuleUtils.cpp:84
unsigned getNumIncomingValues() const
Return the number of incoming edges.
static const size_t kNumberOfAccessSizes
static GlobalVariable * createPrivateNonConstGlobalForString(Module &M, StringRef Str)
Create a non-const global initialized with the given string.
static cl::opt< bool > ClKeepGoing("msan-keep-going", cl::desc("keep going after reporting a UMR"), cl::Hidden, cl::init(false))
raw_ostream & dbgs()
dbgs() - This returns a reference to a raw_ostream for debugging messages.
Definition: Debug.cpp:132
unsigned getVectorNumElements() const
Definition: DerivedTypes.h:462
signed less or equal
Definition: InstrTypes.h:883
Class to represent vector types.
Definition: DerivedTypes.h:393
const Module * getModule() const
Return the module owning the function this instruction belongs to or nullptr it the function does not...
Definition: Instruction.cpp:57
Class for arbitrary precision integers.
Definition: APInt.h:69
IntegerType * getInt8Ty()
Fetch the type representing an 8-bit integer.
Definition: IRBuilder.h:338
Value * CreateShl(Value *LHS, Value *RHS, const Twine &Name="", bool HasNUW=false, bool HasNSW=false)
Definition: IRBuilder.h:1012
const Value * getFalseValue() const
Value * CreatePointerCast(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1488
uint64_t getTypeSizeInBits(Type *Ty) const
Size examples:
Definition: DataLayout.h:530
static cl::opt< ITMode > IT(cl::desc("IT block support"), cl::Hidden, cl::init(DefaultIT), cl::ZeroOrMore, cl::values(clEnumValN(DefaultIT, "arm-default-it", "Generate IT block based on arch"), clEnumValN(RestrictedIT, "arm-restrict-it", "Disallow deprecated IT based on ARMv8"), clEnumValN(NoRestrictedIT, "arm-no-restrict-it", "Allow IT blocks based on ARMv7")))
uint64_t getTypeAllocSize(Type *Ty) const
Returns the offset in bytes between successive objects of the specified type, including alignment pad...
Definition: DataLayout.h:403
Function * getCalledFunction() const
Return the function called, or null if this is an indirect function invocation.
void removeAttributes(unsigned i, const AttrBuilder &Attrs)
removes the attributes from the list of attributes.
Definition: Function.cpp:395
Predicate getPredicate() const
Return the predicate for this instruction.
Definition: InstrTypes.h:927
bool isInlineAsm() const
Check if this call is an inline asm statement.
static cl::opt< bool > ClHandleICmpExact("msan-handle-icmp-exact", cl::desc("exact handling of relational integer ICmp"), cl::Hidden, cl::init(false))
CallInst * CreateMemCpy(Value *Dst, Value *Src, uint64_t Size, unsigned Align, bool isVolatile=false, MDNode *TBAATag=nullptr, MDNode *TBAAStructTag=nullptr, MDNode *ScopeTag=nullptr, MDNode *NoAliasTag=nullptr)
Create and insert a memcpy between the specified pointers.
Definition: IRBuilder.h:423
unsigned getAlignment() const
Return the alignment of the access that is being performed.
Definition: Instructions.h:226
LLVM_NODISCARD bool empty() const
Definition: SmallVector.h:61
Value * getArgOperand(unsigned i) const
getArgOperand/setArgOperand - Return/set the i-th call argument.
StringRef getName() const
Return a constant reference to the value&#39;s name.
Definition: Value.cpp:220
#define I(x, y, z)
Definition: MD5.cpp:58
The C convention as implemented on Windows/x86-64 and AArch64.
Definition: CallingConv.h:154
static const MemoryMapParams Linux_X86_64_MemoryMapParams
static ArrayType * get(Type *ElementType, uint64_t NumElements)
This static method is the primary way to construct an ArrayType.
Definition: Type.cpp:568
LLVM_NODISCARD std::enable_if<!is_simple_type< Y >::value, typename cast_retty< X, const Y >::ret_type >::type dyn_cast(const Y &Val)
Definition: Casting.h:323
This instruction extracts a single (scalar) element from a VectorType value.
void maybeMarkSanitizerLibraryCallNoBuiltin(CallInst *CI, const TargetLibraryInfo *TLI)
Given a CallInst, check if it calls a string function known to CodeGen, and mark it with NoBuiltin if...
Definition: Local.cpp:2220
Value * getReturnValue() const
Convenience accessor. Returns null if there is no return value.
static InlineAsm * get(FunctionType *Ty, StringRef AsmString, StringRef Constraints, bool hasSideEffects, bool isAlignStack=false, AsmDialect asmDialect=AD_ATT)
InlineAsm::get - Return the specified uniqued inline asm string.
Definition: InlineAsm.cpp:43
static cl::opt< bool > ClCheckConstantShadow("msan-check-constant-shadow", cl::desc("Insert checks for constant shadow values"), cl::Hidden, cl::init(false))
static const MemoryMapParams Linux_MIPS64_MemoryMapParams
Value * CreateAnd(Value *LHS, Value *RHS, const Twine &Name="")
Definition: IRBuilder.h:1067
Value * CreatePtrToInt(Value *V, Type *DestTy, const Twine &Name="")
Definition: IRBuilder.h:1440
static const unsigned kOriginSize
static const MemoryMapParams FreeBSD_I386_MemoryMapParams
bool isFPOrFPVectorTy() const
Return true if this is a FP type or a vector of FP.
Definition: Type.h:185
iterator_range< df_iterator< T > > depth_first(const T &G)
This represents the llvm.va_copy intrinsic.
bool isArrayAllocation() const
Return true if there is an allocation size parameter to the allocation instruction that is not 1...
size_type count(const KeyT &Val) const
Return 1 if the specified key is in the map, 0 otherwise.
Definition: ValueMap.h:154
assert(ImpDefSCC.getReg()==AMDGPU::SCC &&ImpDefSCC.isDef())
LoadInst * CreateAlignedLoad(Value *Ptr, unsigned Align, const char *Name)
Definition: IRBuilder.h:1186
static cl::opt< int > ClInstrumentationWithCallThreshold("msan-instrumentation-with-call-threshold", cl::desc("If the function being instrumented requires more than " "this number of checks and origin stores, use callbacks instead of " "inline checks (-1 means never use callbacks)."), cl::Hidden, cl::init(3500))
unsigned getPrimitiveSizeInBits() const LLVM_READONLY
Return the basic size of this type if it is a primitive type.
Definition: Type.cpp:115
void setSuccessOrdering(AtomicOrdering Ordering)
Sets the success ordering constraint of this cmpxchg instruction.
Definition: Instructions.h:573
ArrayRef< unsigned > getIndices() const
Module * getParent()
Get the module that this global value is contained inside of...
Definition: GlobalValue.h:556
LLVM Value Representation.
Definition: Value.h:73
uint64_t getTypeStoreSize(Type *Ty) const
Returns the maximum number of bytes that may be overwritten by storing the specified type...
Definition: DataLayout.h:386
FunctionType * getFunctionType() const
Definition: CallSite.h:320
constexpr char Size[]
Key for Kernel::Arg::Metadata::mSize.
static VectorType * get(Type *ElementType, unsigned NumElements)
This static method is the primary way to construct an VectorType.
Definition: Type.cpp:593
#define LLVM_FALLTHROUGH
LLVM_FALLTHROUGH - Mark fallthrough cases in switch statements.
Definition: Compiler.h:235
std::underlying_type< E >::type Mask()
Get a bitmask with 1s in all places up to the high-order bit of E&#39;s largest value.
Definition: BitmaskEnum.h:81
unsigned getArgumentNo(Value::const_user_iterator I) const
Given a value use iterator, returns the argument that corresponds to it.
Definition: CallSite.h:199
BasicBlock::iterator GetInsertPoint() const
Definition: IRBuilder.h:123
bool doesNotAccessMemory() const
Determine if the call does not access memory.
FunctionPass * createMemorySanitizerPass(int TrackOrigins=0, bool Recover=false)
Value * CreateLShr(Value *LHS, Value *RHS, const Twine &Name="", bool isExact=false)
Definition: IRBuilder.h:1031
static cl::opt< bool > ClPoisonUndef("msan-poison-undef", cl::desc("poison undef temps"), cl::Hidden, cl::init(true))
#define DEBUG(X)
Definition: Debug.h:118
ConstantInt * getInt8(uint8_t C)
Get a constant 8-bit value.
Definition: IRBuilder.h:298
StringRef - Represent a constant reference to a string, i.e.
Definition: StringRef.h:49
Predicate getSwappedPredicate() const
For example, EQ->EQ, SLE->SGE, ULT->UGT, OEQ->OEQ, ULE->UGE, OLT->OGT, etc.
Definition: InstrTypes.h:967
static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams
Type * getArrayElementType() const
Definition: Type.h:362
Value * CreateInsertValue(Value *Agg, Value *Val, ArrayRef< unsigned > Idxs, const Twine &Name="")
Definition: IRBuilder.h:1763
bool isBigEndian() const
Definition: DataLayout.h:217
static IntegerType * getInt8Ty(LLVMContext &C)
Definition: Type.cpp:174
static const char *const kMsanModuleCtorName
static Constant * get(ArrayRef< Constant *> V)
Definition: Constants.cpp:984
iterator_range< arg_iterator > args()
Definition: Function.h:621
signed greater or equal
Definition: InstrTypes.h:881
bool isArrayTy() const
True if this is an instance of ArrayType.
Definition: Type.h:218
Type * getContainedType(unsigned i) const
This method is used to implement the type iterator (defined at the end of the file).
Definition: Type.h:330
A wrapper class for inspecting calls to intrinsic functions.
Definition: IntrinsicInst.h:44
const BasicBlock * getParent() const
Definition: Instruction.h:66
an instruction to allocate memory on the stack
Definition: Instructions.h:60
This instruction inserts a struct field of array element value into an aggregate value.
CallInst * CreateCall(Value *Callee, ArrayRef< Value *> Args=None, const Twine &Name="", MDNode *FPMathTag=nullptr)
Definition: IRBuilder.h:1663