LLVM  4.0.0
CGSCCPassManager.h
Go to the documentation of this file.
1 //===- CGSCCPassManager.h - Call graph pass management ----------*- C++ -*-===//
2 //
3 // The LLVM Compiler Infrastructure
4 //
5 // This file is distributed under the University of Illinois Open Source
6 // License. See LICENSE.TXT for details.
7 //
8 //===----------------------------------------------------------------------===//
9 /// \file
10 ///
11 /// This header provides classes for managing passes over SCCs of the call
12 /// graph. These passes form an important component of LLVM's interprocedural
13 /// optimizations. Because they operate on the SCCs of the call graph, and they
14 /// traverse the graph in post-order, they can effectively do pair-wise
15 /// interprocedural optimizations for all call edges in the program while
16 /// incrementally refining it and improving the context of these pair-wise
17 /// optimizations. At each call site edge, the callee has already been
18 /// optimized as much as is possible. This in turn allows very accurate
19 /// analysis of it for IPO.
20 ///
21 /// A secondary more general goal is to be able to isolate optimization on
22 /// unrelated parts of the IR module. This is useful to ensure our
23 /// optimizations are principled and don't miss oportunities where refinement
24 /// of one part of the module influence transformations in another part of the
25 /// module. But this is also useful if we want to parallelize the optimizations
26 /// across common large module graph shapes which tend to be very wide and have
27 /// large regions of unrelated cliques.
28 ///
29 /// To satisfy these goals, we use the LazyCallGraph which provides two graphs
30 /// nested inside each other (and built lazily from the bottom-up): the call
31 /// graph proper, and a reference graph. The reference graph is super set of
32 /// the call graph and is a conservative approximation of what could through
33 /// scalar or CGSCC transforms *become* the call graph. Using this allows us to
34 /// ensure we optimize functions prior to them being introduced into the call
35 /// graph by devirtualization or other technique, and thus ensures that
36 /// subsequent pair-wise interprocedural optimizations observe the optimized
37 /// form of these functions. The (potentially transitive) reference
38 /// reachability used by the reference graph is a conservative approximation
39 /// that still allows us to have independent regions of the graph.
40 ///
41 /// FIXME: There is one major drawback of the reference graph: in its naive
42 /// form it is quadratic because it contains a distinct edge for each
43 /// (potentially indirect) reference, even if are all through some common
44 /// global table of function pointers. This can be fixed in a number of ways
45 /// that essentially preserve enough of the normalization. While it isn't
46 /// expected to completely preclude the usability of this, it will need to be
47 /// addressed.
48 ///
49 ///
50 /// All of these issues are made substantially more complex in the face of
51 /// mutations to the call graph while optimization passes are being run. When
52 /// mutations to the call graph occur we want to achieve two different things:
53 ///
54 /// - We need to update the call graph in-flight and invalidate analyses
55 /// cached on entities in the graph. Because of the cache-based analysis
56 /// design of the pass manager, it is essential to have stable identities for
57 /// the elements of the IR that passes traverse, and to invalidate any
58 /// analyses cached on these elements as the mutations take place.
59 ///
60 /// - We want to preserve the incremental and post-order traversal of the
61 /// graph even as it is refined and mutated. This means we want optimization
62 /// to observe the most refined form of the call graph and to do so in
63 /// post-order.
64 ///
65 /// To address this, the CGSCC manager uses both worklists that can be expanded
66 /// by passes which transform the IR, and provides invalidation tests to skip
67 /// entries that become dead. This extra data is provided to every SCC pass so
68 /// that it can carefully update the manager's traversal as the call graph
69 /// mutates.
70 ///
71 /// We also provide support for running function passes within the CGSCC walk,
72 /// and there we provide automatic update of the call graph including of the
73 /// pass manager to reflect call graph changes that fall out naturally as part
74 /// of scalar transformations.
75 ///
76 /// The patterns used to ensure the goals of post-order visitation of the fully
77 /// refined graph:
78 ///
79 /// 1) Sink toward the "bottom" as the graph is refined. This means that any
80 /// iteration continues in some valid post-order sequence after the mutation
81 /// has altered the structure.
82 ///
83 /// 2) Enqueue in post-order, including the current entity. If the current
84 /// entity's shape changes, it and everything after it in post-order needs
85 /// to be visited to observe that shape.
86 ///
87 //===----------------------------------------------------------------------===//
88 
89 #ifndef LLVM_ANALYSIS_CGSCCPASSMANAGER_H
90 #define LLVM_ANALYSIS_CGSCCPASSMANAGER_H
91 
94 #include "llvm/IR/CallSite.h"
95 #include "llvm/IR/InstIterator.h"
96 #include "llvm/IR/PassManager.h"
97 #include "llvm/IR/ValueHandle.h"
98 
99 namespace llvm {
100 
101 struct CGSCCUpdateResult;
102 
103 /// Extern template declaration for the analysis set for this IR unit.
104 extern template class AllAnalysesOn<LazyCallGraph::SCC>;
105 
106 extern template class AnalysisManager<LazyCallGraph::SCC, LazyCallGraph &>;
107 /// \brief The CGSCC analysis manager.
108 ///
109 /// See the documentation for the AnalysisManager template for detail
110 /// documentation. This typedef serves as a convenient way to refer to this
111 /// construct in the adaptors and proxies used to integrate this into the larger
112 /// pass manager infrastructure.
113 typedef AnalysisManager<LazyCallGraph::SCC, LazyCallGraph &>
115 
116 // Explicit specialization and instantiation declarations for the pass manager.
117 // See the comments on the definition of the specialization for details on how
118 // it differs from the primary template.
119 template <>
122  CGSCCUpdateResult &>::run(LazyCallGraph::SCC &InitialC,
127 
128 /// \brief The CGSCC pass manager.
129 ///
130 /// See the documentation for the PassManager template for details. It runs
131 /// a sequence of SCC passes over each SCC that the manager is run over. This
132 /// typedef serves as a convenient way to refer to this construct.
136 
137 /// An explicit specialization of the require analysis template pass.
138 template <typename AnalysisT>
141  : PassInfoMixin<RequireAnalysisPass<AnalysisT, LazyCallGraph::SCC,
142  CGSCCAnalysisManager, LazyCallGraph &,
143  CGSCCUpdateResult &>> {
146  (void)AM.template getResult<AnalysisT>(C, CG);
147  return PreservedAnalyses::all();
148  }
149 };
150 
151 /// A proxy from a \c CGSCCAnalysisManager to a \c Module.
152 typedef InnerAnalysisManagerProxy<CGSCCAnalysisManager, Module>
154 
155 /// We need a specialized result for the \c CGSCCAnalysisManagerModuleProxy so
156 /// it can have access to the call graph in order to walk all the SCCs when
157 /// invalidating things.
159 public:
161  : InnerAM(&InnerAM), G(&G) {}
162 
163  /// \brief Accessor for the analysis manager.
164  CGSCCAnalysisManager &getManager() { return *InnerAM; }
165 
166  /// \brief Handler for invalidation of the Module.
167  ///
168  /// If the proxy analysis itself is preserved, then we assume that the set of
169  /// SCCs in the Module hasn't changed. Thus any pointers to SCCs in the
170  /// CGSCCAnalysisManager are still valid, and we don't need to call \c clear
171  /// on the CGSCCAnalysisManager.
172  ///
173  /// Regardless of whether this analysis is marked as preserved, all of the
174  /// analyses in the \c CGSCCAnalysisManager are potentially invalidated based
175  /// on the set of preserved analyses.
176  bool invalidate(Module &M, const PreservedAnalyses &PA,
178 
179 private:
180  CGSCCAnalysisManager *InnerAM;
181  LazyCallGraph *G;
182 };
183 
184 /// Provide a specialized run method for the \c CGSCCAnalysisManagerModuleProxy
185 /// so it can pass the lazy call graph to the result.
186 template <>
189 
190 // Ensure the \c CGSCCAnalysisManagerModuleProxy is provided as an extern
191 // template.
193 
194 extern template class OuterAnalysisManagerProxy<
196 /// A proxy from a \c ModuleAnalysisManager to an \c SCC.
198  LazyCallGraph &>
200 
201 /// Support structure for SCC passes to communicate updates the call graph back
202 /// to the CGSCC pass manager infrsatructure.
203 ///
204 /// The CGSCC pass manager runs SCC passes which are allowed to update the call
205 /// graph and SCC structures. This means the structure the pass manager works
206 /// on is mutating underneath it. In order to support that, there needs to be
207 /// careful communication about the precise nature and ramifications of these
208 /// updates to the pass management infrastructure.
209 ///
210 /// All SCC passes will have to accept a reference to the management layer's
211 /// update result struct and use it to reflect the results of any CG updates
212 /// performed.
213 ///
214 /// Passes which do not change the call graph structure in any way can just
215 /// ignore this argument to their run method.
217  /// Worklist of the RefSCCs queued for processing.
218  ///
219  /// When a pass refines the graph and creates new RefSCCs or causes them to
220  /// have a different shape or set of component SCCs it should add the RefSCCs
221  /// to this worklist so that we visit them in the refined form.
222  ///
223  /// This worklist is in reverse post-order, as we pop off the back in order
224  /// to observe RefSCCs in post-order. When adding RefSCCs, clients should add
225  /// them in reverse post-order.
227 
228  /// Worklist of the SCCs queued for processing.
229  ///
230  /// When a pass refines the graph and creates new SCCs or causes them to have
231  /// a different shape or set of component functions it should add the SCCs to
232  /// this worklist so that we visit them in the refined form.
233  ///
234  /// Note that if the SCCs are part of a RefSCC that is added to the \c
235  /// RCWorklist, they don't need to be added here as visiting the RefSCC will
236  /// be sufficient to re-visit the SCCs within it.
237  ///
238  /// This worklist is in reverse post-order, as we pop off the back in order
239  /// to observe SCCs in post-order. When adding SCCs, clients should add them
240  /// in reverse post-order.
242 
243  /// The set of invalidated RefSCCs which should be skipped if they are found
244  /// in \c RCWorklist.
245  ///
246  /// This is used to quickly prune out RefSCCs when they get deleted and
247  /// happen to already be on the worklist. We use this primarily to avoid
248  /// scanning the list and removing entries from it.
250 
251  /// The set of invalidated SCCs which should be skipped if they are found
252  /// in \c CWorklist.
253  ///
254  /// This is used to quickly prune out SCCs when they get deleted and happen
255  /// to already be on the worklist. We use this primarily to avoid scanning
256  /// the list and removing entries from it.
258 
259  /// If non-null, the updated current \c RefSCC being processed.
260  ///
261  /// This is set when a graph refinement takes place an the "current" point in
262  /// the graph moves "down" or earlier in the post-order walk. This will often
263  /// cause the "current" RefSCC to be a newly created RefSCC object and the
264  /// old one to be added to the above worklist. When that happens, this
265  /// pointer is non-null and can be used to continue processing the "top" of
266  /// the post-order walk.
268 
269  /// If non-null, the updated current \c SCC being processed.
270  ///
271  /// This is set when a graph refinement takes place an the "current" point in
272  /// the graph moves "down" or earlier in the post-order walk. This will often
273  /// cause the "current" SCC to be a newly created SCC object and the old one
274  /// to be added to the above worklist. When that happens, this pointer is
275  /// non-null and can be used to continue processing the "top" of the
276  /// post-order walk.
278 };
279 
280 /// \brief The core module pass which does a post-order walk of the SCCs and
281 /// runs a CGSCC pass over each one.
282 ///
283 /// Designed to allow composition of a CGSCCPass(Manager) and
284 /// a ModulePassManager. Note that this pass must be run with a module analysis
285 /// manager as it uses the LazyCallGraph analysis. It will also run the
286 /// \c CGSCCAnalysisManagerModuleProxy analysis prior to running the CGSCC
287 /// pass over the module to enable a \c FunctionAnalysisManager to be used
288 /// within this run safely.
289 template <typename CGSCCPassT>
291  : public PassInfoMixin<ModuleToPostOrderCGSCCPassAdaptor<CGSCCPassT>> {
292 public:
293  explicit ModuleToPostOrderCGSCCPassAdaptor(CGSCCPassT Pass, bool DebugLogging = false)
294  : Pass(std::move(Pass)), DebugLogging(DebugLogging) {}
295  // We have to explicitly define all the special member functions because MSVC
296  // refuses to generate them.
299  : Pass(Arg.Pass), DebugLogging(Arg.DebugLogging) {}
301  : Pass(std::move(Arg.Pass)), DebugLogging(Arg.DebugLogging) {}
304  using std::swap;
305  swap(LHS.Pass, RHS.Pass);
306  swap(LHS.DebugLogging, RHS.DebugLogging);
307  }
310  swap(*this, RHS);
311  return *this;
312  }
313 
314  /// \brief Runs the CGSCC pass across every SCC in the module.
316  // Setup the CGSCC analysis manager from its proxy.
317  CGSCCAnalysisManager &CGAM =
318  AM.getResult<CGSCCAnalysisManagerModuleProxy>(M).getManager();
319 
320  // Get the call graph for this module.
322 
323  // We keep worklists to allow us to push more work onto the pass manager as
324  // the passes are run.
327 
328  // Keep sets for invalidated SCCs and RefSCCs that should be skipped when
329  // iterating off the worklists.
332 
333  CGSCCUpdateResult UR = {RCWorklist, CWorklist, InvalidRefSCCSet,
334  InvalidSCCSet, nullptr, nullptr};
335 
337  for (auto RCI = CG.postorder_ref_scc_begin(),
338  RCE = CG.postorder_ref_scc_end();
339  RCI != RCE;) {
340  assert(RCWorklist.empty() &&
341  "Should always start with an empty RefSCC worklist");
342  // The postorder_ref_sccs range we are walking is lazily constructed, so
343  // we only push the first one onto the worklist. The worklist allows us
344  // to capture *new* RefSCCs created during transformations.
345  //
346  // We really want to form RefSCCs lazily because that makes them cheaper
347  // to update as the program is simplified and allows us to have greater
348  // cache locality as forming a RefSCC touches all the parts of all the
349  // functions within that RefSCC.
350  //
351  // We also eagerly increment the iterator to the next position because
352  // the CGSCC passes below may delete the current RefSCC.
353  RCWorklist.insert(&*RCI++);
354 
355  do {
356  LazyCallGraph::RefSCC *RC = RCWorklist.pop_back_val();
357  if (InvalidRefSCCSet.count(RC)) {
358  if (DebugLogging)
359  dbgs() << "Skipping an invalid RefSCC...\n";
360  continue;
361  }
362 
363  assert(CWorklist.empty() &&
364  "Should always start with an empty SCC worklist");
365 
366  if (DebugLogging)
367  dbgs() << "Running an SCC pass across the RefSCC: " << *RC << "\n";
368 
369  // Push the initial SCCs in reverse post-order as we'll pop off the the
370  // back and so see this in post-order.
371  for (LazyCallGraph::SCC &C : reverse(*RC))
372  CWorklist.insert(&C);
373 
374  do {
375  LazyCallGraph::SCC *C = CWorklist.pop_back_val();
376  // Due to call graph mutations, we may have invalid SCCs or SCCs from
377  // other RefSCCs in the worklist. The invalid ones are dead and the
378  // other RefSCCs should be queued above, so we just need to skip both
379  // scenarios here.
380  if (InvalidSCCSet.count(C)) {
381  if (DebugLogging)
382  dbgs() << "Skipping an invalid SCC...\n";
383  continue;
384  }
385  if (&C->getOuterRefSCC() != RC) {
386  if (DebugLogging)
387  dbgs() << "Skipping an SCC that is now part of some other "
388  "RefSCC...\n";
389  continue;
390  }
391 
392  do {
393  // Check that we didn't miss any update scenario.
394  assert(!InvalidSCCSet.count(C) && "Processing an invalid SCC!");
395  assert(C->begin() != C->end() && "Cannot have an empty SCC!");
396  assert(&C->getOuterRefSCC() == RC &&
397  "Processing an SCC in a different RefSCC!");
398 
399  UR.UpdatedRC = nullptr;
400  UR.UpdatedC = nullptr;
401  PreservedAnalyses PassPA = Pass.run(*C, CGAM, CG, UR);
402 
403  // We handle invalidating the CGSCC analysis manager's information
404  // for the (potentially updated) SCC here. Note that any other SCCs
405  // whose structure has changed should have been invalidated by
406  // whatever was updating the call graph. This SCC gets invalidated
407  // late as it contains the nodes that were actively being
408  // processed.
409  CGAM.invalidate(*(UR.UpdatedC ? UR.UpdatedC : C), PassPA);
410 
411  // Then intersect the preserved set so that invalidation of module
412  // analyses will eventually occur when the module pass completes.
413  PA.intersect(std::move(PassPA));
414 
415  // The pass may have restructured the call graph and refined the
416  // current SCC and/or RefSCC. We need to update our current SCC and
417  // RefSCC pointers to follow these. Also, when the current SCC is
418  // refined, re-run the SCC pass over the newly refined SCC in order
419  // to observe the most precise SCC model available. This inherently
420  // cannot cycle excessively as it only happens when we split SCCs
421  // apart, at most converging on a DAG of single nodes.
422  // FIXME: If we ever start having RefSCC passes, we'll want to
423  // iterate there too.
424  RC = UR.UpdatedRC ? UR.UpdatedRC : RC;
425  C = UR.UpdatedC ? UR.UpdatedC : C;
426  if (DebugLogging && UR.UpdatedC)
427  dbgs() << "Re-running SCC passes after a refinement of the "
428  "current SCC: "
429  << *UR.UpdatedC << "\n";
430 
431  // Note that both `C` and `RC` may at this point refer to deleted,
432  // invalid SCC and RefSCCs respectively. But we will short circuit
433  // the processing when we check them in the loop above.
434  } while (UR.UpdatedC);
435 
436  } while (!CWorklist.empty());
437  } while (!RCWorklist.empty());
438  }
439 
440  // By definition we preserve the call garph, all SCC analyses, and the
441  // analysis proxies by handling them above and in any nested pass managers.
446  return PA;
447  }
448 
449 private:
450  CGSCCPassT Pass;
451  bool DebugLogging;
452 };
453 
454 /// \brief A function to deduce a function pass type and wrap it in the
455 /// templated adaptor.
456 template <typename CGSCCPassT>
457 ModuleToPostOrderCGSCCPassAdaptor<CGSCCPassT>
458 createModuleToPostOrderCGSCCPassAdaptor(CGSCCPassT Pass, bool DebugLogging = false) {
459  return ModuleToPostOrderCGSCCPassAdaptor<CGSCCPassT>(std::move(Pass), DebugLogging);
460 }
461 
462 /// A proxy from a \c FunctionAnalysisManager to an \c SCC.
463 ///
464 /// When a module pass runs and triggers invalidation, both the CGSCC and
465 /// Function analysis manager proxies on the module get an invalidation event.
466 /// We don't want to fully duplicate responsibility for most of the
467 /// invalidation logic. Instead, this layer is only responsible for SCC-local
468 /// invalidation events. We work with the module's FunctionAnalysisManager to
469 /// invalidate function analyses.
471  : public AnalysisInfoMixin<FunctionAnalysisManagerCGSCCProxy> {
472 public:
473  class Result {
474  public:
475  explicit Result(FunctionAnalysisManager &FAM) : FAM(&FAM) {}
476 
477  /// \brief Accessor for the analysis manager.
479 
482 
483  private:
485  };
486 
487  /// Computes the \c FunctionAnalysisManager and stores it in the result proxy.
489 
490 private:
492  static AnalysisKey Key;
493 };
494 
495 extern template class OuterAnalysisManagerProxy<CGSCCAnalysisManager, Function>;
496 /// A proxy from a \c CGSCCAnalysisManager to a \c Function.
497 typedef OuterAnalysisManagerProxy<CGSCCAnalysisManager, Function>
499 
500 /// Helper to update the call graph after running a function pass.
501 ///
502 /// Function passes can only mutate the call graph in specific ways. This
503 /// routine provides a helper that updates the call graph in those ways
504 /// including returning whether any changes were made and populating a CG
505 /// update result struct for the overall CGSCC walk.
508  CGSCCAnalysisManager &AM, CGSCCUpdateResult &UR, bool DebugLogging = false);
509 
510 /// \brief Adaptor that maps from a SCC to its functions.
511 ///
512 /// Designed to allow composition of a FunctionPass(Manager) and
513 /// a CGSCCPassManager. Note that if this pass is constructed with a pointer
514 /// to a \c CGSCCAnalysisManager it will run the
515 /// \c FunctionAnalysisManagerCGSCCProxy analysis prior to running the function
516 /// pass over the SCC to enable a \c FunctionAnalysisManager to be used
517 /// within this run safely.
518 template <typename FunctionPassT>
521 public:
522  explicit CGSCCToFunctionPassAdaptor(FunctionPassT Pass, bool DebugLogging = false)
523  : Pass(std::move(Pass)), DebugLogging(DebugLogging) {}
524  // We have to explicitly define all the special member functions because MSVC
525  // refuses to generate them.
527  : Pass(Arg.Pass), DebugLogging(Arg.DebugLogging) {}
529  : Pass(std::move(Arg.Pass)), DebugLogging(Arg.DebugLogging) {}
532  using std::swap;
533  swap(LHS.Pass, RHS.Pass);
534  swap(LHS.DebugLogging, RHS.DebugLogging);
535  }
537  swap(*this, RHS);
538  return *this;
539  }
540 
541  /// \brief Runs the function pass across every function in the module.
543  LazyCallGraph &CG, CGSCCUpdateResult &UR) {
544  // Setup the function analysis manager from its proxy.
546  AM.getResult<FunctionAnalysisManagerCGSCCProxy>(C, CG).getManager();
547 
549  for (LazyCallGraph::Node &N : C)
550  Nodes.push_back(&N);
551 
552  // The SCC may get split while we are optimizing functions due to deleting
553  // edges. If this happens, the current SCC can shift, so keep track of
554  // a pointer we can overwrite.
555  LazyCallGraph::SCC *CurrentC = &C;
556 
557  if (DebugLogging)
558  dbgs() << "Running function passes across an SCC: " << C << "\n";
559 
561  for (LazyCallGraph::Node *N : Nodes) {
562  // Skip nodes from other SCCs. These may have been split out during
563  // processing. We'll eventually visit those SCCs and pick up the nodes
564  // there.
565  if (CG.lookupSCC(*N) != CurrentC)
566  continue;
567 
568  PreservedAnalyses PassPA = Pass.run(N->getFunction(), FAM);
569 
570  // We know that the function pass couldn't have invalidated any other
571  // function's analyses (that's the contract of a function pass), so
572  // directly handle the function analysis manager's invalidation here.
573  FAM.invalidate(N->getFunction(), PassPA);
574 
575  // Then intersect the preserved set so that invalidation of module
576  // analyses will eventually occur when the module pass completes.
577  PA.intersect(std::move(PassPA));
578 
579  // Update the call graph based on this function pass. This may also
580  // update the current SCC to point to a smaller, more refined SCC.
582  CG, *CurrentC, *N, AM, UR, DebugLogging);
583  assert(CG.lookupSCC(*N) == CurrentC &&
584  "Current SCC not updated to the SCC containing the current node!");
585  }
586 
587  // By definition we preserve the proxy. And we preserve all analyses on
588  // Functions. This precludes *any* invalidation of function analyses by the
589  // proxy, but that's OK because we've taken care to invalidate analyses in
590  // the function analysis manager incrementally above.
593 
594  // We've also ensured that we updated the call graph along the way.
596 
597  return PA;
598  }
599 
600 private:
601  FunctionPassT Pass;
602  bool DebugLogging;
603 };
604 
605 /// \brief A function to deduce a function pass type and wrap it in the
606 /// templated adaptor.
607 template <typename FunctionPassT>
608 CGSCCToFunctionPassAdaptor<FunctionPassT>
609 createCGSCCToFunctionPassAdaptor(FunctionPassT Pass, bool DebugLogging = false) {
610  return CGSCCToFunctionPassAdaptor<FunctionPassT>(std::move(Pass),
611  DebugLogging);
612 }
613 
614 /// A helper that repeats an SCC pass each time an indirect call is refined to
615 /// a direct call by that pass.
616 ///
617 /// While the CGSCC pass manager works to re-visit SCCs and RefSCCs as they
618 /// change shape, we may also want to repeat an SCC pass if it simply refines
619 /// an indirect call to a direct call, even if doing so does not alter the
620 /// shape of the graph. Note that this only pertains to direct calls to
621 /// functions where IPO across the SCC may be able to compute more precise
622 /// results. For intrinsics, we assume scalar optimizations already can fully
623 /// reason about them.
624 ///
625 /// This repetition has the potential to be very large however, as each one
626 /// might refine a single call site. As a consequence, in practice we use an
627 /// upper bound on the number of repetitions to limit things.
628 template <typename PassT>
630  : public PassInfoMixin<DevirtSCCRepeatedPass<PassT>> {
631 public:
632  explicit DevirtSCCRepeatedPass(PassT Pass, int MaxIterations,
633  bool DebugLogging = false)
634  : Pass(std::move(Pass)), MaxIterations(MaxIterations),
635  DebugLogging(DebugLogging) {}
636 
637  /// Runs the wrapped pass up to \c MaxIterations on the SCC, iterating
638  /// whenever an indirect call is refined.
640  LazyCallGraph &CG, CGSCCUpdateResult &UR) {
642 
643  // The SCC may be refined while we are running passes over it, so set up
644  // a pointer that we can update.
645  LazyCallGraph::SCC *C = &InitialC;
646 
647  // Collect value handles for all of the indirect call sites.
648  SmallVector<WeakVH, 8> CallHandles;
649 
650  // Struct to track the counts of direct and indirect calls in each function
651  // of the SCC.
652  struct CallCount {
653  int Direct;
654  int Indirect;
655  };
656 
657  // Put value handles on all of the indirect calls and return the number of
658  // direct calls for each function in the SCC.
659  auto ScanSCC = [](LazyCallGraph::SCC &C,
660  SmallVectorImpl<WeakVH> &CallHandles) {
661  assert(CallHandles.empty() && "Must start with a clear set of handles.");
662 
663  SmallVector<CallCount, 4> CallCounts;
664  for (LazyCallGraph::Node &N : C) {
665  CallCounts.push_back({0, 0});
666  CallCount &Count = CallCounts.back();
667  for (Instruction &I : instructions(N.getFunction()))
668  if (auto CS = CallSite(&I)) {
669  if (CS.getCalledFunction()) {
670  ++Count.Direct;
671  } else {
672  ++Count.Indirect;
673  CallHandles.push_back(WeakVH(&I));
674  }
675  }
676  }
677 
678  return CallCounts;
679  };
680 
681  // Populate the initial call handles and get the initial call counts.
682  auto CallCounts = ScanSCC(*C, CallHandles);
683 
684  for (int Iteration = 0;; ++Iteration) {
685  PreservedAnalyses PassPA = Pass.run(*C, AM, CG, UR);
686 
687  // If the SCC structure has changed, bail immediately and let the outer
688  // CGSCC layer handle any iteration to reflect the refined structure.
689  if (UR.UpdatedC && UR.UpdatedC != C) {
690  PA.intersect(std::move(PassPA));
691  break;
692  }
693 
694  // Check that we didn't miss any update scenario.
695  assert(!UR.InvalidatedSCCs.count(C) && "Processing an invalid SCC!");
696  assert(C->begin() != C->end() && "Cannot have an empty SCC!");
697  assert((int)CallCounts.size() == C->size() &&
698  "Cannot have changed the size of the SCC!");
699 
700  // Check whether any of the handles were devirtualized.
701  auto IsDevirtualizedHandle = [&](WeakVH &CallH) {
702  if (!CallH)
703  return false;
704  auto CS = CallSite(CallH);
705  if (!CS)
706  return false;
707 
708  // If the call is still indirect, leave it alone.
709  Function *F = CS.getCalledFunction();
710  if (!F)
711  return false;
712 
713  if (DebugLogging)
714  dbgs() << "Found devirutalized call from "
715  << CS.getParent()->getParent()->getName() << " to "
716  << F->getName() << "\n";
717 
718  // We now have a direct call where previously we had an indirect call,
719  // so iterate to process this devirtualization site.
720  return true;
721  };
722  bool Devirt = any_of(CallHandles, IsDevirtualizedHandle);
723 
724  // Rescan to build up a new set of handles and count how many direct
725  // calls remain. If we decide to iterate, this also sets up the input to
726  // the next iteration.
727  CallHandles.clear();
728  auto NewCallCounts = ScanSCC(*C, CallHandles);
729 
730  // If we haven't found an explicit devirtualization already see if we
731  // have decreased the number of indirect calls and increased the number
732  // of direct calls for any function in the SCC. This can be fooled by all
733  // manner of transformations such as DCE and other things, but seems to
734  // work well in practice.
735  if (!Devirt)
736  for (int i = 0, Size = C->size(); i < Size; ++i)
737  if (CallCounts[i].Indirect > NewCallCounts[i].Indirect &&
738  CallCounts[i].Direct < NewCallCounts[i].Direct) {
739  Devirt = true;
740  break;
741  }
742 
743  if (!Devirt) {
744  PA.intersect(std::move(PassPA));
745  break;
746  }
747 
748  // Otherwise, if we've already hit our max, we're done.
749  if (Iteration >= MaxIterations) {
750  if (DebugLogging)
751  dbgs() << "Found another devirtualization after hitting the max "
752  "number of repetitions ("
753  << MaxIterations << ") on SCC: " << *C << "\n";
754  PA.intersect(std::move(PassPA));
755  break;
756  }
757 
758  if (DebugLogging)
759  dbgs() << "Repeating an SCC pass after finding a devirtualization in: "
760  << *C << "\n";
761 
762  // Move over the new call counts in preparation for iterating.
763  CallCounts = std::move(NewCallCounts);
764 
765  // Update the analysis manager with each run and intersect the total set
766  // of preserved analyses so we're ready to iterate.
767  AM.invalidate(*C, PassPA);
768  PA.intersect(std::move(PassPA));
769  }
770 
771  // Note that we don't add any preserved entries here unlike a more normal
772  // "pass manager" because we only handle invalidation *between* iterations,
773  // not after the last iteration.
774  return PA;
775  }
776 
777 private:
778  PassT Pass;
779  int MaxIterations;
780  bool DebugLogging;
781 };
782 
783 /// \brief A function to deduce a function pass type and wrap it in the
784 /// templated adaptor.
785 template <typename PassT>
786 DevirtSCCRepeatedPass<PassT>
788  bool DebugLogging = false) {
789  return DevirtSCCRepeatedPass<PassT>(std::move(Pass), MaxIterations,
790  DebugLogging);
791 }
792 }
793 
794 #endif
Pass interface - Implemented by all 'passes'.
Definition: Pass.h:81
This file provides a priority worklist.
SCC * lookupSCC(Node &N) const
Lookup a function's SCC in the graph.
Result run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM, LazyCallGraph &)
Computes the FunctionAnalysisManager and stores it in the result proxy.
ModuleToPostOrderCGSCCPassAdaptor(ModuleToPostOrderCGSCCPassAdaptor &&Arg)
PreservedAnalyses run(LazyCallGraph::SCC &InitialC, CGSCCAnalysisManager &AM, LazyCallGraph &CG, CGSCCUpdateResult &UR)
Runs the wrapped pass up to MaxIterations on the SCC, iterating whenever an indirect call is refined...
bool insert(const T &X)
Insert a new element into the PriorityWorklist.
size_t i
DevirtSCCRepeatedPass< PassT > createDevirtSCCRepeatedPass(PassT Pass, int MaxIterations, bool DebugLogging=false)
A function to deduce a function pass type and wrap it in the templated adaptor.
SmallPriorityWorklist< LazyCallGraph::RefSCC *, 1 > & RCWorklist
Worklist of the RefSCCs queued for processing.
void intersect(const PreservedAnalyses &Arg)
Intersect this set with another in place.
Definition: PassManager.h:171
A Module instance is used to store all the information related to an LLVM module. ...
Definition: Module.h:52
aarch64 AArch64 CCMP Pass
iterator begin() const
CGSCCToFunctionPassAdaptor & operator=(CGSCCToFunctionPassAdaptor RHS)
Implements a lazy call graph analysis and related passes for the new pass manager.
size_type count(PtrType Ptr) const
count - Return 1 if the specified pointer is in the set, 0 otherwise.
Definition: SmallPtrSet.h:380
OuterAnalysisManagerProxy< CGSCCAnalysisManager, Function > CGSCCAnalysisManagerFunctionProxy
A proxy from a CGSCCAnalysisManager to a Function.
ModuleToPostOrderCGSCCPassAdaptor & operator=(ModuleToPostOrderCGSCCPassAdaptor RHS)
A utility pass template to force an analysis result to be available.
Definition: PassManager.h:1183
Result(CGSCCAnalysisManager &InnerAM, LazyCallGraph &G)
A proxy from a FunctionAnalysisManager to an SCC.
Support structure for SCC passes to communicate updates the call graph back to the CGSCC pass manager...
OuterAnalysisManagerProxy< ModuleAnalysisManager, LazyCallGraph::SCC, LazyCallGraph & > ModuleAnalysisManagerCGSCCProxy
A proxy from a ModuleAnalysisManager to an SCC.
StringRef getName() const
Return a constant reference to the value's name.
Definition: Value.cpp:191
A templated base class for SmallPtrSet which provides the typesafe interface that is common across al...
Definition: SmallPtrSet.h:345
PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM, LazyCallGraph &CG, CGSCCUpdateResult &UR)
Runs the function pass across every function in the module.
ModuleToPostOrderCGSCCPassAdaptor(CGSCCPassT Pass, bool DebugLogging=false)
DevirtSCCRepeatedPass(PassT Pass, int MaxIterations, bool DebugLogging=false)
AnalysisManager< LazyCallGraph::SCC, LazyCallGraph & > CGSCCAnalysisManager
The CGSCC analysis manager.
LazyCallGraph::SCC * UpdatedC
If non-null, the updated current SCC being processed.
This class consists of common code factored out of the SmallVector class to reduce code duplication b...
Definition: APFloat.h:32
postorder_ref_scc_iterator postorder_ref_scc_begin()
ModuleToPostOrderCGSCCPassAdaptor(const ModuleToPostOrderCGSCCPassAdaptor &Arg)
LLVM_NODISCARD bool empty() const
Definition: SmallVector.h:60
auto reverse(ContainerTy &&C, typename std::enable_if< has_rbegin< ContainerTy >::value >::type *=nullptr) -> decltype(make_range(C.rbegin(), C.rend()))
Definition: STLExtras.h:241
CGSCCToFunctionPassAdaptor(const CGSCCToFunctionPassAdaptor &Arg)
A RefSCC of the call graph.
#define F(x, y, z)
Definition: MD5.cpp:51
Value handle that is nullable, but tries to track the Value.
Definition: ValueHandle.h:134
A CRTP mix-in to automatically provide informational APIs needed for passes.
Definition: PassManager.h:311
A lazily constructed view of the call graph of a module.
Result run(IRUnitT &IR, AnalysisManager< IRUnitT, ExtraArgTs...> &AM, ExtraArgTs...)
Run the analysis pass and create our proxy result object.
Definition: PassManager.h:965
static cl::opt< unsigned > MaxIterations("max-cg-scc-iterations", cl::ReallyHidden, cl::init(4))
A set of analyses that are preserved following a run of a transformation pass.
Definition: PassManager.h:107
PassT::Result & getResult(IRUnitT &IR, ExtraArgTs...ExtraArgs)
Get the result of an analysis pass for a given IR unit.
Definition: PassManager.h:653
CGSCCToFunctionPassAdaptor< FunctionPassT > createCGSCCToFunctionPassAdaptor(FunctionPassT Pass, bool DebugLogging=false)
A function to deduce a function pass type and wrap it in the templated adaptor.
The core module pass which does a post-order walk of the SCCs and runs a CGSCC pass over each one...
SmallPtrSetImpl< LazyCallGraph::RefSCC * > & InvalidatedRefSCCs
The set of invalidated RefSCCs which should be skipped if they are found in RCWorklist.
SmallPtrSetImpl< LazyCallGraph::SCC * > & InvalidatedSCCs
The set of invalidated SCCs which should be skipped if they are found in CWorklist.
A CRTP mix-in that provides informational APIs needed for analysis passes.
Definition: PassManager.h:328
friend void swap(ModuleToPostOrderCGSCCPassAdaptor &LHS, ModuleToPostOrderCGSCCPassAdaptor &RHS)
LazyCallGraph::RefSCC * UpdatedRC
If non-null, the updated current RefSCC being processed.
bool any_of(R &&Range, UnaryPredicate P)
Provide wrappers to std::any_of which take ranges instead of having to pass begin/end explicitly...
Definition: STLExtras.h:743
friend void swap(CGSCCToFunctionPassAdaptor &LHS, CGSCCToFunctionPassAdaptor &RHS)
FunctionAnalysisManager & getManager()
Accessor for the analysis manager.
RefSCC & getOuterRefSCC() const
A node in the call graph.
A version of PriorityWorklist that selects small size optimized data structures for the vector and ma...
static PreservedAnalyses all()
Construct a special preserved set that preserves all passes.
Definition: PassManager.h:113
SmallPriorityWorklist< LazyCallGraph::SCC *, 1 > & CWorklist
Worklist of the SCCs queued for processing.
PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM)
Runs the CGSCC pass across every SCC in the module.
CGSCCAnalysisManager & getManager()
Accessor for the analysis manager.
SmallPtrSet - This class implements a set which is optimized for holding SmallSize or less elements...
Definition: SmallPtrSet.h:425
An analysis over an "inner" IR unit that provides access to an analysis manager over a "outer" IR uni...
Definition: PassManager.h:1017
PassManager< LazyCallGraph::SCC, CGSCCAnalysisManager, LazyCallGraph &, CGSCCUpdateResult & > CGSCCPassManager
The CGSCC pass manager.
This is a 'vector' (really, a variable-sized array), optimized for the case when the array is small...
Definition: SmallVector.h:843
const DataFlowGraph & G
Definition: RDFGraph.cpp:206
void invalidate(IRUnitT &IR)
Invalidate a specific analysis pass for an IR module.
Definition: PassManager.h:722
static GCRegistry::Add< ShadowStackGC > C("shadow-stack","Very portable GC for uncooperative code generators")
raw_ostream & dbgs()
dbgs() - This returns a reference to a raw_ostream for debugging messages.
Definition: Debug.cpp:132
void swap(llvm::BitVector &LHS, llvm::BitVector &RHS)
Implement std::swap in terms of BitVector swap.
Definition: BitVector.h:586
A helper that repeats an SCC pass each time an indirect call is refined to a direct call by that pass...
CGSCCToFunctionPassAdaptor(CGSCCToFunctionPassAdaptor &&Arg)
Adaptor that maps from a SCC to its functions.
bool invalidate(IRUnitT &IR, const PreservedAnalyses &PA, typename AnalysisManager< IRUnitT, ExtraArgTs...>::Invalidator &Inv)
Handler for invalidation of the outer IR unit, IRUnitT.
Manages a sequence of passes over a particular unit of IR.
Definition: PassManager.h:389
iterator end() const
void preserveSet()
Mark an analysis set as preserved.
Definition: PassManager.h:135
#define I(x, y, z)
Definition: MD5.cpp:54
#define N
AnalysisManager< Module > ModuleAnalysisManager
Convenience typedef for the Module analysis manager.
Definition: PassManager.h:882
void preserve()
Mark an analysis as preserved.
Definition: PassManager.h:120
An analysis pass which computes the call graph for a module.
PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM, LazyCallGraph &CG, CGSCCUpdateResult &)
API to communicate dependencies between analyses during invalidation.
Definition: PassManager.h:525
bool invalidate(LazyCallGraph::SCC &C, const PreservedAnalyses &PA, CGSCCAnalysisManager::Invalidator &Inv)
ModuleToPostOrderCGSCCPassAdaptor< CGSCCPassT > createModuleToPostOrderCGSCCPassAdaptor(CGSCCPassT Pass, bool DebugLogging=false)
A function to deduce a function pass type and wrap it in the templated adaptor.
assert(ImpDefSCC.getReg()==AMDGPU::SCC &&ImpDefSCC.isDef())
This templated class represents "all analyses that operate over \<a particular IR unit\>" (e...
Definition: PassManager.h:361
An SCC of the call graph.
inst_range instructions(Function *F)
Definition: InstIterator.h:132
A container for analyses that lazily runs them and caches their results.
This header defines various interfaces for pass management in LLVM.
postorder_ref_scc_iterator postorder_ref_scc_end()
InnerAnalysisManagerProxy< CGSCCAnalysisManager, Module > CGSCCAnalysisManagerModuleProxy
A proxy from a CGSCCAnalysisManager to a Module.
LazyCallGraph::SCC & updateCGAndAnalysisManagerForFunctionPass(LazyCallGraph &G, LazyCallGraph::SCC &C, LazyCallGraph::Node &N, CGSCCAnalysisManager &AM, CGSCCUpdateResult &UR, bool DebugLogging=false)
Helper to update the call graph after running a function pass.
A special type used by analysis passes to provide an address that identifies that particular analysis...
Definition: PassManager.h:64
bool empty() const
Determine if the PriorityWorklist is empty or not.
An analysis over an "outer" IR unit that provides access to an analysis manager over an "inner" IR un...
Definition: PassManager.h:905
CGSCCToFunctionPassAdaptor(FunctionPassT Pass, bool DebugLogging=false)