VPlanLowering.cpp File Reference

This file implements VPlan-to-VPlan lowering transformations, which prepare an optimized VPlan for execution. More...

#include "LoopVectorizationPlanner.h"
#include "VPlan.h"
#include "VPlanAnalysis.h"
#include "VPlanCFG.h"
#include "VPlanDominatorTree.h"
#include "VPlanHelpers.h"
#include "VPlanPatternMatch.h"
#include "VPlanTransforms.h"
#include "VPlanUtils.h"
#include "llvm/ADT/APInt.h"
#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/Analysis/IVDescriptors.h"
#include "llvm/Analysis/LoopAccessAnalysis.h"
#include "llvm/Analysis/ScalarEvolutionPatternMatch.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/MDBuilder.h"
#include "llvm/IR/Metadata.h"
#include "llvm/Support/Casting.h"
#include "llvm/Support/TypeSize.h"
#include "llvm/Transforms/Utils/ScalarEvolutionExpander.h"

Go to the source code of this file.

Functions
static VPActiveLaneMaskPHIRecipe *	addVPLaneMaskPhiAndUpdateExitBranch (VPlan &Plan)
static void	expandVPWidenIntOrFpInduction (VPWidenIntOrFpInductionRecipe *WidenIVR)
	Expand a VPWidenIntOrFpInduction into executable recipes, for the initial value, phi and backedge value.
static void	expandVPWidenPointerInduction (VPWidenPointerInductionRecipe *R)
	Expand a VPWidenPointerInductionRecipe into executable recipes, for the initial value, phi and backedge value.
static void	expandVPDerivedIV (VPDerivedIVRecipe *R)
	Expand a VPDerivedIVRecipe into executable recipes.

Detailed Description

This file implements VPlan-to-VPlan lowering transformations, which prepare an optimized VPlan for execution.

Definition in file VPlanLowering.cpp.

Function Documentation

addVPLaneMaskPhiAndUpdateExitBranch()

VPActiveLaneMaskPHIRecipe * addVPLaneMaskPhiAndUpdateExitBranch ( VPlan & Plan )

static

Definition at line 142 of file VPlanLowering.cpp.

References llvm::VPInstruction::ActiveLaneMask, llvm::VPInstruction::BranchOnCond, llvm::VPInstruction::CanonicalIVIncrementForPart, llvm::VPRegionBlock::clearCanonicalIVNUW(), DL, llvm::VPRecipeBase::eraseFromParent(), llvm::VPRegionBlock::getCanonicalIVType(), llvm::VPlan::getConstantInt(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::VPRegionBlock::getOrCreateCanonicalIVIncrement(), llvm::VPBasicBlock::getTerminator(), llvm::VPlan::getTripCount(), llvm::DebugLoc::getUnknown(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::getVectorPreheader(), llvm::VPlan::getVF(), and llvm::VPlan::getZero().

Referenced by llvm::VPlanTransforms::materializeHeaderMask().

expandVPDerivedIV()

void expandVPDerivedIV ( VPDerivedIVRecipe * R )

static

Expand a VPDerivedIVRecipe into executable recipes.

Definition at line 412 of file VPlanLowering.cpp.

References assert(), llvm::FMul, llvm::DebugLoc::getCompilerGenerated(), llvm::FPMathOperator::getFastMathFlags(), llvm::Operator::getOpcode(), llvm::VPValue::getScalarType(), llvm::InductionDescriptor::IK_FpInduction, llvm::InductionDescriptor::IK_IntInduction, llvm::InductionDescriptor::IK_NoInduction, llvm::InductionDescriptor::IK_PtrInduction, llvm::Type::isFloatingPointTy(), llvm::Type::isIntegerTy(), and llvm_unreachable.

Referenced by llvm::VPlanTransforms::convertToConcreteRecipes().

expandVPWidenIntOrFpInduction()

void expandVPWidenIntOrFpInduction ( VPWidenIntOrFpInductionRecipe * WidenIVR )

static

Expand a VPWidenIntOrFpInduction into executable recipes, for the initial value, phi and backedge value.

In the following example:

vector.ph: Successor(s): vector loop

<x1> vector loop: { vector.body: WIDEN-INDUCTION i = phi start, step, vf ... EMIT branch-on-count ... No successors }

WIDEN-INDUCTION will get expanded to:

vector.ph: ... vp<induction.start> = ... vp<induction.increment> = ...

Successor(s): vector loop

<x1> vector loop: { vector.body: ir<i> = WIDEN-PHI vp<induction.start>, vp<vec.ind.next> ... vp<vec.ind.next> = add ir<i>, vp<induction.increment> EMIT branch-on-count ... No successors }

Definition at line 262 of file VPlanLowering.cpp.

References assert(), llvm::VPInstruction::Broadcast, llvm::VPBuilder::createWidenPhi(), DL, llvm::IntegerType::get(), llvm::VPlan::getContext(), llvm::VPRecipeBase::getDebugLoc(), llvm::VPValue::getDefiningRecipe(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::VPWidenInductionRecipe::getInductionDescriptor(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::VPWidenIntOrFpInductionRecipe::getLastUnrolledPartOperand(), llvm::VPRecipeBase::getParent(), llvm::VPBlockBase::getPlan(), llvm::Type::getScalarSizeInBits(), llvm::VPRecipeValue::getScalarType(), llvm::VPValue::getScalarType(), llvm::VPWidenIntOrFpInductionRecipe::getSplatVFValue(), llvm::VPWidenInductionRecipe::getStartValue(), llvm::VPWidenInductionRecipe::getStepValue(), llvm::VPBasicBlock::getTerminator(), llvm::DebugLoc::getUnknown(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::getVectorPreheader(), llvm::VPWidenInductionRecipe::getVFValue(), llvm::InductionDescriptor::IK_IntInduction, llvm::Type::isFloatingPointTy(), llvm::Type::isIntegerTy(), llvm::Next, llvm::VPValue::replaceAllUsesWith(), and llvm::VPInstruction::StepVector.

Referenced by llvm::VPlanTransforms::convertToConcreteRecipes().

expandVPWidenPointerInduction()

void expandVPWidenPointerInduction ( VPWidenPointerInductionRecipe * R )

static

Expand a VPWidenPointerInductionRecipe into executable recipes, for the initial value, phi and backedge value.

In the following example:

<x1> vector loop: { vector.body: EMIT ir<ptr.iv> = WIDEN-POINTER-INDUCTION start, step, vf ... EMIT branch-on-count ... }

WIDEN-POINTER-INDUCTION will get expanded to:

<x1> vector loop: { vector.body: EMIT-SCALAR pointer.phi = phi start, ptr.ind EMIT mul = mul stepvector, step EMIT vector.gep = wide-ptradd pointer.phi, mul ... EMIT ptr.ind = ptradd pointer.phi, vf EMIT branch-on-count ... }

Definition at line 371 of file VPlanLowering.cpp.

References llvm::VPPhiAccessors::addIncoming(), assert(), DL, llvm::VPBlockBase::getExitingBasicBlock(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::VPValue::getScalarType(), llvm::VPBasicBlock::getTerminator(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::hasScalableVF(), llvm::InductionDescriptor::IK_PtrInduction, llvm::Offset, and llvm::VPInstruction::StepVector.

Referenced by llvm::VPlanTransforms::convertToConcreteRecipes().