VPlanEVLTailFolding.cpp File Reference

This file implements the VPlan-to-VPlan transforms related to explicit vector length (EVL) tail folding support. More...

#include "LoopVectorizationPlanner.h"
#include "VPlan.h"
#include "VPlanCFG.h"
#include "VPlanHelpers.h"
#include "VPlanPatternMatch.h"
#include "VPlanTransforms.h"
#include "VPlanUtils.h"
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/IR/Intrinsics.h"

Go to the source code of this file.

Classes
struct	RemoveMask_match< Op0_t, Op1_t >

Functions
template<typename Op0_t, typename Op1_t>
static RemoveMask_match< Op0_t, Op1_t >	m_RemoveMask (const Op0_t &In, Op1_t &Out)
	Match a specific mask In, or a combination of it (logical-and In, Out).
static std::optional< Intrinsic::ID >	getVPDivRemIntrinsic (Intrinsic::ID IntrID)
static VPRecipeBase *	optimizeMaskToEVL (VPValue *HeaderMask, VPRecipeBase &CurRecipe, VPValue &EVL)
	Try to optimize a CurRecipe masked by HeaderMask to a corresponding EVL-based recipe without the header mask.
static void	fixupVFUsersForEVL (VPlan &Plan, VPValue &EVL)
	After replacing the canonical IV with a EVL-based IV, fixup recipes that use VF to use the EVL instead to avoid incorrect updates on the penultimate iteration.

Detailed Description

This file implements the VPlan-to-VPlan transforms related to explicit vector length (EVL) tail folding support.

Definition in file VPlanEVLTailFolding.cpp.

Function Documentation

fixupVFUsersForEVL()

void fixupVFUsersForEVL ( VPlan & Plan, VPValue & EVL )

static

After replacing the canonical IV with a EVL-based IV, fixup recipes that use VF to use the EVL instead to avoid incorrect updates on the penultimate iteration.

Definition at line 348 of file VPlanEVLTailFolding.cpp.

References llvm::all_of(), llvm::any_of(), assert(), llvm::VPBlockUtils::blocksOnly(), llvm::VPBuilder::createScalarZExtOrTrunc(), llvm::VPRegionBlock::getCanonicalIV(), llvm::VPlan::getContext(), llvm::VPValue::getDefiningRecipe(), llvm::VPRegionBlock::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPRegionBlock::getHeaderMask(), llvm::Type::getInt32Ty(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::VPlan::getMiddleBlock(), llvm::VPlan::getOrAddLiveIn(), llvm::VPRecipeBase::getParent(), llvm::VPRegionBlock::getPreheaderVPBB(), llvm::VPValue::getScalarType(), llvm::ConstantInt::getSigned(), llvm::VPBuilder::getToInsertAfter(), llvm::VPlan::getTrue(), llvm::DebugLoc::getUnknown(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::getVF(), llvm::VPlan::getVFxUF(), llvm::CmpInst::ICMP_ULT, llvm::VPRecipeBase::insertBefore(), llvm::isa(), llvm::IsaPred, llvm::PatternMatch::m_c_Add(), llvm::PatternMatchHelpers::m_CombineOr(), llvm::PatternMatchHelpers::m_Isa(), llvm::PatternMatch::m_Specific(), llvm::VPlanPatternMatch::m_VPInstruction(), llvm::VPlanPatternMatch::m_VPValue(), llvm::PatternMatch::match(), llvm::PatternMatch::match_fn(), llvm::VPValue::replaceAllUsesWith(), llvm::VPValue::replaceUsesWithIf(), llvm::VPInstruction::StepVector, llvm::VPValue::users(), llvm::V1, and llvm::vp_depth_first_deep().

Referenced by llvm::VPlanTransforms::addExplicitVectorLength().

getVPDivRemIntrinsic()

std::optional< Intrinsic::ID > getVPDivRemIntrinsic ( Intrinsic::ID IntrID )

static

Definition at line 87 of file VPlanEVLTailFolding.cpp.

Referenced by optimizeMaskToEVL().

m_RemoveMask()

template<typename Op0_t, typename Op1_t>

RemoveMask_match< Op0_t, Op1_t > m_RemoveMask ( const Op0_t & In, Op1_t & Out )

inlinestatic

Match a specific mask In, or a combination of it (logical-and In, Out).

Returns the remaining part Out if so, or nullptr otherwise.

Definition at line 82 of file VPlanEVLTailFolding.cpp.

Referenced by optimizeMaskToEVL().

optimizeMaskToEVL()

VPRecipeBase * optimizeMaskToEVL ( VPValue * HeaderMask, VPRecipeBase & CurRecipe, VPValue & EVL )

static

Try to optimize a CurRecipe masked by HeaderMask to a corresponding EVL-based recipe without the header mask.

Returns nullptr if no EVL-based recipe could be created. HeaderMask Header Mask. CurRecipe Recipe to be transform. EVL The explicit vector length parameter of vector-predication intrinsics.

Adjust any end pointers so that they point to the end of EVL lanes not VF.

Definition at line 109 of file VPlanEVLTailFolding.cpp.

References llvm::cast(), llvm::VPBuilder::createScalarZExtOrTrunc(), DL, llvm::dyn_cast(), llvm::VPlan::getConstantInt(), llvm::VPRecipeBase::getDebugLoc(), llvm::VPIRFlags::getDefaultFlags(), llvm::VPRecipeBase::getParent(), llvm::VPBlockBase::getPlan(), llvm::VPlan::getPoison(), llvm::VPValue::getScalarType(), llvm::VPlan::getTrue(), llvm::DebugLoc::getUnknown(), llvm::VPlan::getVF(), getVPDivRemIntrinsic(), llvm::VPDef::getVPSingleValue(), llvm::VPRecipeBase::insertBefore(), LHS, llvm::VPlanPatternMatch::m_c_BinaryOr(), llvm::PatternMatch::m_Intrinsic(), llvm::VPlanPatternMatch::m_LastActiveLane(), llvm::PatternMatch::m_LogicalAnd(), llvm::PatternMatch::m_MaskedLoad(), llvm::PatternMatch::m_MaskedStore(), m_RemoveMask(), llvm::VPlanPatternMatch::m_Reverse(), llvm::PatternMatch::m_SelectLike(), llvm::PatternMatch::m_Specific(), llvm::PatternMatch::m_TruncOrSelf(), llvm::VPlanPatternMatch::m_VecEndPtr(), llvm::VPlanPatternMatch::m_VPValue(), llvm::PatternMatch::match(), Poison, llvm::Reverse, and RHS.

Referenced by llvm::VPlanTransforms::optimizeEVLMasks().