VPlanEVLTailFolding.cpp File Reference

This file implements the VPlan-to-VPlan transforms related to explicit vector length (EVL) tail folding support. More...

#include "LoopVectorizationPlanner.h"
#include "VPlan.h"
#include "VPlanCFG.h"
#include "VPlanHelpers.h"
#include "VPlanPatternMatch.h"
#include "VPlanTransforms.h"
#include "VPlanUtils.h"
#include "llvm/ADT/SetVector.h"
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/IR/Intrinsics.h"

Go to the source code of this file.

Functions
static std::optional< Intrinsic::ID >	getVPDivRemIntrinsic (Intrinsic::ID IntrID)
static VPRecipeBase *	optimizeMaskToEVL (VPValue *HeaderMask, VPRecipeBase &CurRecipe, VPValue &EVL)
	Try to optimize a CurRecipe masked by HeaderMask to a corresponding EVL-based recipe without the header mask.
static bool	optimizeExpressionRecipeToEVL (VPValue *HeaderMask, VPRecipeBase &CurRecipe, VPValue &EVL, SmallVector< VPRecipeBase * > &OldRecipes)
static void	fixupVFUsersForEVL (VPlan &Plan, VPValue &EVL)
	After replacing the canonical IV with a EVL-based IV, fixup recipes that use VF to use the EVL instead to avoid incorrect updates on the penultimate iteration.

Detailed Description

This file implements the VPlan-to-VPlan transforms related to explicit vector length (EVL) tail folding support.

Definition in file VPlanEVLTailFolding.cpp.

Function Documentation

fixupVFUsersForEVL()

void fixupVFUsersForEVL ( VPlan & Plan, VPValue & EVL )

static

After replacing the canonical IV with a EVL-based IV, fixup recipes that use VF to use the EVL instead to avoid incorrect updates on the penultimate iteration.

Definition at line 367 of file VPlanEVLTailFolding.cpp.

References llvm::all_of(), llvm::any_of(), assert(), llvm::VPBlockUtils::blocksOnly(), llvm::VPBuilder::createScalarZExtOrTrunc(), llvm::VPlan::getContext(), llvm::VPValue::getDefiningRecipe(), llvm::VPRegionBlock::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPRegionBlock::getHeaderMask(), llvm::Type::getInt32Ty(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::VPlan::getMiddleBlock(), llvm::VPlan::getOrAddLiveIn(), llvm::VPRecipeBase::getParent(), llvm::VPRegionBlock::getPreheaderVPBB(), llvm::VPValue::getScalarType(), llvm::ConstantInt::getSigned(), llvm::VPBuilder::getToInsertAfter(), llvm::VPlan::getTrue(), llvm::DebugLoc::getUnknown(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::getVF(), llvm::VPlan::getVFxUF(), llvm::CmpInst::ICMP_ULT, Imm, llvm::VPRecipeBase::insertBefore(), llvm::isa(), llvm::IsaPred, llvm::VPlanPatternMatch::m_VPInstruction(), llvm::VPlanPatternMatch::m_VPValue(), llvm::PatternMatch::match(), llvm::VPValue::replaceAllUsesWith(), llvm::VPValue::replaceUsesWithIf(), llvm::VPInstruction::StepVector, llvm::V1, and llvm::vp_depth_first_deep().

Referenced by llvm::VPlanTransforms::addExplicitVectorLength().

getVPDivRemIntrinsic()

std::optional< Intrinsic::ID > getVPDivRemIntrinsic ( Intrinsic::ID IntrID )

static

Definition at line 64 of file VPlanEVLTailFolding.cpp.

Referenced by optimizeMaskToEVL().

optimizeExpressionRecipeToEVL()

bool optimizeExpressionRecipeToEVL ( VPValue * HeaderMask, VPRecipeBase & CurRecipe, VPValue & EVL, SmallVector< VPRecipeBase * > & OldRecipes )

static

Definition at line 224 of file VPlanEVLTailFolding.cpp.

References llvm::SmallVectorTemplateCommon< T, typename >::back(), llvm::cast_if_present(), llvm::dyn_cast(), llvm::from_range, optimizeMaskToEVL(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), and llvm::replace().

Referenced by llvm::VPlanTransforms::optimizeEVLMasks().

optimizeMaskToEVL()

VPRecipeBase * optimizeMaskToEVL ( VPValue * HeaderMask, VPRecipeBase & CurRecipe, VPValue & EVL )

static

Try to optimize a CurRecipe masked by HeaderMask to a corresponding EVL-based recipe without the header mask.

Returns nullptr if no EVL-based recipe could be created. HeaderMask Header Mask. CurRecipe Recipe to be transform. EVL The explicit vector length parameter of vector-predication intrinsics.

Adjust any end pointers so that they point to the end of EVL lanes not VF.

Definition at line 86 of file VPlanEVLTailFolding.cpp.

References llvm::cast(), llvm::VPBuilder::createScalarZExtOrTrunc(), DL, llvm::dyn_cast(), llvm::VPlan::getConstantInt(), llvm::VPRecipeBase::getDebugLoc(), llvm::VPIRFlags::getDefaultFlags(), llvm::VPRecipeBase::getParent(), llvm::VPBlockBase::getPlan(), llvm::VPlan::getPoison(), llvm::VPValue::getScalarType(), llvm::VPlan::getTrue(), llvm::DebugLoc::getUnknown(), llvm::VPlan::getVF(), getVPDivRemIntrinsic(), llvm::VPDef::getVPSingleValue(), llvm::VPRecipeBase::insertBefore(), LHS, llvm::VPlanPatternMatch::m_c_BinaryOr(), llvm::PatternMatch::m_Intrinsic(), llvm::VPlanPatternMatch::m_LastActiveLane(), llvm::PatternMatch::m_LogicalAnd(), llvm::PatternMatch::m_MaskedLoad(), llvm::PatternMatch::m_MaskedStore(), llvm::VPlanPatternMatch::m_RemoveMask(), llvm::VPlanPatternMatch::m_Reverse(), llvm::PatternMatch::m_SelectLike(), llvm::PatternMatch::m_Specific(), llvm::PatternMatch::m_TruncOrSelf(), llvm::VPlanPatternMatch::m_VecEndPtr(), llvm::VPlanPatternMatch::m_VPValue(), llvm::PatternMatch::match(), Poison, llvm::Reverse, and RHS.

Referenced by llvm::VPlanTransforms::optimizeEVLMasks(), and optimizeExpressionRecipeToEVL().