InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Inheritance diagram for llvm::InnerLoopVectorizer:

[legend]

Public Member Functions
	InnerLoopVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetTransformInfo TTI, AssumptionCache AC, ElementCount VecWidth, unsigned UnrollFactor, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo PSI, GeneratedRTChecks &RTChecks, VPlan &Plan)

virtual	~InnerLoopVectorizer ()=default

virtual BasicBlock *	createVectorizedLoopSkeleton ()
	Creates a basic block for the scalar preheader.

void	fixVectorizedLoop (VPTransformState &State)
	Fix the vectorized code, taking care of header phi's, and more.

void	fixNonInductionPHIs (VPTransformState &State)
	Fix the non-induction PHIs in `Plan`.

Value *	getTripCount () const
	Returns the original loop trip count.

void	setTripCount (Value *TC)
	Used to set the trip count after ILV's construction and after the preheader block has been executed.

BasicBlock *	getAdditionalBypassBlock () const
	Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.

Protected Member Functions
BasicBlock *	createScalarPreheader (StringRef Prefix)
	Create and return a new IR basic block for the scalar preheader whose name is prefixed with `Prefix`.

virtual void	printDebugTracesAtStart ()
	Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

virtual void	printDebugTracesAtEnd ()

Protected Attributes
Loop *	OrigLoop
	The original loop.

PredicatedScalarEvolution &	PSE
	A wrapper around ScalarEvolution used to add runtime SCEV checks.

LoopInfo *	LI
	Loop Info.

DominatorTree *	DT
	Dominator Tree.

const TargetTransformInfo *	TTI
	Target Transform Info.

AssumptionCache *	AC
	Assumption Cache.

ElementCount	VF
	The vectorization SIMD factor to use.

unsigned	UF
	The vectorization unroll factor to use.

IRBuilder	Builder
	The builder that we use.

BasicBlock *	LoopVectorPreHeader = nullptr
	The vector-loop preheader.

Value *	TripCount = nullptr
	Trip count of the original loop.

LoopVectorizationCostModel *	Cost
	The profitablity analysis.

BlockFrequencyInfo *	BFI
	BFI and PSI are used to check for profile guided size optimizations.

ProfileSummaryInfo *	PSI

GeneratedRTChecks &	RTChecks
	Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

BasicBlock *	AdditionalBypassBlock = nullptr
	The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.

VPlan &	Plan

VPBasicBlock *	VectorPHVPBB
	The vector preheader block of `Plan`, used as target for check blocks introduced during skeleton creation.

Friends
class	LoopVectorizationPlanner

Detailed Description

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

It inserts an epilogue loop for handling loops that don't have iteration counts that are known to be a multiple of the vectorization factor.
It handles the code generation for reduction variables.
Scalarization (implementation using scalars) of un-vectorizable instructions. InnerLoopVectorizer does not perform any vectorization-legality checks, and relies on the caller to check for the different legality aspects. The InnerLoopVectorizer relies on the LoopVectorizationLegality class to provide information about the induction and reduction variables that were found to a given vectorization factor.

Definition at line 498 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

llvm::InnerLoopVectorizer::InnerLoopVectorizer	(	Loop *	OrigLoop,
		PredicatedScalarEvolution &	PSE,
		LoopInfo *	LI,
		DominatorTree *	DT,
		const TargetTransformInfo *	TTI,
		AssumptionCache *	AC,
		ElementCount	VecWidth,
		unsigned	UnrollFactor,
		LoopVectorizationCostModel *	CM,
		BlockFrequencyInfo *	BFI,
		ProfileSummaryInfo *	PSI,
		GeneratedRTChecks &	RTChecks,
		VPlan &	Plan
	)

inline

Definition at line 500 of file LoopVectorize.cpp.

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )

virtualdefault

Member Function Documentation

◆ createScalarPreheader()

BasicBlock * InnerLoopVectorizer::createScalarPreheader ( StringRef Prefix )

protected

Create and return a new IR basic block for the scalar preheader whose name is prefixed with Prefix.

Definition at line 2371 of file LoopVectorize.cpp.

References assert(), DT, llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::BasicBlock::getTerminator(), llvm::LoopBase< BlockT, LoopT >::getUniqueLatchExitBlock(), llvm::ElementCount::isVector(), LI, LoopVectorPreHeader, OrigLoop, llvm::SplitBlock(), and VF.

Referenced by createVectorizedLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::createVectorizedLoopSkeleton(), and llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton().

◆ createVectorizedLoopSkeleton()

BasicBlock * InnerLoopVectorizer::createVectorizedLoopSkeleton ( )

virtual

Creates a basic block for the scalar preheader.

Both EpilogueVectorizerMainLoop and EpilogueVectorizerEpilogueLoop overwrite the method to create additional blocks and checks needed for epilogue vectorization.

Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.

Definition at line 2423 of file LoopVectorize.cpp.

References createScalarPreheader(), and llvm::BasicBlock::getSinglePredecessor().

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ fixNonInductionPHIs()

void InnerLoopVectorizer::fixNonInductionPHIs ( VPTransformState & State )

Fix the non-induction PHIs in Plan.

Definition at line 2587 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::get(), llvm::VPlan::getEntry(), llvm::VPPhiAccessors::incoming_values_and_blocks(), P, Plan, llvm::IRBuilderBase::SetInsertPoint(), llvm::vp_depth_first_shallow(), and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by fixVectorizedLoop().

◆ fixVectorizedLoop()

void InnerLoopVectorizer::fixVectorizedLoop ( VPTransformState & State )

Fix the vectorized code, taking care of header phi's, and more.

Definition at line 2552 of file LoopVectorize.cpp.

References llvm::VPTransformState::CFG, cse(), estimateElementCount(), fixNonInductionPHIs(), llvm::vputils::getFirstLoopHeader(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), LI, OrigLoop, llvm::VPTransformState::Plan, llvm::setProfileInfoAfterUnrolling(), UF, VF, llvm::VPTransformState::CFGState::VPBB2IRBB, and llvm::VPTransformState::VPDT.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getAdditionalBypassBlock()

BasicBlock * llvm::InnerLoopVectorizer::getAdditionalBypassBlock ( ) const

inline

Return the additional bypass block which targets the scalar loop by skipping the epilogue loop after completing the main loop.

Definition at line 537 of file LoopVectorize.cpp.

References AdditionalBypassBlock, and assert().

Referenced by llvm::LoopVectorizationPlanner::executePlan(), and llvm::LoopVectorizePass::processLoop().

◆ getTripCount()

Value * llvm::InnerLoopVectorizer::getTripCount ( ) const

inline

Returns the original loop trip count.

Definition at line 528 of file LoopVectorize.cpp.

References TripCount.

Referenced by llvm::EpilogueVectorizerMainLoop::createIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), llvm::LoopVectorizePass::processLoop(), and llvm::LoopVectorizationPlanner::selectEpilogueVectorizationFactor().

◆ printDebugTracesAtEnd()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtEnd ( )

inlineprotectedvirtual

Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.

Definition at line 553 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ printDebugTracesAtStart()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtStart ( )

inlineprotectedvirtual

Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.

Definition at line 552 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ setTripCount()

void llvm::InnerLoopVectorizer::setTripCount ( Value * TC )

inline

Used to set the trip count after ILV's construction and after the preheader block has been executed.

Note that this always holds the trip count of the original loop for both main loop and epilogue vectorization.

Definition at line 533 of file LoopVectorize.cpp.

References TripCount.

Referenced by llvm::LoopVectorizationPlanner::executePlan(), and llvm::LoopVectorizePass::processLoop().

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

friend class LoopVectorizationPlanner

friend

Definition at line 544 of file LoopVectorize.cpp.

Member Data Documentation

◆ AC

AssumptionCache* llvm::InnerLoopVectorizer::AC

protected

Assumption Cache.

Definition at line 573 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationCostModel::collectValuesToIgnore(), llvm::LoopVectorizationPlanner::executePlan(), and processLoopInVPlanNativePath().

◆ AdditionalBypassBlock

BasicBlock* llvm::InnerLoopVectorizer::AdditionalBypassBlock = nullptr

protected

The additional bypass block which conditionally skips over the epilogue loop after executing the main loop.

Needed to resume inductions and reductions during epilogue vectorization.

Definition at line 608 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), and getAdditionalBypassBlock().

◆ BFI

BlockFrequencyInfo* llvm::InnerLoopVectorizer::BFI

protected

BFI and PSI are used to check for profile guided size optimizations.

Definition at line 598 of file LoopVectorize.cpp.

Referenced by getScalarEpilogueLowering(), and processLoopInVPlanNativePath().

◆ Builder

IRBuilder llvm::InnerLoopVectorizer::Builder

protected

The builder that we use.

Definition at line 584 of file LoopVectorize.cpp.

Referenced by addScalarResumePhis(), llvm::EpilogueVectorizerMainLoop::createIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), fixNonInductionPHIs(), llvm::VPRecipeBuilder::handleReplication(), and llvm::VPRecipeBuilder::tryToCreatePartialReduction().

◆ Cost

LoopVectorizationCostModel* llvm::InnerLoopVectorizer::Cost

protected

The profitablity analysis.

Definition at line 595 of file LoopVectorize.cpp.

◆ DT

DominatorTree* llvm::InnerLoopVectorizer::DT

protected

Dominator Tree.

Definition at line 567 of file LoopVectorize.cpp.

Referenced by createScalarPreheader(), llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), llvm::LoopVectorizationPlanner::executePlan(), and processLoopInVPlanNativePath().

◆ LI

LoopInfo* llvm::InnerLoopVectorizer::LI

protected

Loop Info.

Definition at line 564 of file LoopVectorize.cpp.

Referenced by collectSupportedLoops(), llvm::LoopVectorizationCostModel::collectValuesToIgnore(), createScalarPreheader(), llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), fixVectorizedLoop(), and processLoopInVPlanNativePath().

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader = nullptr

protected

The vector-loop preheader.

Definition at line 589 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerMainLoop::createIterationCountCheck(), createScalarPreheader(), llvm::EpilogueVectorizerMainLoop::createVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), and llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck().

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop

protected

The original loop.

Definition at line 556 of file LoopVectorize.cpp.

◆ Plan

VPlan& llvm::InnerLoopVectorizer::Plan

protected

Definition at line 610 of file LoopVectorize.cpp.

Referenced by addExitUsersForFirstOrderRecurrences(), llvm::LoopVectorizationPlanner::addMinimumIterationCheck(), addScalarResumePhis(), calculateEarlyExitCost(), createWidenInductionRecipes(), llvm::LoopVectorizationPlanner::emitInvalidCostRemarks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), fixNonInductionPHIs(), hasReplicatorRegion(), llvm::EpilogueVectorizerMainLoop::introduceCheckBlockInVPlan(), isOutsideLoopWorkProfitable(), planContainsAdditionalSimplifications(), llvm::LoopVectorizationPlanner::selectInterleaveCount(), llvm::VPRecipeBuilder::tryToCreatePartialReduction(), and willGenerateVectors().

◆ PSE

PredicatedScalarEvolution& llvm::InnerLoopVectorizer::PSE

protected

A wrapper around ScalarEvolution used to add runtime SCEV checks.

Applies dynamic knowledge to simplify SCEV expressions and converts them to a more usable form.

Definition at line 561 of file LoopVectorize.cpp.

◆ PSI

ProfileSummaryInfo* llvm::InnerLoopVectorizer::PSI

protected

Definition at line 599 of file LoopVectorize.cpp.

Referenced by getScalarEpilogueLowering(), and processLoopInVPlanNativePath().

◆ RTChecks

GeneratedRTChecks& llvm::InnerLoopVectorizer::RTChecks

protected

Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

Definition at line 603 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), and llvm::LoopVectorizationPlanner::executePlan().

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr

protected

Trip count of the original loop.

Definition at line 592 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), getTripCount(), and setTripCount().

◆ TTI

const TargetTransformInfo* llvm::InnerLoopVectorizer::TTI

protected

Target Transform Info.

Definition at line 570 of file LoopVectorize.cpp.

◆ UF

unsigned llvm::InnerLoopVectorizer::UF

protected

The vectorization unroll factor to use.

Each scalar is vectorized to this many different vector instructions.

Definition at line 581 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::addMinimumIterationCheck(), llvm::EpilogueVectorizerMainLoop::createIterationCountCheck(), fixVectorizedLoop(), and isIndvarOverflowCheckKnownFalse().

◆ VectorPHVPBB

VPBasicBlock* llvm::InnerLoopVectorizer::VectorPHVPBB

protected

The vector preheader block of Plan, used as target for check blocks introduced during skeleton creation.

Definition at line 614 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), and llvm::EpilogueVectorizerMainLoop::introduceCheckBlockInVPlan().

◆ VF

ElementCount llvm::InnerLoopVectorizer::VF

protected

The vectorization SIMD factor to use.

Each vector will have this many vector elements.

Definition at line 577 of file LoopVectorize.cpp.

The documentation for this class was generated from the following file:

lib/Transforms/Vectorize/LoopVectorize.cpp

Public Member Functions

Protected Member Functions

Protected Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

◆ ~InnerLoopVectorizer()

Member Function Documentation

◆ createScalarPreheader()

◆ createVectorizedLoopSkeleton()

◆ fixNonInductionPHIs()

◆ fixVectorizedLoop()

◆ getAdditionalBypassBlock()

◆ getTripCount()

◆ printDebugTracesAtEnd()

◆ printDebugTracesAtStart()

◆ setTripCount()

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

Member Data Documentation

◆ AC

◆ AdditionalBypassBlock

◆ BFI

◆ Builder

◆ Cost

◆ DT

◆ LI

◆ LoopVectorPreHeader

◆ OrigLoop

◆ Plan

◆ PSE

◆ PSI

◆ RTChecks

◆ TripCount

◆ TTI

◆ UF

◆ VectorPHVPBB

◆ VF