R&D · UE5 Global Shaders · RDG · HLSL

Stencil Vision

Stencil Vision

A custom UE5 global shader pipeline built outside the Material Editor. A FViewExtension hooks into the Render Dependency Graph, reads the depth-stencil buffer in HLSL, and composites a coloured UV mask over stencil-tagged objects. Everything — C++ injection point, shader permutations, and .usf files — is hand-written.

FViewExtension FRDGBuilder HLSL .usf Stencil Buffer UE5 RHI C++
What I Built
  • FViewExtension hook — 1 injection point — 1 PrePostProcessPass_RenderThread() override injects the custom pass into the post-process pipeline; guards on PF_DepthStencil availability with 0 Blueprint dependency.
  • FRDGBuilder — 2 RDG passes — adds 2 screen-pass texture renders (UV mask + combine) via FScreenPassTexture and GetGlobalShaderMap; 0 Blueprint or Material graph involved.
  • UVMaskMainPS — 1 stencil value — HLSL pixel shader reads stencil channel; 1 value (stencil.y == 1) selects the masked region; all other pixels output zero.
  • CombineMainPS — 1 Color parameter — second pass lerps the masked UV region with 1 configurable Color parameter and 1 BlendAmount scalar, compositing custom highlight over scene colour.
  • PosToUV helper — 1 function, 2 shaders — 1 clip-space to UV conversion helper shared across 2 shaders, ensuring stencil reads align with scene texture samples.
C++ — Injecting into the RDG Post-Process Chain

PrePostProcessPass_RenderThread() is called by the engine right before the post-process stack; this is the injection point for the custom stencil pass.

MyViewExtension.cpp — PrePostProcessPass_RenderThread() 
void FMyViewExtension::PrePostProcessPass_RenderThread(
    FRDGBuilder& GraphBuilder,
    const FSceneView& View,
    const FPostProcessingInputs& Inputs)
{
    // Guard — depth stencil must be available
    if (!Inputs.SceneTextures || !(*Inputs.SceneTextures)->UniformBuffer)
        return;
    const FSceneTextureParameters& SceneTex =
        (*Inputs.SceneTextures)->UniformBuffer->GetContents();
    if (SceneTex.SceneDepthTexture->Desc.Format() != PF_DepthStencil)
        return;

    FScreenPassTexture SceneColor((*Inputs.SceneTextures)->SceneColorTexture);
    FGlobalShaderMap* ShaderMap = GetGlobalShaderMap(GMaxRHIFeatureLevel);

    // Pass 1: UV mask from stencil
    FRDGTextureRef MaskTexture = AddUVMaskPass(GraphBuilder, SceneColor,
                                                 SceneTex, ShaderMap);

    // Pass 2: Combine mask with scene colour
    AddCombinePass(GraphBuilder, SceneColor, MaskTexture, ShaderMap, TintColor);
}
HLSL — Stencil Read & Combine

Two pixel shaders: the first reads the stencil value and outputs a UV mask; the second lerps the mask region with a custom tint colour over the scene.

MyCustomShader.usf — UVMaskMainPS + CombineMainPS 
// Convert clip position to UV
float2 PosToUV(float4 Pos, float2 ViewportSize)
{
    return (Pos.xy / Pos.w) * float2(0.5f, -0.5f) + 0.5f;
}

// Pass 1: output UV where stencil == 1, else 0
float4 UVMaskMainPS(FScreenVertexOutput Input) : SV_Target
{
    float2 UV  = Input.UV;
    float4 DepthStencil = SceneDepthTexture.Sample(SceneSampler, UV);

    // stencil stored in .y channel after decode
    if (DepthStencil.y == 1.0f)
        return float4(UV, 0.0f, 1.0f);   // masked region — output UV
    return float4(0.0f, 0.0f, 0.0f, 0.0f);  // unmasked — transparent
}

// Pass 2: blend masked UV region with custom tint
float4 CombineMainPS(FScreenVertexOutput Input) : SV_Target
{
    float2 UV        = Input.UV;
    float4 SceneCol  = SceneColorTexture.Sample(SceneSampler, UV);
    float4 MaskCol   = MaskTexture.Sample(SceneSampler, UV);

    // Where mask alpha > 0, blend in the custom Color
    return MaskCol.a > 0.0f
        ? lerp(SceneCol, Color, BlendAmount)
        : SceneCol;
}
Engine Unreal Engine 5 Language C++ + HLSL Hook FViewExtension → RDG Shaders UVMaskMainPS · CombineMainPS Buffer Stencil read via PF_DepthStencil Category R&D · Graphics Programming Source github.com/khaled71612000 ↗
View Source
Connect