Ray Tracing Shadows for Foliage in Unreal Engine 4.25 & 4.26

 I wanted to take some time and consolidate my findings on the problems related to Ray Tracing and Foliage painted instances. This is important to me since I have released several foliage Asset Packs on the Marketplace and I want to be clear for users on the limitations when using the nvidia RTX technologies with Foliage placed assets. There are numerous forum posts on the issues but it takes some time to sort through them and figure out what's really going on and what choices you have as a developer to get the best results. 

All the differences that really matter involve shadows and the Sub-Surface properties of the Two Sided Foliage shader. There are some key differences, not just when using Ray Tracing vs. Cascading Shadow Maps (CSM), but even between the Ray Tracing options in 4.26 vs, earlier releases.

To start with, at this time, the best shading results with large, foliage painted scenes where the Foliage Instances are using the Two-Sided Foliage Shader are with old school CSMs, .....depending on how picky you are. This involves disabling Ray Tracing Shadows altogether on your primary Directional Light source, globally for the whole scene. Obviously not the solution most of us want to hear. But until Unreal can create a solution for the limitations in Ray Tracing and the Two Sided Foliage Shader, this is the only option I have found.

For the purposes below, I am assuming a generic setup as you would get by starting a new level, creating a landscape and painting some foliage onto it. In addition, the primary Directional Light has been set to "Movable" as is the recommended workflow for a large environment. I am not dealing with a Static Light source, trying to bake foliage shadows into a large environment scene.

Sub Surface Issues:

Lets take a look at 2 renders, one using CSM and one using Ray Tracing Shadows. Note that the 2 are currently mutually exclusive. If you have Ray-Tracing Shadows enabled, adjusting CSM properties makes no difference. They are not being used. The material assigned to the low poly Aspen tree model is of course a Material Instance and the Master Material has "Cast Ray Traced Shadows" enabled and is using the Two Sided Foliage shading model.

Aspen Tree Render, Ray Tracing Shadows enabled

As you can see in the above rendering, the Sub Surface attributes of the Two Sided Foliage shader are not being computed when using Ray Traced Shadows.

Now the exact same shot, but with Ray Traced Shadows disabled:

with Ray Traced Shadows disabled, shadows are computed using Cascading Shadow Maps and now we see the Sub Surface properties of the Two Sided Foliage Shader. 

Obviously the look with Sub Surface properties rendered is more what we want and expect. Rendering Sub Surface properties with Ray Traced Shadows enabled is not currently possible up to version 4.26. I refer to this forum post for the explanation where Epic Staff member YuriyODonnell writes on 4/24/20 in version 4.25 at the time:

"In general, there are some limits to how ray tracing works with subsurface scattering. Shadow maps are currently required for certain effects (such as transmission) to work correctly. This is because ray traced shadows work by starting a ray at the receiving surface and tracing it toward the light. Shadow maps do the opposite: start from light and "trace" towards scene. Shadow maps therefore allow us to approximate object thickness.
My gut feeling is that a shadow map will be required to achieve the subsurface look you are after (i.e. shadow maps would still need to be rendered even when RT shadows are on, but only used for SSS). Perhaps a specialized path can be also implemented using ray tracing, but no work has been done on this specifically so far."

One possible solution that comes to mind is to disable the "Cast Ray Traced Shadows" option on the material itself. Then perhaps as we've seen with the the Scene light option, the foliage will now use CSMs for its shadows:

"Cast Ray Traced Shadows" disabled on the material

At first glance in this render that seems to work:

Ray Traced shadows disabled on the material but enabled for the scene light

Unfortunately this currently results in the foliage casting no shadows whatsoever:

Ray Traced Shadows disabled on the Material but enabled in the scene results in no shadows at all for objects with that material.

As in the above forum post response, perhaps a separate path could be used in the future such that objects with materials using SSS will be rendered using CSM while all others would still use Ray Tracing if enabled.

Of course if we disable Ray Traced Shadows again on the light source, we now default back to CSM calculation and we get the traditional shadows as we would have seen in versions prior to Ray Tracing:

Cascading Shadow Maps only for the scene light, no Ray Tracing.

As it stands, you will have to decide whether the benefits of Ray Tracing for your scene outweigh the benefits of Sub Surface rendering on your foliage. The foliage still renders OK with ray tracing, but granted, the look is not as realistic. Let's hope for a good solution in the future.

4.25 vs. 4.26, Culling and Console Commands

One major problem I had with my scenes when transitioning to Ray Tracing versions of Unreal Engine versus the earlier versions prior to RTX was somewhat common, but with one critical omission from any of the forum posts I could find. I'm guessing this problem is a little too new with 4.26 being a relatively recent release and developers having already solved their shadow problems based on the above issues.

My scene renders as expected in in 4.25 except that the shadows only go so far into my foliage before they disappear. This has been a known issue discussed on several forum posts. I will be referring to this post in particular regarding Ray Traced Shadows on Foliage actors in 4.24. But this issue is valid in 4.25 as well as 4.26 as I will discuss.

In release 4.25 below is a render from a scene built in 4.18 (no RTX) but opened in 4.25 with Ray Tracing enabled:

Ray Traced Shadows enabled but "Culled" by default

The simple solution to this problem is to invoke a CVAR that disables the Culling of the Ray Traced Shadows. Open a command console type-in field and enter this:

"r.RayTracing.InstancedStaticMeshes.Culling 0"

The results here:

There are indeed a series of commands related to controlling the ray traced shadows on Instanced Static Meshes, kindly documented in the above forum post by contributor Duongngochieu on 03-24-2020

Problem Solved Right?? Not so fast. Now let's try opening the scene in 4.26. Without any adjustments we get this:

Ray Traced Shadows on Foliage painted actors in UE 4.26 with default settings

What I get here is even more bizarre, with random actors casting shadows and in fact, as I navigate about the scene the shadows come and go with seemingly no rationale. Sometimes as I get closer the shadows vanish on a tree. Sometimes vanishing as I orbit or even just pan. The shadows come and go on seemingly random trees having nothing to do with culling distance.

So the Solution! Let's invoke the CVAR from above: "r.RayTracing.InstancedStaticMeshes.Culling 0"

UE 4.26 with CVAR: "r.RayTracing.InstancedStaticMeshes.Culling 0"

So the results are not what we expect. Actually, no change at all. This console command does not appear to be working to solve the problem. After a lot of research, trying other commands and fiddling with ray tracing settings for days I discovered that the console commands for Ray Tracing on Instanced Static Meshes have changed in 4.26. I have found no specific documentation on this but it works if you invoke the same console command constructed this way:

"r.RayTracing.Geometry.InstancedStaticMeshes.Culling 0"

In fact, all commands related to Ray tracing Instanced Static Meshes in 4.26 need this new declaration structure with the insertion of the word "Geometry" into the command.

Below are most of the related commands as they are implemented in 4.25, again, courtesy of forum contributor Duongngochieu from his above linked post:

Whether to include Instanced Static Meshes in BVH:

r.RayTracing.InstancedStaticMeshes [0|1]

Whether to enable culling of instance data:

r.RayTracing.InstancedStaticMeshes.Culling [0|1]

Scale factor on how large clusters are. Number represents a scale against the size of instances. 

5.0 means that clusters are 5x the size of the largest instance object:

r.RayTracing.InstancedStaticMeshes.CullClusterMaxRadiusMultiplier

Distance at which to cull a cluster: 

r.RayTracing.InstancedStaticMeshes.CullClusterRadius 

Threshold for when an object is considered small: 

r.RayTracing.InstancedStaticMeshes.LowScaleRadiusThreshold

Distance at which to cull small objects: 

r.RayTracing.InstancedStaticMeshes.LowScaleCullRadius

I'm assuming all these commands will work in 4.26 SO LONG AS you insert "Geometry" between "r.RayTracing." and ".InstancedStaticMeshes". It seems "InstancedStaticMeshes" are now parented by "Geometry" in the code.

I came upon this almost by accident in my research when watching this video by nvidia developer Richard Cowgill. It's well worth checking out. He is discussing new options for dealing with WPO offset animations in your foliage and lists the new commands which strongly resemble the formatting of the other rayTracing commands I had been working with. Near the end he mentions that simply removing the word "Geometry" from the new commands would make them work in UE 4.25. That would also make them follow the currently known formatting for the other commands related to RayTracing InstancedStaticMeshes in 4.25. Seemed a logical extension to try out the old CVAR's with this new formatting, inserting the word "Geometry" and sure enough, things start behaving in 4.26 as they do in 4.25.

I leave other settings related to Ray Traced Shadows up to you to evaluate and implement as you see fit for best results in your own scenes. I did however want to take the time to clearly (I hope) explain some of these most critical issues in a single post. And great thanks as always to the courageous forum posters putting themselves out there and allowing all of us to benefit from the wisdom and experience of others.

David Sparks