Ray Tracing Shadows for Foliage in Unreal Engine 4.25 & 4.26

 I wanted to take some time and consolidate my findings on the problems related to Ray Tracing and Foliage painted instances. This is important to me since I have released several foliage Asset Packs on the Marketplace and I want to be clear for users on the limitations when using the nvidia RTX technologies with Foliage placed assets. There are numerous forum posts on the issues but it takes some time to sort through them and figure out what's really going on and what choices you have as a developer to get the best results. 

All the differences that really matter involve shadows and the Sub-Surface properties of the Two Sided Foliage shader. There are some key differences, not just when using Ray Tracing vs. Cascading Shadow Maps (CSM), but even between the Ray Tracing options in 4.26 vs, earlier releases.

To start with, at this time, the best shading results with large, foliage painted scenes where the Foliage Instances are using the Two-Sided Foliage Shader are with old school CSMs, .....depending on how picky you are. This involves disabling Ray Tracing Shadows altogether on your primary Directional Light source, globally for the whole scene. Obviously not the solution most of us want to hear. But until Unreal can create a solution for the limitations in Ray Tracing and the Two Sided Foliage Shader, this is the only option I have found.

For the purposes below, I am assuming a generic setup as you would get by starting a new level, creating a landscape and painting some foliage onto it. In addition, the primary Directional Light has been set to "Movable" as is the recommended workflow for a large environment. I am not dealing with a Static Light source, trying to bake foliage shadows into a large environment scene.

Sub Surface Issues:

Lets take a look at 2 renders, one using CSM and one using Ray Tracing Shadows. Note that the 2 are currently mutually exclusive. If you have Ray-Tracing Shadows enabled, adjusting CSM properties makes no difference. They are not being used. The material assigned to the low poly Aspen tree model is of course a Material Instance and the Master Material has "Cast Ray Traced Shadows" enabled and is using the Two Sided Foliage shading model.

Aspen Tree Render, Ray Tracing Shadows enabled

As you can see in the above rendering, the Sub Surface attributes of the Two Sided Foliage shader are not being computed when using Ray Traced Shadows.

Now the exact same shot, but with Ray Traced Shadows disabled:

with Ray Traced Shadows disabled, shadows are computed using Cascading Shadow Maps and now we see the Sub Surface properties of the Two Sided Foliage Shader. 

Obviously the look with Sub Surface properties rendered is more what we want and expect. Rendering Sub Surface properties with Ray Traced Shadows enabled is not currently possible up to version 4.26. I refer to this forum post for the explanation where Epic Staff member YuriyODonnell writes on 4/24/20 in version 4.25 at the time:

"In general, there are some limits to how ray tracing works with subsurface scattering. Shadow maps are currently required for certain effects (such as transmission) to work correctly. This is because ray traced shadows work by starting a ray at the receiving surface and tracing it toward the light. Shadow maps do the opposite: start from light and "trace" towards scene. Shadow maps therefore allow us to approximate object thickness.
My gut feeling is that a shadow map will be required to achieve the subsurface look you are after (i.e. shadow maps would still need to be rendered even when RT shadows are on, but only used for SSS). Perhaps a specialized path can be also implemented using ray tracing, but no work has been done on this specifically so far."

One possible solution that comes to mind is to disable the "Cast Ray Traced Shadows" option on the material itself. Then perhaps as we've seen with the the Scene light option, the foliage will now use CSMs for its shadows:

"Cast Ray Traced Shadows" disabled on the material

At first glance in this render that seems to work:

Ray Traced shadows disabled on the material but enabled for the scene light

Unfortunately this currently results in the foliage casting no shadows whatsoever:

Ray Traced Shadows disabled on the Material but enabled in the scene results in no shadows at all for objects with that material.

As in the above forum post response, perhaps a separate path could be used in the future such that objects with materials using SSS will be rendered using CSM while all others would still use Ray Tracing if enabled.

Of course if we disable Ray Traced Shadows again on the light source, we now default back to CSM calculation and we get the traditional shadows as we would have seen in versions prior to Ray Tracing:

Cascading Shadow Maps only for the scene light, no Ray Tracing.

As it stands, you will have to decide whether the benefits of Ray Tracing for your scene outweigh the benefits of Sub Surface rendering on your foliage. The foliage still renders OK with ray tracing, but granted, the look is not as realistic. Let's hope for a good solution in the future.

4.25 vs. 4.26, Culling and Console Commands

One major problem I had with my scenes when transitioning to Ray Tracing versions of Unreal Engine versus the earlier versions prior to RTX was somewhat common, but with one critical omission from any of the forum posts I could find. I'm guessing this problem is a little too new with 4.26 being a relatively recent release and developers having already solved their shadow problems based on the above issues.

My scene renders as expected in in 4.25 except that the shadows only go so far into my foliage before they disappear. This has been a known issue discussed on several forum posts. I will be referring to this post in particular regarding Ray Traced Shadows on Foliage actors in 4.24. But this issue is valid in 4.25 as well as 4.26 as I will discuss.

In release 4.25 below is a render from a scene built in 4.18 (no RTX) but opened in 4.25 with Ray Tracing enabled:

Ray Traced Shadows enabled but "Culled" by default

The simple solution to this problem is to invoke a CVAR that disables the Culling of the Ray Traced Shadows. Open a command console type-in field and enter this:

"r.RayTracing.InstancedStaticMeshes.Culling 0"

The results here:

There are indeed a series of commands related to controlling the ray traced shadows on Instanced Static Meshes, kindly documented in the above forum post by contributor Duongngochieu on 03-24-2020

Problem Solved Right?? Not so fast. Now let's try opening the scene in 4.26. Without any adjustments we get this:

Ray Traced Shadows on Foliage painted actors in UE 4.26 with default settings

What I get here is even more bizarre, with random actors casting shadows and in fact, as I navigate about the scene the shadows come and go with seemingly no rationale. Sometimes as I get closer the shadows vanish on a tree. Sometimes vanishing as I orbit or even just pan. The shadows come and go on seemingly random trees having nothing to do with culling distance.

So the Solution! Let's invoke the CVAR from above: "r.RayTracing.InstancedStaticMeshes.Culling 0"

UE 4.26 with CVAR: "r.RayTracing.InstancedStaticMeshes.Culling 0"

So the results are not what we expect. Actually, no change at all. This console command does not appear to be working to solve the problem. After a lot of research, trying other commands and fiddling with ray tracing settings for days I discovered that the console commands for Ray Tracing on Instanced Static Meshes have changed in 4.26. I have found no specific documentation on this but it works if you invoke the same console command constructed this way:

"r.RayTracing.Geometry.InstancedStaticMeshes.Culling 0"

In fact, all commands related to Ray tracing Instanced Static Meshes in 4.26 need this new declaration structure with the insertion of the word "Geometry" into the command.

Below are most of the related commands as they are implemented in 4.25, again, courtesy of forum contributor Duongngochieu from his above linked post:

Whether to include Instanced Static Meshes in BVH:

r.RayTracing.InstancedStaticMeshes [0|1]

Whether to enable culling of instance data:

r.RayTracing.InstancedStaticMeshes.Culling [0|1]

Scale factor on how large clusters are. Number represents a scale against the size of instances. 

5.0 means that clusters are 5x the size of the largest instance object:

r.RayTracing.InstancedStaticMeshes.CullClusterMaxRadiusMultiplier

Distance at which to cull a cluster: 

r.RayTracing.InstancedStaticMeshes.CullClusterRadius 

Threshold for when an object is considered small: 

r.RayTracing.InstancedStaticMeshes.LowScaleRadiusThreshold

Distance at which to cull small objects: 

r.RayTracing.InstancedStaticMeshes.LowScaleCullRadius

I'm assuming all these commands will work in 4.26 SO LONG AS you insert "Geometry" between "r.RayTracing." and ".InstancedStaticMeshes". It seems "InstancedStaticMeshes" are now parented by "Geometry" in the code.

I came upon this almost by accident in my research when watching this video by nvidia developer Richard Cowgill. It's well worth checking out. He is discussing new options for dealing with WPO offset animations in your foliage and lists the new commands which strongly resemble the formatting of the other rayTracing commands I had been working with. Near the end he mentions that simply removing the word "Geometry" from the new commands would make them work in UE 4.25. That would also make them follow the currently known formatting for the other commands related to RayTracing InstancedStaticMeshes in 4.25. Seemed a logical extension to try out the old CVAR's with this new formatting, inserting the word "Geometry" and sure enough, things start behaving in 4.26 as they do in 4.25.

I leave other settings related to Ray Traced Shadows up to you to evaluate and implement as you see fit for best results in your own scenes. I did however want to take the time to clearly (I hope) explain some of these most critical issues in a single post. And great thanks as always to the courageous forum posters putting themselves out there and allowing all of us to benefit from the wisdom and experience of others.

David Sparks

Realistic Foliage Pack 2 "Trees" for Unreal Engine

 

Realistic Foliage Pack2 “Trees” 

Available as of Christmas Day on the UE4 Marketplace. Below find some detailed support about the foliage pack. Watch the demo video here:

Realistic Foliage Pack Vol. 2 Demo Video

Also make sure and check out the  "Bushes" Foliage Pack!:

DAS Foliage Pack Vol 1 "Bushes"

About:

This UE4 Asset Pack contains 20 individual meshes representing 5 common species of trees and 5 unique indoor plant pots. Each tree species includes 3 procedural variations allowing you to create forests or large groupings without obvious repetition.

These trees are meant to fill the gap between high poly “Cinematic” models and lower poly gaming/VR assets. All “Hero” LOD’s are less than 20K polys with most below 15K and several below 10K. These models are ideal for architectural visualization but also most game environments.

There are Demo levels for each tree and a “Library” Level containing just the core content. Note that the staging environments for the Demo Levels are for demonstration purposes only and contain a few additional assets from the UE4 Starter Content and Content Examples. Each Demo Level also contains the UE4 Mannequin for scale reference.

*A Note About Foliage Instancing with Ray Tracing: 

If you plan on using the foliage assets with a Ray Tracing version of the Unreal Engine, at this time I recommend disabling "Cast Ray Traced Shadows" on your light source if you are planning to use the trees with Foliage Instancing. This will result in the use of Cascading Shadow Maps. There have been numerous posts since the start of RTX regarding issues with Foliage Instances and it seems yet to be fully resolved.

For an in-depth explanation of the problems and solutions regarding Ray Tracing and Foliage Instanced Static Meshes, please refer to my more recent post here. It's a bit long-winded but I think it will make many things clear if you have questions.

 Be aware that the renderings below are all from ver. 4.18 (no ray tracing) in an attempt to show the "worst case scenario".
Also note that you can choose whether to set the skylight to 'Stationary" or "Moveable" and this has a dramatic effect on the appearance of the foliage. I prefer the look with the skylight set to Moveable/Dynamic but other factors may determine this for you. In many of my scenes the skylight is set to Stationary but I find this leads to shadows/shading that are too dark. You will no doubt be importing the foliage into your own scene and will set the lighting as needed there rather than inheriting the lighting scheme from any of my demo scenes.

Here are a few shots from the asset pack:

(Note: All pictures and videos in this blog can be clicked on for larger versions)

Meshes:

1.       Coconut Palm

2.       Aspen

3.       Honey Locust

4.       Schefflera (Indoor)

5.       Spruce (Including winter snow version via separate material)

6.       Pots: 5 individual models, can be used with or without a tree.

For the Spruce tree there is a separate winter Snow Material, and though the mesh is the same as the summer version this can be considered a separate tree.

LODs:

Each set of trees contains seamless LOD’s that reduce to 3D cross-hatched planes (Front, Side and Up facing) plus an ultra low poly trunk. The exception is the indoor Schefflera tree that does not reduce all the way down to the cross-hatched planes since it is not meant for foresting. All models contain 4 LOD’s except for the Spruce trees which have 5. The final LOD for the trees is typically close to 25 polys (except the Schefflera as noted).

Here is a sample video showing the typical LOD's for one of the Aspen trees in the collection:

Here is another sample video showing the LOD's on one of the pots in the collection:

 All mesh models have custom UVW channel 2 layouts for baked shadows. Here is a sample of the LOD0 layout for one of the Honey Locust trees:

Materials:

There are a total of 3 Master Materials and 26 Material Instances for the core content of the asset pack.

1 Master Material for all the trees, 1 Master Material for the final Cross-Hatched planar LOD’s and 1 generic Master Material for the plant pots.

Several other materials, notably Landscape materials, are used for each staging environment. The staging also includes slightly modified versions of materials from the Starter Content and Content Examples where needed. You are welcome to use these if you so desire but they are not integral to the trees and pots themselves.

The FoliageMaster2 Material is the master material for all the trees which then use their own Instance based on this. Except for the final, planar LOD, this is an Atlas Material. There are 2 materials for the final, planar LOD version each tree. The original Atlas Instance from the full 3D mesh version as it is used on the trunk and an Instance of a Cross-Hatch Master Material. Each Instance of the “X-Hatch” Material is unique to each mesh model to match the shape and appearance of that particular tree variation.

 

Special Material Parameters, "Sub-Color":

The FoliageMaster2 Material includes a set of sub-routines called “Sub Color”. This enables additional features for some of the trees which can benefit from some localized, per-Instance Color variations. For example, the Aspen tree colors can by randomized for Fall colors trending toward yellow and red hues. With the Palm and Spruce trees, “Sub-Color” can be set to a brown hue resulting in the appearance of dead or dying areas on some of the fronds. The areas of “Sub-Color” variation are declared using either the Mask included with the Roughness, Alpha and Mask texture (the Mask declares just the “Leaf” portions of every tree) or through the Blue Vertex Color channel which has been set in the mesh for localized regions on some of the fronds. The Sub-Color routines can also be used for controlled, per-Instance variations on any tree if desired. Though there is already a Material Function included in the MasterFoliage2 material for this, the specific color of the variation cannot be controlled by the user with that function.

When using Sub-Color on any of the trees, be aware that the final Cross-Hatch LOD does NOT have any Blue painted vertices and so localized sub-colors declared using the Blue Vertex Channel checkbox will not translate over. When use of the Blue Vertex Channel is disabled, the effect uses the Mask texture to apply the effect to all the “leaf” portions of the trees in both the full 3D LOD’s and the Cross-hatched LOD. Just make sure and set the sub-color parameters identical across both Materials.

Below are a few examples of the kind of variation that can be achieved using the "Sub-Color" property:

A sample palm frond with no "Sub-Color" material property variation

The same frond with "Sub-Color" variation configured using the blue vertex channel in the mesh

The Spruce trees with no Sub-Color variation applied

The same spruce trees with Sub-Color variation applied using the blue vertex channel

The same trees again with Sub-Color applied via the default Mask texture (blue vertex unchecked)

The Spruce trees with modified Sub-Color making Blue Spruces

A normal summertime Aspen forest scene

The same Aspen forest using Sub-Colors to mimic a fall season scene

Here is an external link to a detailed video explanation (with audio descriptions) of the Material "Sub-Color" parameters and how they work:

Video: RealisticTrees_SubColor Parameters Explained

Wind:

Wind is supported in the FoliageMaster2 Material on all the trees through the Red Vertex Color Channel with parameters for Size, Speed and Strength.

Here is a demo video on the Wind Parameters as used in the collection:

Collision:

Collision is supported using the Simple primitives method provided for in UE4. All the trees have Capsule collision around the trunk allowing you to move through the leaves.

Inside the red outline you can see the green Capsule collision primitive shaped to encompass just the trunk on this Spruce tree

Another Simple Collision using 2 capsules to best match the trunk of one of the Palm trees

Material Parameters Exposed:

Diffuse – Texture, Tint, Output, Color Variation per Instance

Emissive – Amount

Normals – Texture, Intensity, Flip Green

Roughness – Texture, Invert, Intensity

Specular – Amount

Sub-Surface – Intensity

Wind – Size, Speed, Strength

“Sub-Color” – (See above) localized random variation per instance

 

Textures:

All Atlas textures are at 4K resolution. Cross-hatched 3D planar version textures are 1K except for 2 of the layouts which happen to fit nicely into a 1024 x 512 texture size.

The “RAM” texture contains separate channels for “Roughness” – Red, “Alpha” –Green, and a “Mask” (used by the color variation Material Functions) – Blue. This setup allows a single master material to be used for all the trees regardless of their unique requirements.

Here are some examples of the "RAM" texture used on the Aspen trees in the Atlas material:

Red Channel mask used for Roughness

Green Channel mask used for Alpha cutout

Blue Channel mask used to control color variation on only the leaves

A similar mask texture is used on the final, LOD3 Cross-hatched Material but Roughness has been replaced by an additional mask used to make adjustments to the Up-facing plane on the cross-hatched planes that make up the canopy of the tree on this ultra-low poly LOD version of the mesh.

Honey Locust Tree. The Final LOD Cross-Hatched Texture Mask ("MAM") 

A breakdown of the Mask channels for the Honey Locust Tree Cross-Hatched Final LOD texture seen above:

Cross-Hatched Final LOD Texture mask: Red Channel = Up-facing Polygon Mask

Cross-Hatched Final LOD Texture mask: Green Channel = Alpha Mask

Cross-Hatched Final LOD Texture mask: Blue Channel = Leaves Only Mask

Providing a separate Up Facing mask allows you to control this polygon independently of the rest. Facing upwards, it often receives a bit more light intensity the the rest. With a separate mask you can adjust this polygon to match the rest of the tree in any given lighting scenario. The separate Blue Channel is used by the Sub-Color routines where needed so that only the "Leaf" portions of the tree are affected by custom color adjustments, not the trunks and branches.

I'll also note here that the Normal texture map for these final, cross-hatched LOD's has been rendered from the "Render to Texture_LevelBP" setup in the Engine Content folder. This produces a World Space Normal texture map that needs to be converted (in the Master X-Hatch Material) to Local coordinates. Hence the Normal texture maps for the planar tree versions look different than you might expect.

There are a total of 99 textures. 82 are unique to the core foliage asset pack content and 17 are from the UE4 Starter or Example Content and are merely for the staging environments and are not integral to the core foliage.