Poor Normal Map Sampling with Render to Texture

If you've used Render to Texture (RTT) with Mental Ray as your renderer, you may have been frustrated by the poor anti-aliasing quality of Normal Maps produced through RTT. Below are some samples and a description of the problems and my recommended solution at the bottom. If there's a solution I'm not aware of using Mental Ray, please let me know what I'm missing.

I'll assume at this point you are familiar with RTT and have at least dabbled in it to produce various image outputs for use, most likely , in some sort of real-time or gaming environment. I do not profess to be an expert in any of these areas, just another experienced user, frustrated by some unexpected behavior in Max.

1st you likely have a high detail model and you plan to use RTT to render out some good maps to use on a lower detail model. Pretty typical. You probably want to create a normal map to add detail from the high poly version onto the lower poly version. Projection mapping is required and RTT is set up to deal with this.
So next you would align your 2 models, create the Projection Modifier (either by adding it in RTT or manually) and refine the projection cage to your liking. BTW there are tons of great tutorials and forum posts related to these tasks online. However, I have only found a few dealing with the anti-aliasing issue and none addressing the specific issues with Mental Ray (that doesn't mean there isn't something out there of course).

For my examples below I'm using a hubcap model. I've setup the low detail model and Projection Modifier. now let's look at some of the renders RTT produced and some of the settings that are confusing.
Assuming we leave Mental Ray at some generic settings, we have Final Gather enabled, some generic lights and we're using Unified Sampling for anti-aliasing.
Setting up all our settings correctly and doing a render from RTT at 512 x 512 we get a reasonable looking complete map, but the Normal Map seems very pixelated:

Note: Click the images to see the full resolution details

Complete Map with generic settings

Normal Map output by RTT from same render as above Complete Map

 Note the pixelation of the normal map that doesn't seem to be an issue on the Complete Map.

Now to try and fix this within  RTT, we look at the "Options" dialog under the "Objects to Bake" rollout, projection mapping group:

Notice the sampling "setup" which just takes you to the render dialog's sampling group for Mental Ray. At the Unified default of minimum = 1 and maximum = 128 it appears that RTT is only using the minimum for the Normal Map portion of the render and NOT taking advantage of the adaptive sampling up to 128 that we see in the complete map. So to test that, lets set both Max and Min to 1 and see what we get:

Complete Map rendered with Unified Sampling set to Minimum=1 and Maximum=1

Indeed, it looks like RTT is only using the "minimum" sampling setting to render Normal Maps. Ok, so let's set the minimum up to 16 and see what we get.

Normal Map with UnifiedSampling Minimum set to 16

The quality of the anti-aliasing seems no different and in fact if we look at the "Projection Mapping" "Options" we see that the "Samples per Pixel" still says "1". In fact, with Unified Sampling, it will always say 1 and will never render to a higher sampling, leaving your Normal Maps with this poor aliasing.

So what if we change to the old "Classic/Raytraced" sampling method. Now if we change the Minimum, the RTT "Samples per Pixel" option updates. Yippie! Problem solved right? Test our theory by rendering with a minimum of 64 samples per pixel, pretty high quality that should yield nice anti-aliasing:

Normal Map with Classic/Raytraced sampling and minimum set to 64 samples

Well...that doesn't seem any better. In fact, you can set the minimum to something outrageous like 256 and the quality will not improve:

Normal Map with Classic/Raytraced sampling and minimum set to 256 samples

Not only does rendering with 256 samples as a MINIMUM not improve the anti-aliasing, it also takes 7 minutes now.

So, in conclusion, it seems that Render To Texture will only sample Normal Maps at 1 sample per pixel when rendering with Mental Ray leaving you with poorly anti-aliased images.
Solution: render at a higher resolution (2048 or 4096) and downsize the image as needed in Photoshop. The added render time is minimal considering that Normal Maps do not require Final Gather or Shadows so you can turn them off before rendering. And you can set the Sampling quality to minimum=1 and maximum=1:

Normal Map rendered at 2048 x 2048 downsized to 512 x 512. Sampling =1 with no Shadows or Final Gather

On a final note, for down-resing your image in Photoshop, you might want to pick the Bicubic "Smoother" option rather than "Sharper" filtering option to avoid some artifacts. This is contrary to the common practice for re-sizing images in Photoshop.