I'll assume at this point you are familiar with RTT and have at least dabbled in it to produce various image outputs for use, most likely , in some sort of real-time or gaming environment. I do not profess to be an expert in any of these areas, just another experienced user, frustrated by some unexpected behavior in Max.
1st you likely have a high detail model and you plan to use RTT to render out some good maps to use on a lower detail model. Pretty typical. You probably want to create a normal map to add detail from the high poly version onto the lower poly version. Projection mapping is required and RTT is set up to deal with this.
So next you would align your 2 models, create the Projection Modifier (either by adding it in RTT or manually) and refine the projection cage to your liking. BTW there are tons of great tutorials and forum posts related to these tasks online. However, I have only found a few dealing with the anti-aliasing issue and none addressing the specific issues with Mental Ray (that doesn't mean there isn't something out there of course).
For my examples below I'm using a hubcap model. I've setup the low detail model and Projection Modifier. now let's look at some of the renders RTT produced and some of the settings that are confusing.
Assuming we leave Mental Ray at some generic settings, we have Final Gather enabled, some generic lights and we're using Unified Sampling for anti-aliasing.
Setting up all our settings correctly and doing a render from RTT at 512 x 512 we get a reasonable looking complete map, but the Normal Map seems very pixelated:
Note: Click the images to see the full resolution details
Complete Map with generic settings
|Normal Map output by RTT from same render as above Complete Map|
Now to try and fix this within RTT, we look at the "Options" dialog under the "Objects to Bake" rollout, projection mapping group:
|Complete Map rendered with Unified Sampling set to Minimum=1 and Maximum=1|
|Normal Map with UnifiedSampling Minimum set to 16|
So what if we change to the old "Classic/Raytraced" sampling method. Now if we change the Minimum, the RTT "Samples per Pixel" option updates. Yippie! Problem solved right? Test our theory by rendering with a minimum of 64 samples per pixel, pretty high quality that should yield nice anti-aliasing:
|Normal Map with Classic/Raytraced sampling and minimum set to 64 samples|
Well...that doesn't seem any better. In fact, you can set the minimum to something outrageous like 256 and the quality will not improve:
|Normal Map with Classic/Raytraced sampling and minimum set to 256 samples|
So, in conclusion, it seems that Render To Texture will only sample Normal Maps at 1 sample per pixel when rendering with Mental Ray leaving you with poorly anti-aliased images.
Solution: render at a higher resolution (2048 or 4096) and downsize the image as needed in Photoshop. The added render time is minimal considering that Normal Maps do not require Final Gather or Shadows so you can turn them off before rendering. And you can set the Sampling quality to minimum=1 and maximum=1:
|Normal Map rendered at 2048 x 2048 downsized to 512 x 512. Sampling =1 with no Shadows or Final Gather|