Dear Yingyu,
I recall getting that same error about the inertia tensor being ill formed, but don’t remember what caused it. I suggest you ask MSC Technical Support if you are concerned. The aggregate mass result out of ADAMS looks reasonable to me.
But the signs in your transformation from the nacelle coordinate system to the center-of-mass are not correct. See e.g. my post dated Nov 23, 2009 in the following forum topic for an example where I show the proper transformation: Tower fore-aft modes shapes (this post uses the WindPACT 1.5-MW turbine instead of the NREL 5-MW turbine, but the transformations are the same). My guess is you’ll match my results much better if you use the proper signs.
You didn’t actually state the error you receive when running with the ru-user command in ADAMS, but regardless, ADAMS-related questions are better answered by MSC Technical Support.
Best regards,