Dear Hadi,
The nonlinear response is not surprising to me and likely depends on the shaft tilt, deflections, and gravity loads. I would not expect you’d match this exactly with the linear model, but the large difference in amplitude you are showing is concerning.
Regarding your plots of the C and S components, where is the large nonzero mean coming from? If the output equation is written as y = Cx + Du and the states and inputs are initialized to zero (are they?), why would the value at time zero be nonzero? Where have you added the operating point (OP) values to the perturbations derived from the linear model output?
Best regards,