Dear Masih,
Your model appears to be either hydrostatically or numerically unstable in one of the platform-rotation degrees of freedom (DOFs) e.g. roll, pitch, or yaw. Is the metacentric height for your system positive? I suggest that you eliminate DOFs to debug e.g. is your model hydrostatically stable with only the platform-surge and -heave DOFs? What happens when you add the platform-pitch DOF?
Best regards,