Statistical research involves drawing inference about unknown quantities (e.g., parameters) in the presence of randomness in which distribution assumptions of random variables (e.g., error terms in regression analysis) play a central role. However, a fundamental issue of preserving the distribution assumptions has been more or less ignored by many inference methods and applications. As a result, the further inference of studied problems and related decisions based on the estimated parameter values may be inferior. This paper proposes a continuous distribution preserving estimation approach for various kinds of non-regular and regular statistical studies. The paper establishes a fundamental theorem which guarantees the transformed order statistics (to a given marginal) from the assumed distribution of a random variable (or an error term) to be arbitrarily close to the order statistics of a simulated sequence of the same marginal distribution. Different from the Kolmogorov-Smirnov test which is based on absolute errors between the empirical distribution and the assumed distribution, the statistics proposed in the paper are based on relative errors of the transformed order statistics to the simulated ones. Upon using the constructed statistic (or the pivotal quantity in estimation) as a measure of the relative distance between two ordered samples, we estimate parameters such that the distance is minimized. Unlike many existing methods, e.g., maximum likelihood estimation, which rely on some regularity conditions and/or the explicit form of probability density function, the new method only assumes a mild condition that the cumulative distribution function can be approximated to a satisfied precision. The paper illustrates simulation examples to show its superior performance. Under the linear regression settings, the proposed estimation performs exceptionally well regarding preserving the error terms (i.e., the residuals) to be normally distributed which is a fundamental assumption in the linear regression theory and applications. (Joint work with Bingyan Wang)