By ever increasing application fields wherein robots are being deployed the need for high speed performance, more precision, decreased the energy consumption, the researchers and manufacturers were both encouraged to design and implement lighter manipulators. Decreasing the weight of the manipulators caused flexibility while at the same time resulted in vibration problem. The flexible-link is an infinite dimension system. To design controller a reduced order model of system is employed. There are unmodeled dynamics, some other uncertainties and external disturbances, which affect the closed loop performance and must be eliminated. Sliding mode controller is a well-known nonlinear robust controller. The equivalent control method is a conventional way to design this controller, usually based on linear switching surface. The sliding surface equation has a great rule in controller performance. To eliminate high frequency oscillations a continuous approximation of hard switch will be used in boundary layer. In this case the equation of switching surface affects the controller performance directly in boundary layer. In the other hand increasing the number of nonlinear terms in dynamic equations causes higher order terms will be needed in sliding surface to reach a better performance. Increasing the sliding surface degree will affect some performance indices such as domain of attraction. At the first step neural networks will be used to adjust two main parameters of smooth switch, slow rate and amplitude. Because loss of stability prove and a few degrees of freedom, this method was not distributed. By assuming these important points and for modification of the sliding mode controller, a. general_. approach to design nonlinear sliding surface for systems in regular form will be presented. To improve the stability conditions and optimize a conventional cost function, the nonlinear optimal approach was mixed with sliding mode method. To prove the stability of sliding phase the global asymptotic stability idea was used to design the equivalent control signal. All of previous steps were proved analytically. At last, the expansion of domain of attraction of closed loop system by increasing the order of the surface is verified by simulation results. The nonlinear surface decreases the level of cost function in all of simulation cases. To solve mathematics equations, the Maple software was used. Simulations was done in Matlab and Simulink environments. The thesis text is typed in FarsiTEX that is the Farsi version of the LATEX. The great parts of figures were plotted by LATEXcad.