Achieving optimal design of phase-locked loop (PLL) is a major challenge in WiMax technology in order to improve system behavior against noise and to enhance Quality of Service (QOS). A new loop filter design method for phase locked loop (PLLs) is introduced taking into consideration various design objectives: small settling time, small overshoot and meeting Mobile WiMax requirements. Optimizing conflicting objectives is accomplished via linear programming and semidefinite programming (especially Linear Matrix Inequality (LMI)) in conjunction with appropriate adjustment of certain design parameters. Digital filters, Infinite Impulse Response (IIR) and Finite Impulse Response (FIR) are designed using linear programming and convex programming.Simulations show that IIR digital lowpass filter with narrow transition band could not work properly with mobile WiMax system. Simulations show that FIR digital lowpass filter utilizing linear programming managed to improve the transient behavior. The FIR digital lowpass filter utilizing semidefinite programming (LMI) will much improve the transient behavior, therefore it is recommended for mobile WiMax systems.