We present a novel vector filtering technique for color image restoration that incorporates a new fuzzy inference system for noise detection. This is combined with a switching scheme to select between an identity filter output and the output from a proposed L-filter design. The proposed L-filter is designed to exploit the ordering techniques of the vector median filters, and thus it requires only a set of two coefficients. These coefficients are trained using a constrained least-mean squares approach, which is capable of converging to the optimum set within a short period of time. The new algorithm treats the intensity and color of each pixel individually until the final output is to be calculated, thus, the optimal magnitude and direction of the pixel vectors are used.