From the French olivier, meaning "olive grower/merchant", introduced into Britain by the Normans.

In old documents in Latin, the form Oliverus may be found, with appropriate case endings.