Born in Lake Charles, Louisiana in 1923, Bishop Olivier was the fifth of eight children. He attended Sacred Heart of Jesus grade school in Lake Charles and then St. Augustine Seminary in Bay St. Louis, Mississippi and St. Mary's Seminary in Techny, Illinois. He was ordained a Divine Word Missionary in Bay St. Louis in 1951.
From 1951 to 1973, he was assistant dean, then dean of seminarians and, finally, rector of the religious community. From 1974 to 1982, he was Secretary of Studies for all United States Divine Word seminaries and rector of the religious community at Divine Word Seminary in Epworth, Iowa.
St. Anthony's parish in Lafayette, Louisiana was his first pastorate. In the last two years of that assignment, then-Father Olivier also was part-time Vicar for Black Catholics in the Diocese of Lafayette. He was named full time Vicar in 1986, a position he held until appointed Auxiliary Bishop of Washington by Pope John Paul II on November 7, 1988.
Ordained bishop on December 20, 1988, Bishop Olivier has served at different times as Regional Bishop of Deaneries in the District of Columbia, Prince George's County and Southern Maryland. He is a Fourth Degree Knight of St. Peter Claver, a Knight of St. John and a Knight of Columbus.
He served as the convenor of the African American Catholic Bishops and as the Episcopal liaison for the National African American Catholic Youth Ministry Network, and has served on a number of committees for the United States Conference of Catholic Bishops.