Jarvis was born in Cheltenham, Gloucestershire, the son of Denys Harry Jarvis and Margot Lillian Scottney, and grew up in South Norwood and Sanderstead, South Croydon. Jarvis was educated at Whitgift School, an independent school in Croydon in south London, and at the Royal Academy of Dramatic Art.
He married Rosalind Ayres on 23 November 1974 in Ealing and has two sons by a previous marriage. He met Ayres when playing in Hamlet together, where she played Ophelia. Together with his wife, Jarvis runs the radio production company "Jarvis & Ayres Productions", frequently used by Radio 4. He was awarded the OBE in 2000. He has also published a book of memoirs titled Acting Strangely: A Funny Kind of Life (ISBN 0413728501 hardback, ISBN 0-413-74550-3 paperback). He has houses in London and Los Angeles.