The only daughter of James Fox and Mary Elizabeth Piper, she has four brothers, including the actors Laurence and Jack Fox. Her father's parents were the theatrical agent Robin Fox and the actress Angela Worthington, for whom Noël Coward wrote the song "Don't Put Your Daughter on the Stage, Mrs Worthington". The Worthingtons did not have any daughters; their two granddaughters, Lydia included, are actresses.Fox was educated at the University of Cambridge. While there, in 2001 she played Juliet in a Marlowe Society production of Romeo and Juliet, opposite Tom Hiddleston, to be reviewed as "a slightly nonchalant Juliet".On 8 September 2007 Fox married the actor, writer, director and comedian Richard Ayoade. They have two daughters, Esme Bibi (born 2009) and Ida.