Birth of Christ

Hi all. So I have recently begun trying to study through the birth of Christ as it is related in Matthew and Luke. The two accounts are presented from different standpoints for different reasons, so this has been a bit of a struggle. There is a verse in Luke 2 that says that the family, Joseph, Mary, and Jesus, returned to Jerusalem every year for the Passover. However, during the early part of Jesus' life the family lived in Egypt. I am just wondering if it's accurate that the family traveled from Egypt to Jerusalem every year for the Passover, or if this is an interpretation issue.
My basic time line is...
1. Jesus is born in Bethlehem, shortly outside of Jerusalem.
2. At 8 days old Jesus is circumcised in Jerusalem.
3. Family flees to Egypt to avoid Herod.
4. Family returns to Jerusalem every year until Herod dies (probably 2-5 years)
5. Family moves to Nazareth to avoid Herod's prodigy.
As I understand it, Herod didn't trust his kids, so his kids wouldn't be nearly the same level of threat to Jesus' life. The two Gospels also phrase the family's return as that they were afraid of the persecution in Jerusalem, so they instead went to Nazareth. Does this mean that they were planning to move to Jerusalem after Jesus' birth? My understanding of Biblical wording is weak at best. Any advice you can offer would be appreciated. I tried googling this topic, but the information I found was all written by unbelievers. Thanks again, Helen