Begin comes from the Old English beginnan, which was just begin with a soft G sound. It actually wasn’t used that much; the word onginnan was used instead, and now I want to know why we switched! But that’s going to have to stay a mystery because I didn’t see a reason why. Anyway, beginnan is be- + -ginnan. The be- is like how you’d call something “besotted” or “bedeviled”, while -ginnan is West Germanic in origin, although it’s another mystery as to where it comes from. -Ginnan is one of those suffixes that is part of words but never exists on its own.
Start showed up as a noun in the late fourteenth century and as a verb sometime before that, but back then it only had to do with jumping suddenly. It didn’t start meaning to begin until the seventeenth century, and it’s thought that it comes from the idea of startling an animal out of its lair. As to its origins, start comes from the Old English steortian/stiertan/styrtan (it depends on which dialect you look at), which comes from the Proto Germanic stert-, which might be from the Proto Indo European ster-, which means… stiff. No, I have no idea how that happened.
Birth showed up in the thirteenthcentury and it’s thought to be Scandinavian in origin. It’s related somehow to the Old Norse byrðr, which is pronounced something like “birther” and comes from the Proto Germanic gaburthis, which is actually the origin of the Old English word that birth replaced, gebyrde. Gaburthis, which is a funny word to say for some reason, can be traced back to the Proto Indo European bhrto, the past participle of bher-, or to bear.
Initial showed up in the sixteenth century meaning relating to a beginning (initial as in short for a word came a century later). It comes from the Middle French initial and classical Latin initialis, which is just initial and is the past participle of inire, which is something like to begin or to go into. The in- is in (duh) and the ire means to go here although it’s more commonly known as anger. Ire can also be traced back to the Proto Indo European ei-, to go, another word piece that shows up in everything. I’m not getting into it all right now. Just trust me on this one.
Tony Jebson’s page on the Origins of Old English