Taking a closer look at the building blocks of Large Language Models