How can you be sure its the faces ?
I've read that any graphics you have in game only affect the initial loading speed of the game and not in game itself.
Try removing the facepack then seeing if they still take a while to load, could be your pc specs or graphics card