Parsing face images into their constituent parts (e.g., eyes, nose, lips) is an important task in many computer vision applications (e.g., face recognition). Unfortunately, automating this task is difficult in practice due to the wide variety of conditions in the real world. For example, current approaches often fail on non-frontal faces and faces with exaggerated expressions. To better highlight and address real-world challenges, researchers have introduced several "in-the-wild" face datasets in recent years. However, each dataset typically has its own set of facial landmark definitions. As a result, models trained on one dataset often cannot be evaluated on others, and inconsistencies between datasets make it difficult to train robust models on more than one dataset. Another limitation of current approaches is that, with few exceptions, they parse face images using landmarks or contours, which have limited representational power. To address these limitations, this dissertation proposes an exemplar-based approach to face image parsing that models shape and appearance in a nonparametric way. First, a facial landmark localization algorithm is introduced that combines a nonparametric model of the local appearance context of each landmark with an exemplar-based shape regularization technique. Second, this algorithm is extended to address the problem of automatically transferring landmark definitions across different face datasets. The result is a large dataset in which each image includes a union of all landmark types as output. Third, a pixel-wise labeling algorithm is introduced to parse face images into their constituent parts using a soft segmentation. This dissertation will show that all three algorithms are able to parse challenging in-the-wild face images with state-of-the-art accuracy. The results suggest that an exemplar-based approach that models face shape and local appearance in a nonparametric way is (1) flexible enough to parse faces according to landmark- and segment-based representations, (2) can be used to combine different face datasets, and (3) is well-suited to parsing faces depicted in challenging real-world conditions.