Scoring facial attractiveness with deep convolutional neural networks: How training on standardized images reduces the bias of facial expressions

Dorothea Obwegeser; Radu Timofte; Christoph Mayer; Michael M Bornstein; Marc A Schätzle; Raphael Patcas

doi:10.1111/ocr.12820

. 2024 Jun 2;27(Suppl 2):25–32. doi: 10.1111/ocr.12820

Scoring facial attractiveness with deep convolutional neural networks: How training on standardized images reduces the bias of facial expressions

Dorothea Obwegeser ¹, Radu Timofte ^2,³, Christoph Mayer ², Michael M Bornstein ⁴, Marc A Schätzle ¹, Raphael Patcas ^1,^✉

PMCID: PMC11654357 PMID: 38825845

Abstract

Objective

In many medical disciplines, facial attractiveness is part of the diagnosis, yet its scoring might be confounded by facial expressions. The intent was to apply deep convolutional neural networks (CNN) to identify how facial expressions affect facial attractiveness and to explore whether a dedicated training of the CNN is able to reduce the bias of facial expressions.

Materials and Methods

Frontal facial images (n = 840) of 40 female participants (mean age 24.5 years) were taken adapting a neutral facial expression and the six universal facial expressions. Facial attractiveness was computed by means of a face detector, deep convolutional neural networks, standard support vector regression for facial beauty, visual regularized collaborative filtering and a regression technique for handling visual queries without rating history. CNN was first trained on random facial photographs from a dating website and then further trained on the Chicago Face Database (CFD) to increase its suitability to medical conditions. Both algorithms scored every image for attractiveness.

Results

Facial expressions affect facial attractiveness scores significantly. Scores from CNN additionally trained on CFD had less variability between the expressions (range 54.3–60.9 compared to range: 32.6–49.5) and less variance within the scores (P ≤ .05), but also caused a shift in the ranking of the expressions' facial attractiveness.

Conclusion

Facial expressions confound attractiveness scores. Training on norming images generated scores less susceptible to distortion, but more difficult to interpret. Scoring facial attractiveness based on CNN seems promising, but AI solutions must be developed on CNN trained to recognize facial expressions as distractors.

Keywords: artificial intelligence, deep convolutional neural networks, emotions, facial attractiveness, facial expressions

1. INTRODUCTION

The analysis of the face is a popular field of research in dentistry and medicine. Specifically, facial attractiveness is a key outcome measurement in many disciplines such as orthodontics, plastic surgery, dermatology or maxillofacial surgery. Medical sciences have thus since long endeavoured to deduce information contained in images of the human face, lately with help of machine learning. Yet determining facial attractiveness automatically remains a challenging pattern recognition problem. ¹ Although facial beauty is certainly an inherently subjective attribute, it is evident that specific shapes of the face are ubiquitously perceived as beautiful ² and that the aesthetic judgement often align in consensus among groups. ³ This last fact is the foundation in the approach to use big data for data‐driven perception of facial attractiveness in medical sciences in the quest to obtain more objective attractiveness scores. ⁴ , ⁵

The general trend to apply artificial intelligence (AI) solutions to medicine, and more specifically to diagnosis and treatment planning, has affected the intent to appraise the face to a great measure, too. ⁶ , ⁷ The last decade has witnessed the introduction of several commercially available AI solutions aiming to appraise the face based on photographic images, allowing not just the recognition of a face but also a detailed evaluation of its traits, such as the prediction of gender, ⁸ age, ⁴ , ⁹ , ¹⁰ , ¹¹ , ¹² attractiveness, ⁵ , ¹⁰ , ¹³ mood and emotions ¹⁴ , ¹⁵ , ¹⁶ or race and ethnicity. ¹⁷ , ¹⁸ Some of these AI solutions are enjoying widespread popularity on social media and are commercially available. ⁷ Based on their popularity and accessibility, and certainly also because of their proven robustness and accuracy, ¹⁷ an increasing amount of scientific contributions have, sometimes in no uncertain terms, advocated their use in a medical setting. ⁸ , ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁸ , ¹⁹

One of the most elementary problems in AI‐based facial recognition is to ensure that all confounders that might affect the feature extraction (and as such distort the prediction) are recognized and removed. Certain confounders can easily be revealed as distractors (e.g. background) and their impact be moderated by training the network to exclude clues outside the region of interest. Other confounders can be more challenging to tackle. One of the more serious persisting limitations is that facial attractiveness can be affected by facial expressions. As case in point, happiness increases facial attractiveness. ²⁰ Moreover, every facial expression will affect facial attractiveness differently. ²¹ This is particularly disturbing in evaluating the attractiveness of facial images for medical purposes, where facial expressions are considered transient factors which must be excluded as they may frustrate the analysis.

While it remains uncontested that facial analysis in orthodontics should be performed on standardized images, the broader aim of this study was to ascertain whether it is acceptable to apply AI solutions that are trained and tested on unconstrained data.

More specifically, the objective of this study was to identify how an AI‐based facial attractiveness scoring system derived from deep convolutional neural networks trained on unconstrained images is affected by different facial expressions, and whether training the model on standardized photographs from a norming dataset would achieve a reduction of bias owed to facial expressions. The null hypothesis was that training could not alter the attractiveness scores of the face under different facial expressions.

2. MATERIALS AND METHODS

This analysis was performed on frontal facial images of 40 voluntary female test persons. The participants were either dental students, dentists or dental nurses from the Center of Dental Medicine of the local university, all aged between 20 and 30 years (mean age: 24.5 years).

Before taking the portrait photos, participants received all pertinent information in form of a flyer and gave their written informed consent, especially agreeing to the use of their facial images for research purpose and publication in a scientific journal. The photographs were taken under standardized conditions. The participants were standing in front of a monochrome dark blue background, facing the photographer at a two‐meter distance. A single‐lens reflex camera with a dedicated flash and fixed settings (aperture F11, shutter speed 1/200, ISO 200) was used for all of the pictures.

From each participant, three sets of 7 pictures (n = 840) were taken with varying facial expressions (Figure 1), including a neutral expression and the six universal facial expressions: disgust, sadness, happiness, fear, anger, surprise. ²² At the photo shooting session, the participants first had to learn the required facial expressions. They were shown a representative chart of every facial expression and had to impersonate them in front of a mirror. Three pictures were taken of each facial expression. Between the shots, the participants enjoyed a small pause in order to relax their face muscles. The photos were not post‐processed and only aligned and cropped to an equal size of 256 × 256 pixels, to correct face alignment and to reach an equal 40% background margin.

Sample images of a participant showing the seven standardized facial expressions analysed in this study. From left to right: (top row) neutral, disgust, sadness, happiness, (bottom row) fear, anger, surprise. Consent for publication was obtained from this individual.

The facial attractiveness of every photograph was assessed based on artificial intelligence (AI), ²³ comprising a face detector ²⁴ and deep convolutional neural networks (CNN) constructed with DenseNet‐201 architecture for image recognition. ²⁵ In contrast to other CNN models, DenseNet‐201 shortens the connections between layers close to the input and those close to the output to allow a maximum information flow between layers in the network and achieves a more efficient, more accurate and overall enhanced feature recognition.

The CNN model was pre‐trained for image classification on ImageNet dataset ²⁶ and further trained for attractiveness assessment in two different ways: an algorithm trained for attractiveness in real world (‘UNCONTROLLED’) and a fine‐tuned algorithm to medical‐imaging conditions (‘STANDARDIZED’). The UNCONTROLLED attractiveness score is based on training the CNN on the BLINQ dataset with unconstrained facial images and corresponding ratings in the BLINQ dating app. ²³ The dataset of the dating site (BLINQ) contains more than 13 000 facial profile images (mostly Caucasians, age range: 18–37 years, mean age: 25 years) taken in uncontrolled environments, annotated with more than 17 million ratings for attractiveness.

To improve the suitability to medical conditions, the entire above‐mentioned CNN model trained on BLINQ dataset was further trained (i.e. fine‐tuned) with pictures from the Chicago face database (CFD). ²⁷ This dataset contains standardized facial photographs for norming data in a research context. As such, the second (STANDARDIZED) model used the BLINQ dataset for pretraining to learn features that are relevant to face‐based attractiveness judgement in a more general and unconstrained context, and CFD was used to further train those facial feature predictions on data with controlled conditions and labelling. A subset with neutral frontal face images of males and females (n = 158; ages range: 18–40 years), taken in a controlled setting, using a monochrome background, equal exposure to light and avoiding any possible disturbances was selected. Training was done on an ensemble of ten different models with different train‐validation splits but with the same test set. Fine‐tuning was done for 24 epochs on the pre‐trained model with the same number of classes and loss function. The ensemble averages the attractiveness assessment of all models.

For inference, both models were used and images were augmented ten times for each model, leading up to 100 predictions per image which were then respectively averaged. Test‐Time Augmentation was performed to complement model ensembling in order to achieve an improved prediction quality. Each facial photograph taken in this present study (n = 840) received two attractiveness scores (‘UNCONTROLLED’ based on unconstrained images, and ‘STANDARDIZED’ based on an algorithm fine‐tuned on controlled data of the CFD). The attractiveness scores ranged from 0 to 100 (0 = extremely unattractive, 100 = extremely attractive) and were determined based each on the above‐mentioned 100 averaged predictions of both CNN pipelines separately and subsequently averaged for every facial expression of each participant.

2.1. Ethics

The protocol of the study was reviewed by the governmental ethics committee of the canton of Zurich prior to the execution of the study and a formal waiver was issued (BASEC Req. 2019‐0030). Guidelines in medical ethics, as stipulated in the WMA Declaration of Helsinki (64th WMA General Assembly, Fortaleza, Brazil, October 2013), were strictly followed, along with adherence to all relevant juridical regulations.

2.2. Statistical analysis

Data were analysed in SPSS software (IBM SPSS Statistics for Windows, version 24 [IBM Corp., Armonk, NY, USA]). The attractiveness scores (overall and for each facial expression separately) derived from both the UNCONTROLLED CNN and the STANDARDIZED CNN were descriptively reviewed. Exploratory analyses of the scores were performed, including box‐and‐whisker plots for each individual score and a bar diagram for variance. Levene's tests were applied to compare the homogeneity of the variables.

Neutral facial expression was established as reference value, and mean differences in attractiveness score for all facial expressions were calculated. Kolmogorov–Smirnov tests were computed to verify the normal distribution of the differences, together with Student's t‐tests for paired samples to disclose statistical differences. P‐values of <.05 were considered significant.

3. RESULTS

The direct comparison between the CNN trained solely on unconstrained data (referred to as UNCONTROLLED) and the CNN further trained on norming data of the CFD to make the CNN medically useful (referred to as STANDARDIZED) revealed that the attractiveness scores differed for all facial expressions, both in general terms and for each individual facial expression (Figure 2). Overall, the medical‐trained algorithm contained a clear bias towards a higher attractiveness score (UNCONTROLLED overall mean: 40.9 [standard deviation (SD) 17.1]; STANDARDIZED overall mean: 56.4 [SD 10.3]).

Box and whisker plots for attractiveness score (0–100) of different facial expressions: Social media‐trained AI (UNCONTROLLED, in red) versus retrained AI on CFD images (STANDARDIZED, in blue).

When descriptively reviewed, the results indicate that CNN trained on the CFD norming data yielded not only a higher attractiveness score for each facial expression but also demonstrated that CNN trained on the CFD norming data scores had less variability between the expressions (Figure 2) and less variance within the scores (Figure 3). Perhaps interestingly, according to the algorithm further trained on standardized photographs, all facial expressions were less attractive than neutral expression (mean: 60.9), including happiness (mean 57.4), while for UNCONTROLLED happiness resulted in higher attractiveness score than neutral expression.

Bar diagram for variances of the variables.

UNCONTROLLED and STANDARDIZED mean attractiveness scores of all facial expressions were tabled together with their variance and juxtaposed with Levene's‐tests, demonstrating in all but one expression statistically significant differences in the homogeneity of the variables (Table 1).

TABLE 1.

Mean, standard deviation and variance of all facial expressions, according to the training used.

Facial expression	N	Mean (SD)	Variance	Levene's‐test for homogeneity
Neutral
UNCONTROLLED	120	43.4 (18.9)	356.6	<0.001
STANDARDIZED	120	60.9 (9.1)	82.3	<0.001
Disgust
UNCONTROLLED	120	32.6 (14.9)	222.3	0.052
STANDARDIZED	120	50.5 (11.6)	135.5	0.052
Sadness
UNCONTROLLED	120	38.7 (18.8)	354.4	<0.001*
STANDARDIZED	120	55.6 (10.5)	110.3	<0.001*
Happiness
UNCONTROLLED	120	49.5 (17.8)	318.3	<0.001
STANDARDIZED	120	57.4 (10.6)	111.4	<0.001
Fear
UNCONTROLLED	120	41.7 (15.5)	240.2	<0.001
STANDARDIZED	120	58.4 (8.9)	79.6	<0.001
Anger
UNCONTROLLED	120	35.7 (18.4)	338.9	<0.001
STANDARDIZED	120	57.5 (11.0)	121.9	<0.001
Surprise
UNCONTROLLED	120	44.7 (15.6)	244.6	<0.001*
STANDARDIZED	120	54.3 (10.2)	103.8	<0.001*

Open in a new tab

Note: Levene's test to disclose statistical differences in homogeneity of the variables. UNCONTROLLED: CNN trained on social media data. STANDARDIZED: CNN trained on social media data and retrained on standardized images (CFD) used to norm data in a medical context.

Differences significant at P < .05.

Lastly, the mean differences in attractiveness score between each facial expression and neutral expression were listed (Table 2). These differences followed normal distribution (Kolmogorov–Smirnov for all: >0.05) and were investigated with Student's t‐tests for paired samples. For STANDARDIZED, all facial expressions had a statistically significant impact on the attractiveness scores. For UNCONTROLLED, however, only disgust, happiness and anger had a statistically significant impact.

TABLE 2.

Impact of the various facial expressions on attractiveness as difference to neutral expression (as reference): Mean, standard deviation, P‐value based on Student's t‐test for paired samples.

Difference to neutral	UNCONTROLLED	STANDARDIZED	UNCONTROLLED	STANDARDIZED
Difference to neutral	Mean (SD)	Mean (SD)	P‐value	P‐value
Disgust	−10.9 (13.1)	−10.4 (8.9)	<.001^*	<.001^*
Sadness	−4.7 (9.7)	−5.4 (4.9)	.054	<.001^*
Happiness	6.0 (7.0)	−3.6 (5.8)	.011^*	.005^*
Fear	−1.8 (8.5)	−2.6 (4.0)	.429	.027^*
Anger	−7.8 (9.9)	−3.5 (5.9)	.001^*	.008^*
Surprise	−1.4 (10.5)	−6.7 (7.6)	.548	<.001^*

Open in a new tab

Note: UNCONTROLLED: CNN trained on social media data. STANDARDIZED: CNN trained on social media data and retrained on standardized images (CFD) used to norm data in a medical context.

Differences significant at P < .05.

4. DISCUSSION

This investigation reveals that facial expressions can affect the scores of attractiveness derived from deep convolutional neural networks. It also demonstrates that a retraining on standardized images may reduce the variance of the result and as such could lessen the impact of facial expressions on the scores.

Since the first attempts to apply machine learning to facial attractiveness, ² , ²⁸ major improvements have been made. Given the high dimensionality and redundancy of visual data, the task of learning facial attractiveness, however, remains undoubtedly a challenging one. As outlined in the introduction, many disciplines in medicine could benefit from an AI‐based diagnostic tool to reliably establish facial attractiveness, in order to decide on treatment objectives, assess treatment need, or determine therapy outcome.

Surprisingly, the effects of facial expressions on facial attractiveness have been little explored. ²¹ , ²⁹ , ³⁰ Facial expression presents itself in discrete changes of fiducials of the face surface. As an example, in a surprised face, the eyelids are opened, the white of the eye is showing above and below the pupil, the eyebrows are raised and curved, the skin below the brow is stretched, horizontal wrinkles show across the forehead, the jaw drops open and teeth are parted without tension or stretching of the mouth. In machine learning, the face is represented in various ways, either as a whole unit (holistic representation), as a set of features (analytic representation) or as a combination of these (hybrid approach). Independent of the approach, the notable changes of any facial expressions will always be part of the data extracted and processed. Within the context of medical imaging, these facial expressions act as momentary distractors which are capable of influencing the attractiveness score. ²⁹ This study aimed therefore to identify how an AI‐based facial attractiveness scoring system derived from deep convolutional neural networks trained on unconstrained data – and its fine‐tuned derivate trained for medical imaging – were affected by different facial expressions. We hypothesized that retraining the model on standardized images could not affect the attractiveness score of a face displaying different facial expressions.

Retraining the algorithm on a set of additional data to reduce an identified bias is a common and accepted approach. ³¹ Here, CNN was retrained on neutral, standardized reference images commonly used for norming data in a research context, which resulted in scores with less variability between the expressions and less variance within the scores. Specifically, the significant outcome of the Levene's test demonstrates conclusively that the scores have different variances, and the null hypothesis is to be rejected. The results therefore indicate that fine‐tuning the model on medical reference images solidified the scoring to be less susceptible to different facial expressions, a positive feature when assessing facial physiognomy for attractiveness with a diagnostic purpose. With less variance and more consistent results, the algorithm seems more robust and more appropriate for intra‐individual comparison, which is of key importance in medical imaging.

In statistical terms, the ramification of smaller variances is a reduction of the uncertainty, which in turn produces more significant results in inferential statistical analyses. Hence, the STANDARDIZED data yielded more significant results and enabled to disclose all the assessed changes in facial expressions from ‘neutral’ as statistically significant differences. Thus, the trained algorithm was less vulnerable to the impact of facial expressions in absolute terms of the scoring results, but the slight shifts that nevertheless occurred became all statistically significant.

Apparently, the benefits of training the algorithm on norming images come with a trade‐off. Besides the fact that the scores are skewed towards overall increased attractiveness (a drawback that could easily be corrected through re‐dimensioning the scale), the scores differed in the rank of absolute scores of the facial expressions. This is a fundamental observation which bears important consequences. According to fine‐tuned STANDARDIZED algorithm, all facial expressions were less attractive than neutral, including happiness. On the other hand, the ranking by UNCONTROLLED algorithm was clearly more in line with common perception. ²¹ Data‐driven AI generates an output in a purely computational manner, unable to disclose the decision‐making process in a medically acknowledgeable format and remains not conducive to verification. ³² The lack of interpretability and transparency is, therefore, especially frustrating in cases as the present, where the absolute ranking of the different facial expression does not correspond to empiric evidence. Nonetheless, for the ‘favouritism’ of neutral faces, the following reasoning can be submitted: By training specifically the CNN on norming data, i.e. standardized neutral faces, the algorithm learned the difference between a neutral face and an expressive face. It learned to find a neutral expression attractive and classified all facial expression not conforming to a neutral posture as less attractive. The attractiveness here is therefore a normalized attractiveness for neutral faces, the effect of the facial expressions is unknown to the trained predictor.

The fact that the STANDARDIZED‐CNN produces scores that are different from UNCONTROLLED‐CNN in images with facial expressions does not disqualify this algorithm as a diagnostic tool. On the contrary, the more precise results are based on physiognomy rather than facial expression. Moreover, the unconstrained environment of UNCONTROLLED‐CNN contained ‐ apart from facial expression ‐ several other variables that may have influenced the attractiveness scoring algorithm, such as lighting conditions, distractors such as hairband, jewellery or glasses, and facial expression. ³³ As such, the STANDARDIZED‐CNN model is closer to the ideal system outlined by Pantic and Rothkrantz, which, in order to be universal, should perform robustly despite changes due to distractors. ³³ , ³⁴ Being cognizant that any facial expression may impact the scoring, each algorithm should be applied for its intended use. So while UNCONTROLLED‐CNN will remain the tool it was designed for, i.e. a scoring system for societal attractiveness, which takes facial expression and other predictors into account (there is no objection to pose with an exultant smile on social media), the STANDARDIZED‐CNN is incontestably the more appropriate tool for attractiveness scoring for medical diagnostics, where standardized neutral faces are assessed.

The ramifications of the presented results are serious, as they underline the unsuitability of available commercial AI solutions for facial assessment within a medical setting and challenge the methodology of many published studies. The present study offers a possible solution to this problem by demonstrating the two‐step approach of retraining unconstrained big data‐based CNNs on norming data to be a promising path.

In a broader context, this study also sheds light onto the fact that the impact of facial expression on attractiveness is not a self‐evident inherent quality deducible from facial features, but is rather a functional assessment governed by evolution and shaped by societal agreement. ³⁵ Perhaps a truism, but the effect of facial expression on attractiveness can never be learned by observing neutral faces. So while many recent works in image analysis and pattern recognition aim at automatic detection of emotional facial signals. ¹⁴ , ¹⁵ , ¹⁶ , ³⁶ Few if none explore the opposite, namely deflating any emotional signals from the prediction in order to score facial attractiveness purely on physiognomy. As such, the presented approach is novel and may have wider implications, as it affects all attempts of applying AI solutions to assess the face in a medical context.

4.1. Limitations

Given the evidence that CNNs perform better when trained on pooled data from different sources, ³⁷ caution should be applied prior considering unrestrained deployment of the presented systems without first assessing its generalizability.

Another limitation of the present study might be the use of 6 categories of facial expression as described by Ekman. ²² It is true that the universality of these categories has been challenged by some authors. ³⁸ , ³⁹ Nevertheless, we did not refrain from using this labelling system, since most studies on vision‐based facial expression analysis rely on it and its application allows direct comparison, and also because the question whether these labels are truly universal or not is not of this investigation's concern.

5. CONCLUSION

Juxtaposing the scores of a CNN for facial attractiveness based on data with uncontrolled environment to a model retrained on normative images, it is apparent that the scores cannot be directly compared. Retraining the model on standardized images used in a medical context had the beneficial effect of reducing the variance of the result, thereby lessening the impact of facial expressions on the scores and providing results that are more robust. It conclusively demonstrates that the architectural design of commercially available software for societal attractiveness is unsuitable for medical diagnostics and therapy assessments.

This study furthermore highlights the dire necessity to interpret any retraining and ultimately validate any model employed for medical purpose, owing to the lack of transparency in the AI decision‐making process.

FUNDING INFORMATION

This study was supported with a grant of the Swiss Association of dentomaxillofacial radiology (SADMFR, grant number 19/01).

CONFLICT OF INTEREST STATEMENT

The authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

ACKNOWLEDGEMENT

Open access funding provided by Universitat Zurich.

Obwegeser D, Timofte R, Mayer C, Bornstein MM, Schätzle MA, Patcas R. Scoring facial attractiveness with deep convolutional neural networks: How training on standardized images reduces the bias of facial expressions. Orthod Craniofac Res. 2024;27(Suppl. 2):25‐32. doi: 10.1111/ocr.12820

DATA AVAILABILITY STATEMENT

A code repository providing pre‐trained models and codes for reproducibility of results and attractiveness predictions is publicly available at https://github.com/2006pmach/facial_attractiveness_prediction.

REFERENCES

1. Zhai Y, Cao H, Deng W, Gan J, Piuri V, Zeng J. BeautyNet: joint multiscale CNN and transfer learning method for unconstrained facial beauty prediction. Comput Intel Neurosc. 2019;2019:1910624. [DOI] [PMC free article] [PubMed] [Google Scholar]
2. Eisenthal Y, Dror G, Ruppin E. Facial attractiveness: beauty and the machine. Neural Comput. 2006;18(1):119‐142. [DOI] [PubMed] [Google Scholar]
3. Laurentini A, Bottino A. Computer analysis of face beauty: a survey. Comput Vis Image Und. 2014;125:184‐199. [Google Scholar]
4. Patcas R, Bernini DAJ, Volokitin A, Agustsson E, Rothe R, Timofte R. Applying artificial intelligence to assess the impact of orthognathic treatment on facial attractiveness and estimated age. Int J Oral Max Surg. 2019;48(1):77‐83. [DOI] [PubMed] [Google Scholar]
5. Patcas R, Timofte R, Volokitin A, et al. Facial attractiveness of cleft patients: a direct comparison between artificial‐intelligence‐based scoring and conventional rater groups. Eur J Orthodont. 2019;41(4):428‐433. [DOI] [PubMed] [Google Scholar]
6. Mohammad‐Rahimi H, Nadimi M, Rohban MH, Shamsoddin E, Lee VY, Motamedian SR. Machine learning and orthodontics, current trends and the future opportunities: a scoping review. Am J Orthod Dentofac. 2021;160(2):170‐192. [DOI] [PubMed] [Google Scholar]
7. Patcas R, Bornstein MM, Schatzle MA, Timofte R. Artificial intelligence in medico‐dental diagnostics of the face: a narrative review of opportunities and challenges. Clin Oral Investig. 2022;26(12):6871‐6879. [DOI] [PMC free article] [PubMed] [Google Scholar]
8. Chen K, Lu SM, Cheng R, et al. Facial recognition neural networks confirm success of facial feminization surgery. Plast Reconstr Surg. 2020;145(1):203‐209. [DOI] [PubMed] [Google Scholar]
9. Dorfman R, Chang I, Saadat S, Roostaeian J. Making the subjective objective: machine learning and rhinoplasty. Aesthet Surg J. 2019;40(5):493‐498. [DOI] [PubMed] [Google Scholar]
10. Khetpal S, Peck C, Parsaei Y, et al. Perceived age and attractiveness using facial recognition software in rhinoplasty patients: a proof‐of‐concept study. J Craniofac Surg. 2022;33(5):1540‐1544. [DOI] [PubMed] [Google Scholar]
11. Rezende Machado AL, Dezem TU, Bruni AT, Alves da Silva RH. Age estimation by facial analysis based on applications available for smartphones. J Forensic Odontostomatol. 2017;35(2):55‐65. [PMC free article] [PubMed] [Google Scholar]
12. Talwar AA, Desai AA, McAuliffe PB, Broach RB, Percec I. 17. Artificially intelligent facial feature quantification after facial filler injection. Plast Reconstr Surg Global Open. 2022;10(6S):11. [Google Scholar]
13. Peck CJ, Patel VK, Parsaei Y, et al. Commercial artificial intelligence software as a tool for assessing facial attractiveness: a proof‐of‐concept study in an orthognathic surgery cohort. Aesthet Plast Surg. 2022;46(2):1013‐1016. [DOI] [PubMed] [Google Scholar]
14. Boonipat T, Lin J, Bite U. Detection of baseline emotion in brow lift patients using artificial intelligence. Aesthet Plast Surg. 2021;45(6):2742‐2748. [DOI] [PubMed] [Google Scholar]
15. Boonipat T, Hebel N, Zhu A, Lin J, Shapiro D. Using artificial intelligence to analyze emotion and facial action units following facial rejuvenation surgery. J Plast Reconstr Aes. 2022;75(9):3628‐3651. [DOI] [PubMed] [Google Scholar]
16. Kuntzler T, Hofling TTA, Alpers GW. Automatic facial expression recognition in standardized and non‐standardized emotional expressions. Front Psychol. 2021;12:627561. [DOI] [PMC free article] [PubMed] [Google Scholar]
17. S‐G Jung, An J, Kwak H, Salminen J, Jansen B. Assessing the accuracy of four popular face recognition tools for inferring gender, age, and race. Proceedings of the International AAAI Conference on Web and Social Media. ICWSM. Association for the Advancement of Artificial Intelligence (AAAI); 2018:12(1). [Google Scholar]
18. Li S, Guo L, Liu J. Towards east Asian facial expression recognition in the real world: a new database and deep recognition baseline. Sensors. 2022;22(21):8089. [DOI] [PMC free article] [PubMed] [Google Scholar]
19. Gupta N, Hughes SJ, Chirwa R, Cheng Q. Facial recognition software use on surgically altered faces: a systematic review. J Craniofac Surg. 2022;33(8):2443‐2446. [DOI] [PubMed] [Google Scholar]
20. Golle J, Mast FW, Lobmaier JS. Something to smile about: the interrelationship between attractiveness and emotional expression. Cogn Emotion. 2014;28(2):298‐310. [DOI] [PubMed] [Google Scholar]
21. Morrison ER, Morris PH, Bard KA. The stability of facial attractiveness: is it what You've got or what you do with it? J Nonverbal Behav. 2013;37(2):59‐67. [Google Scholar]
22. Ekman P. Facial expression and emotion. Am Psychol. 1993;48(4):384‐392. [DOI] [PubMed] [Google Scholar]
23. Rothe R, Timofte R, Van Gool L. Some like it hot—visual guidance for preference prediction. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE); 2016:5553‐61. [Google Scholar]
24. Mathias M, Benenson R, Pedersoli M, van Gool L. Face detection without bells and whistles. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T, eds. Computer Vision–ECCV 2014. ECCV 2014. Lecture Notes in Computer Science. Springer International Publishing; 2014:720‐735. doi: 10.1007/978-3-319-10593-2_47 [DOI] [Google Scholar]
25. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE);2017:2261‐69. [Google Scholar]
26. Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge. Int J Comput Vision. 2015;115(3):211‐252. [Google Scholar]
27. Ma DS, Correll J, Wittenbrink B. The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods. 2015;47(4):1122‐1135. [DOI] [PubMed] [Google Scholar]
28. Graf ABA, Wichmann FA, Bülthoff HH, Schölkopf B. Classification of faces in man and machine. Neural Comput. 2006;18(1):143‐165. [DOI] [PubMed] [Google Scholar]
29. Sutherland CAM, Young AW, Rhodes G. Facial first impressions from another angle: how social judgements are influenced by changeable and invariant facial properties. Brit J Psychol. 2017;108(2):397‐415. [DOI] [PubMed] [Google Scholar]
30. Li J, He D, Zhou L, et al. The effects of facial attractiveness and familiarity on facial expression recognition. Front Psychol. 2019;10:Article 2496. [DOI] [PMC free article] [PubMed] [Google Scholar]
31. Tian JH, Xie HL, Hu SY, Liu J. Multidimensional face representation in a deep convolutional neural network reveals the mechanism underlying AI racism. Front Comput Neurosc. 2021;15:Article 620281. [DOI] [PMC free article] [PubMed] [Google Scholar]
32. Shan T, Tay FR, Gu L. Application of artificial intelligence in dentistry. J Dent Res. 2021;100(3):232‐244. [DOI] [PubMed] [Google Scholar]
33. Pantic M, Rothkrantz LJM. Automatic analysis of facial expressions: the state of the art. IEEE T Pattern Anal. 2000;22(12):1424‐1445. [Google Scholar]
34. Obwegeser D, Timofte R, Mayer C, et al. Using artificial intelligence to determine the influence of dental aesthetics on facial attractiveness in comparison to other facial modifications. Eur J Orthodont. 2022;44(4):445‐451. [DOI] [PubMed] [Google Scholar]
35. Little AC, Jones BC, DeBruine LM. Facial attractiveness: evolutionary based research. Philos T R Soc B. 2011;366(1571):1638‐1659. [DOI] [PMC free article] [PubMed] [Google Scholar]
36. Al‐Omair OM, Huang SH. A comparative study on detection accuracy of cloud‐based emotion recognition services. Proceedings of the 2018 International Conference on Signal Processing and Machine Learning. Association for Computing Machinery;2018:142‐148. [Google Scholar]
37. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross‐sectional study. PLoS Med. 2018;15(11):e1002683. [DOI] [PMC free article] [PubMed] [Google Scholar]
38. Russell JA. Facial expressions of emotion: what lies beyond minimal universality? Psychol Bull. 1995;118(3):379‐391. [DOI] [PubMed] [Google Scholar]
39. Elfenbein HA, Ambady N. On the universality and cultural specificity of emotion recognition: a meta‐analysis. Psychol Bull. 2002;128(2):203‐235. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

[ocr12820-bib-0001] 1. Zhai Y, Cao H, Deng W, Gan J, Piuri V, Zeng J. BeautyNet: joint multiscale CNN and transfer learning method for unconstrained facial beauty prediction. Comput Intel Neurosc. 2019;2019:1910624. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0002] 2. Eisenthal Y, Dror G, Ruppin E. Facial attractiveness: beauty and the machine. Neural Comput. 2006;18(1):119‐142. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0003] 3. Laurentini A, Bottino A. Computer analysis of face beauty: a survey. Comput Vis Image Und. 2014;125:184‐199. [Google Scholar]

[ocr12820-bib-0004] 4. Patcas R, Bernini DAJ, Volokitin A, Agustsson E, Rothe R, Timofte R. Applying artificial intelligence to assess the impact of orthognathic treatment on facial attractiveness and estimated age. Int J Oral Max Surg. 2019;48(1):77‐83. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0005] 5. Patcas R, Timofte R, Volokitin A, et al. Facial attractiveness of cleft patients: a direct comparison between artificial‐intelligence‐based scoring and conventional rater groups. Eur J Orthodont. 2019;41(4):428‐433. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0006] 6. Mohammad‐Rahimi H, Nadimi M, Rohban MH, Shamsoddin E, Lee VY, Motamedian SR. Machine learning and orthodontics, current trends and the future opportunities: a scoping review. Am J Orthod Dentofac. 2021;160(2):170‐192. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0007] 7. Patcas R, Bornstein MM, Schatzle MA, Timofte R. Artificial intelligence in medico‐dental diagnostics of the face: a narrative review of opportunities and challenges. Clin Oral Investig. 2022;26(12):6871‐6879. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0008] 8. Chen K, Lu SM, Cheng R, et al. Facial recognition neural networks confirm success of facial feminization surgery. Plast Reconstr Surg. 2020;145(1):203‐209. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0009] 9. Dorfman R, Chang I, Saadat S, Roostaeian J. Making the subjective objective: machine learning and rhinoplasty. Aesthet Surg J. 2019;40(5):493‐498. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0010] 10. Khetpal S, Peck C, Parsaei Y, et al. Perceived age and attractiveness using facial recognition software in rhinoplasty patients: a proof‐of‐concept study. J Craniofac Surg. 2022;33(5):1540‐1544. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0011] 11. Rezende Machado AL, Dezem TU, Bruni AT, Alves da Silva RH. Age estimation by facial analysis based on applications available for smartphones. J Forensic Odontostomatol. 2017;35(2):55‐65. [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0012] 12. Talwar AA, Desai AA, McAuliffe PB, Broach RB, Percec I. 17. Artificially intelligent facial feature quantification after facial filler injection. Plast Reconstr Surg Global Open. 2022;10(6S):11. [Google Scholar]

[ocr12820-bib-0013] 13. Peck CJ, Patel VK, Parsaei Y, et al. Commercial artificial intelligence software as a tool for assessing facial attractiveness: a proof‐of‐concept study in an orthognathic surgery cohort. Aesthet Plast Surg. 2022;46(2):1013‐1016. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0014] 14. Boonipat T, Lin J, Bite U. Detection of baseline emotion in brow lift patients using artificial intelligence. Aesthet Plast Surg. 2021;45(6):2742‐2748. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0015] 15. Boonipat T, Hebel N, Zhu A, Lin J, Shapiro D. Using artificial intelligence to analyze emotion and facial action units following facial rejuvenation surgery. J Plast Reconstr Aes. 2022;75(9):3628‐3651. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0016] 16. Kuntzler T, Hofling TTA, Alpers GW. Automatic facial expression recognition in standardized and non‐standardized emotional expressions. Front Psychol. 2021;12:627561. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0017] 17. S‐G Jung, An J, Kwak H, Salminen J, Jansen B. Assessing the accuracy of four popular face recognition tools for inferring gender, age, and race. Proceedings of the International AAAI Conference on Web and Social Media. ICWSM. Association for the Advancement of Artificial Intelligence (AAAI); 2018:12(1). [Google Scholar]

[ocr12820-bib-0018] 18. Li S, Guo L, Liu J. Towards east Asian facial expression recognition in the real world: a new database and deep recognition baseline. Sensors. 2022;22(21):8089. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0019] 19. Gupta N, Hughes SJ, Chirwa R, Cheng Q. Facial recognition software use on surgically altered faces: a systematic review. J Craniofac Surg. 2022;33(8):2443‐2446. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0020] 20. Golle J, Mast FW, Lobmaier JS. Something to smile about: the interrelationship between attractiveness and emotional expression. Cogn Emotion. 2014;28(2):298‐310. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0021] 21. Morrison ER, Morris PH, Bard KA. The stability of facial attractiveness: is it what You've got or what you do with it? J Nonverbal Behav. 2013;37(2):59‐67. [Google Scholar]

[ocr12820-bib-0022] 22. Ekman P. Facial expression and emotion. Am Psychol. 1993;48(4):384‐392. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0023] 23. Rothe R, Timofte R, Van Gool L. Some like it hot—visual guidance for preference prediction. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE); 2016:5553‐61. [Google Scholar]

[ocr12820-bib-0024] 24. Mathias M, Benenson R, Pedersoli M, van Gool L. Face detection without bells and whistles. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T, eds. Computer Vision–ECCV 2014. ECCV 2014. Lecture Notes in Computer Science. Springer International Publishing; 2014:720‐735. doi: 10.1007/978-3-319-10593-2_47 [DOI] [Google Scholar]

[ocr12820-bib-0025] 25. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE);2017:2261‐69. [Google Scholar]

[ocr12820-bib-0026] 26. Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge. Int J Comput Vision. 2015;115(3):211‐252. [Google Scholar]

[ocr12820-bib-0027] 27. Ma DS, Correll J, Wittenbrink B. The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods. 2015;47(4):1122‐1135. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0028] 28. Graf ABA, Wichmann FA, Bülthoff HH, Schölkopf B. Classification of faces in man and machine. Neural Comput. 2006;18(1):143‐165. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0029] 29. Sutherland CAM, Young AW, Rhodes G. Facial first impressions from another angle: how social judgements are influenced by changeable and invariant facial properties. Brit J Psychol. 2017;108(2):397‐415. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0030] 30. Li J, He D, Zhou L, et al. The effects of facial attractiveness and familiarity on facial expression recognition. Front Psychol. 2019;10:Article 2496. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0031] 31. Tian JH, Xie HL, Hu SY, Liu J. Multidimensional face representation in a deep convolutional neural network reveals the mechanism underlying AI racism. Front Comput Neurosc. 2021;15:Article 620281. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0032] 32. Shan T, Tay FR, Gu L. Application of artificial intelligence in dentistry. J Dent Res. 2021;100(3):232‐244. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0033] 33. Pantic M, Rothkrantz LJM. Automatic analysis of facial expressions: the state of the art. IEEE T Pattern Anal. 2000;22(12):1424‐1445. [Google Scholar]

[ocr12820-bib-0034] 34. Obwegeser D, Timofte R, Mayer C, et al. Using artificial intelligence to determine the influence of dental aesthetics on facial attractiveness in comparison to other facial modifications. Eur J Orthodont. 2022;44(4):445‐451. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0035] 35. Little AC, Jones BC, DeBruine LM. Facial attractiveness: evolutionary based research. Philos T R Soc B. 2011;366(1571):1638‐1659. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0036] 36. Al‐Omair OM, Huang SH. A comparative study on detection accuracy of cloud‐based emotion recognition services. Proceedings of the 2018 International Conference on Signal Processing and Machine Learning. Association for Computing Machinery;2018:142‐148. [Google Scholar]

[ocr12820-bib-0037] 37. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross‐sectional study. PLoS Med. 2018;15(11):e1002683. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocr12820-bib-0038] 38. Russell JA. Facial expressions of emotion: what lies beyond minimal universality? Psychol Bull. 1995;118(3):379‐391. [DOI] [PubMed] [Google Scholar]

[ocr12820-bib-0039] 39. Elfenbein HA, Ambady N. On the universality and cultural specificity of emotion recognition: a meta‐analysis. Psychol Bull. 2002;128(2):203‐235. [DOI] [PubMed] [Google Scholar]

PERMALINK

Scoring facial attractiveness with deep convolutional neural networks: How training on standardized images reduces the bias of facial expressions

Dorothea Obwegeser

Radu Timofte

Christoph Mayer

Michael M Bornstein

Marc A Schätzle

Raphael Patcas