Human-Computer Interaction through Hybrid Inception V3 and LSTM based Facial Emotion Recognition: An Experimental Study

Shylaja Purakkadavath¹ and Thomas Chacko²
1. Department of Information Technology Kannur University, Kannur, Kerala, India
2. Department of Computer Science St. Pius X College Kasaragod, Kerala, India
Correspondence to: Shylaja Purakkadavath,  shylajap@kannuruniv.ac.in

DOI: https://doi.org/10.70389/PJS.100213

Additional information

• Ethical approval: N/a
• Consent: N/a
• Funding: No industry funding
• Conflicts of interest: N/a
• Author contribution: Shylaja Purakkadavath and Thomas Chacko – Conceptualization, Writing – original draft, review and editing
• Guarantor: Shylaja Purakkadavath
• Provenance and peer-review: Unsolicited and externally peer-reviewed
• Data availability statement: N/a

Keywords: Inception V3–LSTM hybrid, Facial emotion recognition, Temporal-spatial feature fusion, Human-computer interaction, Image sequence classification

Peer Review
Received: 14 August 2025
Last revised: 23 September 2025
Accepted: 17 December 2025
Version accepted: 2
Published: 12 January 2026

Introduction

The identification of human emotions from facial expressions remains a challenging area in Human-Computer Interaction (HCI), as machines cannot fully interpret the human emotions.¹ Emotions play a vital role in HCI and should be considered to have interaction between humans and machines. Emotion detection enables computer systems to understand better, by improving their communication and user experience. Facial emotion recognition (FER) systems try to analyze key facial features such as the eyes, mouth, and eyebrows to understand emotional states.² Facial images are captured with cameras and processed computationally to discover meaningful patterns. Effective FER needs not only the recognition of visible expressions but also an understanding of the underlying psychological and physiological origins of those emotions. Emotions may be conveyed verbally through speech or non-verbally via tone of voice, facial expressions, and physiological changes. But such things are not always reliable, as they may be intentionally controlled or may not reflect the true internal state.³

Even though considerable progress is made, still FER remains prone to variations in illumination, head pose, occlusion, and dynamic expressions. Traditional methods rely either on geometric methods, which focus on facial landmarks such as the nose, eyes, and mouth (e.g., the Facial Action Coding System (FACS) and Active Appearance Models (AAM)), or on appearance-based methods that extract texture featuresusing techniques such as Local Binary Patterns (LBP) and Gabor filters. While these methods have achieved moderate success, they struggle with robustness to pose variation, illumination changes, and subtle or micro-expressions. Also, they often require manual feature engineering, limiting adaptability, and are computationally expensive for real-time use.⁴ While traditional FER techniques provide useful insights, they lack robustness, generalization across demographics, and the ability to capture temporal dynamics of facial expressions. This gap motivates the integration of deep learning (DL)-based methods that can automatically learn both spatial and temporal features for improved real-world performance.

DL models have demonstrated superior performance by handling variations in pose, lighting, and occlusion more effectively, while also showing better generalization across demographic groups. Importantly, they enable recognition of subtle and micro-expressions with higher accuracy, supporting real-time applications. A major limitation across both speech and physiological systems is their susceptibility to noise and complexity, which drastically affects generalizability. Speech Emotion Recognition (SER) models, such as those using the Multilayer Perceptron (MLP) or the Hybrid LSTM Network and Transformer Encoder, are compromised by variation in human speech patterns and pervasive background noise. Similarly, physiological methods utilizing EEG or ECG signals face challenges from the inherent variability of signals and noise within the data itself, which reduces the reliability of classifiers like SVM and hinders the performance of ML models. While Deep Learning (DL) aims to handle variations like pose and lighting, even thermal video-based HCI models have struggled to recognize emotions in real-time.

More over, many existing Facial Emotion Recognition (FER) approaches is their reliance on static feature extraction, failing to capture the dynamic, sequential dependencies of emotional change, which is crucial for real-time applications. Models using VGGNet or MobileNet architectures demonstrate difficulties in identifying subtle facial emotions or micro-expressions due to insufficient training data. This generalized inability to model how expressions unfold over time, as noted in the limitations of traditional geometric and appearance-based methods, highlights the necessity of temporal sequence learning—a gap directly validated by research showing the benefit of integrating Long Short-Term Memory (LSTM) into FER pipelines. Also, many models, including those based on VGGNet and those using physiological datasets like DEAP, suffer from the use of imbalanced or limited-size datasets, restricting their ability to generalize reliably across diverse demographics. Furthermore, advanced techniques—whether utilizing wavelet scattering for ECG analysis or neurosymbolic AI like SenticNet 8 for textual analysis—introduce high computational complexity and high computing needs, which ultimately hinder their scalability and feasibility for deployment in real-time HCI systems.

Current Facial Emotion Recognition (FER) techniques lack the necessary robustness, generalization across demographics, and the ability to effectively capture the temporal dynamics of facial expressions, which motivates the integration of deep learning methods that automatically learn both spatial and sequential features. This study addresses this limitation by developing and evaluating a hybrid deep learning model that integrates Inception V3 for spatial feature extraction with Long Short-Term Memory (LSTM) networks for temporal sequence learning, achieving a high accuracy of 94.5% on the FER-2013 dataset. The objective is to develop and evaluate a hybrid Facial Emotion Recognition (FER) model that integrates Inception V3 for spatial feature extraction with Long Short-Term Memory (LSTM) networks for temporal sequence learning. The ultimate goal is to achieve robust classification performance across varying conditions and to demonstrate the model’s potential for real-time applications in Human-Computer Interaction (HCI). The remainder of this article is structured as follows: Section 2 reviews existing FER studies. Section 3 describes the proposed methodology. Section 4 presents results and discussion, and Section 5 concludes with key findings and future work.

Literature Review

Alnuaim et al.⁵ utilized AI techniques to identify emotion from human vocals. The RAVDESS dataset includes 2000 recorded data of 24 actors as songs and talks. Data were collected from the artists for eight distinct moods. For classification supervised learning technique, the multilayer perceptron (MLP) classifier and got 81% accuracy. The model faced limitations in identifying the emotion from human vocals due to variation in human speech patterns and background noise, which affected the generalizability of the classifier. Chowdary et al.⁶ recognized the emotion from facial expression by using the transfer learning technique. The authors utilized pre-trained models and used CK+ database to test the experimental studies, and the MobileNet attained the best accuracy of 94.2% for emotion detection. The model faced difficulties identifying the subtle emotions due to insufficient data. Challenges faced in recognizing the real-time emotions using DL techniques.

Sharma et al.⁷ developed an emotion classifier using the machine learning (ML) models like Naïve Bayes (NB) and Logistic Regression (LR) for textual emotion detection. While comparing their accuracy, LR performed better than NB. The authors conducted another comparison between the RNN models and the LSTM model performed better than the GRU model. However, the model faced limitations in predicting the emotions due to the complexity in human expression and the unbalanced dataset. The computational cost of training faced difficulties in real-time analysis. Cambria et al.⁸ introduced SenticNet 8, a neurosymbolic AI system for fusing emotion that attempts to address the symbol grounding issue by combining structured attention networks with sensible information presentation. The authors evaluated the SenticNet 8 on three downstream tasks against ChatGPT and three additional baselines. According to the results, SenticNet 8 performs better than all baselines on all tasks in general. The system faced limitations in capturing human emotions due to diverse data sources. High computing needs hindered scalability.

Khaireddin et al.⁹ presented a model for FER using VGGNet architecture. The model achieved the highest single network classification accuracy on the FER2013 dataset and attained 73.28% accuracy without using extra training data. The model faced difficulties with imbalance dataset and real-time face emotion recognition. The model also faced challenges in identifying the subtle facial emotions. Algarni et al.¹⁰ suggested a DL-based method for using EEG signals to identify emotions. The Binary Grey Wolf Optimiser was used for the feature selection operation. The Bi-LSTM model had been used in the classification stage to identify human emotions. The model attained better accuracy when compared to traditional models. Xu et al.¹¹ suggested a method to increase the precision of Speech Emotion Recognition (SER) based on the multi-head attention mechanism. The authors conducted experiment with the interactive emotional dyadic motion capture (IEMOCAP) dataset. The model attained an accuracy of 76.18% in detecting the speech emotion. The model faced limitations in complex speech variations. Islam et al.¹² employed a ML model with convolutional neural network (CNN) to identify the emotion from EEG signals. The model used DEAP dataset and the emotions were recognised by using the LR algorithm. The model attained 78.22% accuracy on valence and 74.92% on arousal. Complexity in ECG signals hindered the performance of the model.

Lian et al.¹³ introduced a conversational transformer network (CTNet), a multimodal multi-party network for conversational emotion identification. CTNet models provide an interaction between modalities for multimodal features. The IEMOCAP and MELD dataset indicated the efficiency of the CTNet in emotion recognition. The system struggled to recognize the emotions due to variations in speech patterns. The lack of diverse training and the long conservations impacted the performance of the system. Khateeb et al.¹⁴ proposed an emotion recognition model for nine emotions based on signal data. The authors used the DEAP dataset from 32 persons and explored the multi-domain features. The model utilized support vector machine to act as classifier and recognized the 9 types of emotions with an average accuracy of 65.92%. The limited size of the DEAP dataset affected the effectiveness of the system and the variability of ECG signals affected the model accuracy. The noise in ECG signals reduced the reliability of the model.

Liu et al.¹⁵ introduced a multi-channel characteristic based on the EEG signals for human emotion identification. The author used multi- channel textual feature fusion and multi-channel EEG to identify the human emotions. For emotion classification, the model fused the six statistical features in time domain. The SVM model was trained for human emotion identification. The DEAP dataset showed better accuracy rate compared to the frequency domain feature-based emotion recognition algorithms. However, the model struggled in integrating the multi-channel textual and EEG features which increase the complexity of the model. The efficacy of the model was hindered by the limited availability of high-quality labeled EEG data.

Andayani et al.¹⁶ developed a LSTM Network and Transformer Encoder to recognize the long-term dependencies in voice signals and categorize the emotions. The outcomes indicated that the model achieved a significant recognition improvement by using hybrid model and reached an accuracy of 85.55% in Emo-DB dataset. The background noise in speech audio files hindered the efficiency of the model. The model faced the limitation in recognizing the similar emotions which led to misclassification. Sepúlveda et al.¹⁷ enhanced the effectiveness of wavelet transform for signal processing in the recognition of emotions from ECG signals. ECG signal features was extracted using a wavelet scattering approach from the AMIGOS database. The wavelet scattering algorithm showed a better performance for classify emotions in valence dimension and arousal. However, the varied population affected the performance of the AMIGOS database and the wavelet scattering algorithm increased the computational complexity.

Alsabhan et al.¹⁸ introduced a CNN architecture using attention layers and LSTM and 2D-CNN was used to recognize the emotions in 3 different languages. The MFCC, ZCR, and RMSE features were recovered prior to begin the model training procedure, and the learning models were implemented and their results exhibited on an audio dataset in three languages. The performance of 2D-CNN showed superior for the “low” emotion class. However, the model showed limitations in complex evaluation. Nayak et al.¹⁹ introduced a three-stage real-time HCI system for recognising human emotions using multivariate time series (MTS) thermal video sequences. In the first stage, nose, eye and face detected by using a Faster R-CNN. The multiple instance learning system was used for tracking facial ROIs over the thermal videos. In the second stage dynamic time warping (DTW) method was used to classify the emotional states. In the third stage HCI provided relevant suggestions from a various perspective. The model showed better accuracy than the other classification methods. However, the HCI model struggled to recognize the emotions in real-time. Zhang et al.²⁰ pointed out that deep CNNs improve FER accuracy under occlusion. Similarly, the use of temporal modeling in validates the integration of LSTM in FER pipelines. 

While existing emotion recognition systems have made significant strides in utilizing various data sources, like facial expressions, speech, and physiological signals, they still face limitations in handling dynamic and complex real-time scenarios. For instance, models relying on speech face difficulties in generalizing due to variations in human speech patterns and background noise, leading to reduced accuracy in real-world settings.⁵ Similarly, facial expression recognition models struggle with identifying subtle emotions due to insufficient data, hindering performance in real-time emotion detection⁶ These gaps highlight the need for more robust systems that can effectively integrate multimodal data and adapt to diverse conditions while maintaining high accuracy in real-time applications. Facial expression recognition (FER) is used here due to the importance of practicality, scalability, and strong alignment with non-intrusive human-computer interaction. Compared to speech-based models, which can be impaired by noise, language barriers, and cultural nuances, or EEG-based models, which are intrusive and require specialized hardware, FER offers a universally interpretable and cost-efficient alternative. Table 1 shows the existing work details.

Table 1: Summary table of existing works.
Reference	Model/ Technique Used	Dataset(s) Used	Reported Accuracy / Performance	Key Limitations Identified
Alnuaim et al.5	Multilayer Perceptron (MLP) Classifier	RAVDESS (2000 recorded data of 24 actors)	81% accuracy	Variation in human speech patterns and background noise, affecting the generalizability of the classifier.
Chowdary et al.6	Transfer learning (MobileNet)	CK+ database	94.2% accuracy	Difficulties identifying subtle emotions due to insufficient data; challenges in recognizing real-time emotions using Deep Learning (DL) techniques.
Sharma et al.7	Machine Learning (Naïve Bayes, Logistic Regression), RNNs (LSTM, GRU)	Textual data (not specified)	LR > NB; LSTM > GRU	Complexity in human expression; unbalanced dataset; high computational cost hindering real-time analysis.
Cambria et al.8	SenticNet 8 (neurosymbolic AI system)	Evaluated on three downstream tasks (against ChatGPT and baselines)	Performed better than baselines	Diverse data sources and high computing needs hindered scalability.
Khaireddin et al.9	VGGNet architecture	FER2013 dataset	73.28% accuracy	Imbalanced dataset; challenges with real-time face emotion recognition; difficulties identifying subtle facial emotions.
Algarni et al.10	DL-based (Bi-LSTM, Binary Grey Wolf Optimiser)	EEG signals (not specified dataset name)	Better accuracy than traditional models	(No specific limitations for this model directly stated in the literature review).
Xu et al.11	Multi-head attention mechanism (for SER)	IEMOCAP dataset	76.18% accuracy	Complex speech variations.
Islam et al. 12	ML with CNN, Logistic Regression	DEAP dataset (for EEG signals)	78.22% (valence), 74.92% (arousal)	Complexity in ECG signals hindered the performance of the model .
Lian et al. 13	Conversational Transformer Network (CTNet)	IEMOCAP and MELD datasets	Indicated efficiency	Variations in speech patterns; lack of diverse training; long conversations impacted performance.
Khateeb et al.14	SVM classifier (multi-domain features)	DEAP dataset (from 32 persons, ECG data)	65.92% average accuracy	Limited size of DEAP dataset; variability of signals reduced reliability.
Liu et al.15	SVM model (multi-channel textual and EEG features)	DEAP dataset	Better accuracy than frequency-domain methods	Struggled in integrating multi-channel textual and EEG features (increased complexity); limited availability of high-quality labeled EEG data.
Andayani et al.16	Hybrid LSTM Network and Transformer Encoder	Emo-DB dataset	85.55% accuracy	Background noise in speech audio files hindered efficiency; limitation in recognizing similar emotions.
Sepúlveda et al.17	Wavelet scattering approach (for ECG signals)	AMIGOS database	Showed better performance	Varied population affected the performance of the AMIGOS database; wavelet scattering algorithm increased computational complexity.
Alsabhan et al.18	CNN architecture (attention layers, LSTM, 2D-CNN)	Audio dataset (three languages)	Superior for “low” emotion class	Model showed limitations in complex evaluation.
Nayak et al.19	Faster R-CNN, Multiple Instance Learning, Dynamic Time Warping (DTW)	Multivariate Time Series (MTS) thermal video sequences	Better accuracy than other methods	Human-Computer Interaction  model struggled to recognize emotions in real-time.

Materials and Methods

FER system focuses on recognizing facial expressions that convey basic emotions. It is significant in HCI and in areas such as online gaming, digital advertising, healthcare, and customer feedback assessments. But challenges exist in accurately recognizing emotions under realistic conditions, including subtle differences between similar expressions and variations in facial pose, illumination, and occlusion. The proposed model aims to enhance the classification accuracy of FER through a novel deep learning (DL) approach integrating Inception V3 and LSTM networks. Pre-processed images are fed into the model, which classifies emotions such as anger, neutral, fear, happiness, sadness, and surprise. The block diagram of the proposed model is illustrated in Figure 1.

Dataset

The dataset plays an important role in training models for best facial emotion recognition. We have used the FER-2013 dataset for training and evaluating the proposed model. This dataset is widely utilized in facial emotion recognition studies and contains six emotions for this study: sad, surprised, neutral, happy, anger, and fear. All images are in grayscale format with 48 × 48 pixels, and cropped to include facial regions only with respective labels. Figure 2 shows sample images from the dataset. The dataset contains 61,412 images for training and 15,353 for testing after necessary filtering and balancing. Figure 3 presents a bar plot illustrating the distribution of the six emotions, providing a clear view of dataset balance.

Image Pre-processing

Data pre-processing is essential to ensure that the data is cleaned, transformed, and integrated to improve image quality and prepare it for analysis. All images were resized to 299 × 299 to match Inception V3 input. Pixel values were normalized to a range of [0,1]. The dataset features are standardised to have a standard deviation of one and a mean of zero, ensuring that all attributes are on the same scale. To address the class imbalance in the target variable, random oversampling is applied on the training datasets. After image preprocessing the dataset is split into testing and training data in a ratio 80:20 before being fed into the proposed DL network.

Model Development

The study utilizes a hybrid model of Inception V3 and LSTM model for face emotion recognition using a DL network. This model captures both spatial and temporal features from facial expressions, improving the accuracy of emotion detection.

Inception V3

Inception V3 is CNN architecture known for its efficient design and high performance in image classification tasks.²⁰ The Inception V3 network uses several inception modules, each consisting of multiple convolutional layers with different kernel sizes running in parallel as in Figure 4.

This allows the network to capture features at multiple scales. An inception module can be mathematically represented by Equation 1.

where concat denotes the concatenation of feature maps, Conv_kxk represents a convolution operation with a kxk kernel, and MaxPool_3´3 is a max-pooling operation with a 3 × 3 window. The input X is processed in parallel through these operations, and their outputs are concatenated. To reduce computational complexity, inceptionV3 factorizes larger convolutions into smaller ones. This factorization decreases the number of parameters and computational cost while preserving the capability to capture complex patterns. Batch normalization is used to stabilize and accelerate the training process. It normalizes the input of each layer so that it has zero mean and a variance of one. For an input X with mini batch mean μ_B and variance,

where Є is a small constant to avoid division by zero and the learnable parameters are given by Equation 3.

where γ and β are learnable parameters which allow the model to scale and shift the normalized output. InceptionV3 includes auxiliary classifiers to provide additional gradient signal and regularization during training. The expression of auxiliary classifier loss is given by Equation 4.

where γ and β are learnable parameters which allow the model to scale and shift the normalized output. InceptionV3 includes auxiliary classifiers to provide additional gradient signal and regularization during training. The expression of auxiliary classifier loss is given by Equation 4.

where W_f  is the weight matrix for the forget gate, h_t–1 denote the past hidden state and b_f is the bias term. The output f_t indicates how much of the previous information should be retained or discarded. The input gate decides which new data will be added to the cell state. Equation 6 represents the input gate and Equation 7 shows the new candidate values to be added to the cell state.

where i_t controls which values to update, and represents the candidate values. These values are then used to modify the cell state. The cell state is the memory of the LSTM, and it gets updated by combining the previous cell state with the new candidate values, modulated by the forget and input gates. This update is represented by Equation 8.

where C_t is the updated cell state. The forget gate f_t controls how much of the previous cell state C_t–1 is retained, while the input gate i_t determines how much of the new candidate values  is to be added to the cell state. Finally, the output gate chooses what part of the updated cell state will be passed as the hidden state to the following time step. The output gate uses the current hidden state and the updated cell state to generate the final output, as shown in Equation 8 and Equation 9.

where tanh is hyperbolic tangent activation function which squashes numbers into the range (–1,1).

Proposed Deep Learning Model

The proposed hybrid model integrates InceptionV3 for spatial feature extraction and LSTM for temporal sequence learning to effectively recognize emotions from facial images. First, the input facial images are passed through the InceptionV3 network, which captures spatial features by learning key facial landmarks and expression-related patterns. After passing through the initial convolutional and pooling layers, batch normalization is applied to enhance training and the model’s generalization. Next, the feature maps extracted by InceptionV3 are fed into an LSTM layer, which captures the temporal dependencies in the sequence of facial expressions, a crucial step for handling video data or image sequences.

The output from the LSTM layer (figure 5) is then processed through a fully connected layer and a softmax activation function to compute the probability distribution over the six facial emotions. The softmax function ultimately predicts the class corresponding to one of these seven emotions, based on the spatial and temporal features extracted from the facial images. By combining the strengths of both InceptionV3 for feature extraction and LSTM for sequence learning, this hybrid model efficiently handles the complexity of emotion recognition from facial images. Unlike simple CNNs or shallow RNNs, this CNN-RNN hybrid allows the model to retain sequential dependencies while maintaining robust spatial understanding.

Hardware and Software Setup

The suggested model was developed and trained using Keras on Google Colab with Python utilizing the computational power of TensorFlow. The experimental system consisted 16 GB of RAM, GPU acceleration, and 68.50 GB of storage, running in a 64-bit Windows 10 environment. The system used an Intel Core i7-6850K processor and an NVIDIA GeForce GTX 1080Ti GPU to improve computational efficiency. The predictions from the test dataset were used to determine its highest accuracy. Hyper parameter tuning plays an important role in model development, a wide range of hyper parameter configurations were explored, and Table 2 summarizes the optimal configuration that yielded the best classification performance.

Table 2: Hyper parameters for the suggested model.
Hyper parameters	Values
Loss Function	Categorical Cross entropy
Number of Epochs	40
Dropout	0.2
Learning rate	0.001
Batch Size	16
Optimizer	Adam
Activation Function	Softmax

Results and Discussion

The model’s performance across epochs is illustrated by the accuracy and loss charts in Figure 6. Upward patterns in training and validation accuracy indicate good learning and generalization. Initially, validation accuracy remains stable about 0.5 till epoch 15 whereas training accuracy gradually increases from 0.67 and reaches near 0.945 by epoch 40.

For evaluating the model’s performance for recognising the emotions, specific evaluation metrics are used, as given in Equations (11) to (14).

Where, TF = True Negative, TP = True Positive,
FP = False Positive and FN = False Negative. Table 2 offers the classification report of the proposed FER system. The performance metrics of the model demonstrate excellent results for facial emotion recognition. An accuracy of 94.5% specifies that the model correctly classified nearly all the emotions. With a precision of 95.1%, the model is highly reliable in predicting positive emotion classes without many false positives. The recall of 94.3% shows that the model successfully identifies most of the true positive instances, minimizing false negatives. The F1-score of 94.6% strikes a balance between precision and recall, confirming that the model has strong overall performance in detecting facial emotions.

The performance metrics demonstrate excellent results for facial emotion recognition on the FER-2013 dataset. Table 3 summarizes the performance evaluation metrics of the proposed model. An accuracy of 94.5% indicates the model correctly classified almost all emotions. A precision of 95.1% suggests high reliability in predicting positive emotion classes with minimal false positives. A recall of 94.3% shows that most true positive instances were identified, minimizing false negatives. The F1-score of 94.6% strikes a balance between precision and recall, confirming the model’s strong overall performance.

Table 3: Performance evaluation of the proposed model.
Performance metrics	Result (%)
Accuracy	94.5
Precision	95.1
Recall	94.3
F1-Score	94.6

The most accurately classified emotion was Neutral (3,977 correct predictions), followed by Sad (3,892) and Happy (3,692). Angry and Surprise were also well-identified, with 1,301 and 1,213 correct predictions, respectively. Fear, although less frequent in the FER-2013 dataset, was accurately classified 1,192 times. Misclassifications were minimal, with the highest confusion occurring between Neutral and neighboring emotions (e.g., 13 instances misclassified as Sad or Surprise). Such misclassifications often arise due to subtle similarities in facial features, particularly in near-neutral expressions where slight eyebrow or lip variations suggest sadness or surprise. Table 4 illustrates the works done on FER-2913 dataset and the proposed work.

Table 4: Comparison of models on FER-2013 dataset.
Author	Model	Accuracy
Khaireddin et al.	VGGNet, CNN	73.28%
Xu et al.	1. ResNet-50, CNN 2. LSTM, RNN	57.48% 71%
Jayaraman et al.	CNN+LSTM	82%
Debnath et al.	Four layer ConvNet, CNN	91.01%
Al-Shabi et al.	CNN+ SIFT	73.4%
Proposed Model	CNN+BiLSTM	94.5%

Conclusion

The study successfully determined the effectiveness of a hybrid Deep Learning (DL) model that integrates Inception V3 for spatial feature extraction with Long Short-Term Memory (LSTM) for temporal sequence learning, effectively addressing the need for robust Facial Emotion Recognition (FER). This architecture was designed to capture both spatial patterns from faces and the temporal relationships between expressions, thereby overcoming the limitations of static feature extraction methods. Tested on the FER-2013 dataset, the proposed system demonstrated superior performance, achieving 94.5% accuracy, 95.1% precision, 94.3% recall, and a 94.6% F1-score, confirming that combining convolutional networks with sequence models significantly boosts recognition performance in emotion recognition tasks.

This high reliability suggests the model is well-suited for deployment in real-world applications in Human-Computer Interaction (HCI), healthcare, and security, allowing it to generalize well across diverse datasets. Future work, however, must address the potential reflection of biases inherent in the training data by focusing on building more demographically diverse datasets and exploring adaptive data augmentation techniques to further enhance the model’s robustness and generalizability.

References

1. Khaireddin Y, Chen Z. Facial emotion recognition: State of the art performance on FER2013. arXiv [Preprint]. 2021[10 p.]
2. Jamshidnejad A, Jamshidined A. Facial emotion recognition for human computer interaction using a fuzzy model in the e-business. In: 2009 Innovative Technologies in Intelligent Systems and Industrial Applications; 2009 Jul 25–27; Kuala Lumpur, Malaysia. IEEE; 2009. p. 202–4.
3. Fernández I, Carrera P, Sánchez F, Paez D, Candia L. Differences between cultures in emotional verbal and non-verbal reactions. Psicothema. 2000;12(Su1):83–92.
4. Gunawan AA. Face expression detection on Kinect using active appearance model and fuzzy logic. Procedia Comput Sci. 2015;59:268–74.
5. Alnuaim AA, Zakariah M, Shukla PK, Alhadlaq A, Hatamleh WA, Tarazi H, et al. Human-computer interaction for recognizing speech emotions using multilayer perceptron classifier. J Healthc Eng. 2022;2022:6005446.
6. Chowdary MK, Nguyen TN, Hemanth DJ. Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Comput Appl. 2023;35(32):23311–28.
7. Sharma T, Diwakar M, Singh P, Lamba S, Kumar P, Joshi K. Emotion Analysis for predicting the emotion labels using Machine Learning approaches. In: 2021 IEEE 8th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON); 2021 Nov 11–13; Greater Noida, India. IEEE; 2021. p. 1–6.
8. Cambria E, Zhang X, Mao R, Chen M, Kwok K. SenticNet 8: Fusing emotion AI and commonsense AI for interpretable, trustworthy, and explainable affective computing. In: International Conference on Human-Computer Interaction; 2024 Jun 29–Jul 4; Washington, DC. Springer Nature Switzerland; 2024. p. 197–216.
9. Greff K, Srivastava RK, Koutník J, Steunebrink BR, Schmidhuber J. LSTM: A search space odyssey. IEEE Trans Neural Netw Learn Syst. 2016;28(10):2222–32.
10. Algarni M, Saeed F, Al-Hadhrami T, Ghabban F, Al-Sarem M. Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM). Sensors. 2022;22(8):2976.
11. Xu M, Zhang F, Zhang W. Head fusion: Improving the accuracy and robustness of speech emotion recognition on the IEMOCAP and RAVDESS dataset. IEEE Access. 2021;9:74539–49.
12. Islam MR, Islam MM, Rahman MM, Mondal C, Singha SK, Ahmad M, et al. EEG channel correlation based model for emotion recognition. Comput Biol Med. 2021;136:104757.
13. Lian Z, Liu B, Tao J. CTNet: Conversational transformer network for emotion recognition. IEEE/ACM Trans Audio Speech Lang Process. 2021;29:985–1000.
14. Khateeb M, Anwar SM, Alnowami M. Multi-domain feature fusion for emotion classification using DEAP dataset. IEEE Access. 2021;9:12134–42.
15. Liu Y, Fu G. Emotion recognition by deeply learned multi-channel textual and EEG features. Future Gener Comput Syst. 2021;119:1–6.
16. Andayani F, Theng LB, Tsun MT, Chua C. Hybrid LSTM-transformer model for emotion recognition from speech audio files. IEEE Access. 2022;10:36018–27.
17. Sepúlveda A, Castillo F, Palma C, Rodriguez-Fernandez M. Emotion recognition from ECG signals using wavelet scattering and machine learning. Appl Sci. 2021;11(11):4945.
18. Alsabhan W. Human–computer interaction with a real-time speech emotion recognition with ensembling techniques 1D convolution neural network and attention. Sensors. 2023;23(3):1386.
19. Nayak S, Nagesh B, Routray A, Sarma M. A Human–Computer Interaction framework for emotion recognition through time-series thermal video sequences. Comput Electr Eng. 2021;93:107280.
20. Tio AE. Face shape classification using inception v3. arXiv [Preprint]. 2019 [6 p.]

Cite this article as:
Purakkadavath S and Chacko T. Human-Computer Interaction Through Hybrid Inception V3 and LSTM based Facial Emotion Recognition: An Experimental Study. Premier Journal of Science 2025;15:100213

Export to EndNote Export to Mendeley