Penerapan Model Hybrid CNN-LSTM dengan Integrasi Text-to-Speech untuk Pengenalan Gerakan Isyarat SIBI ke dalam Teks Suara

Hidayat, Syahrul (2024) Penerapan Model Hybrid CNN-LSTM dengan Integrasi Text-to-Speech untuk Pengenalan Gerakan Isyarat SIBI ke dalam Teks Suara. Undergraduate thesis, UPN Veteran Jawa Timur.

	Text (Cover) 20081010076-COVER.pdf Download (1MB)
	Text (Bab 1) 20081010076-BAB 1.pdf Download (213kB)
	Text (Bab 2) 20081010076-BAB 2.pdf Restricted to Repository staff only until 16 July 2026. Download (891kB)
	Text (Bab 3) 20081010076-BAB 3.pdf Restricted to Repository staff only until 16 July 2026. Download (501kB)
	Text (Bab 4) 20081010076-BAB 4.pdf Restricted to Repository staff only until 16 July 2026. Download (3MB)
	Text (Bab 5) 20081010076-BAB 5.pdf Download (194kB)
	Text (Daftar Pustaka) 20081010076-DAFTARPUSTAKA.pdf Download (212kB)

Abstract

The Indonesian Sign Language System (SIBI) is an essential tool for communication within the deaf and mute community in Indonesia. However, the limited public understanding of SIBI often creates barriers in communication. This study develops a model to recognize SIBI gestures into voice text to facilitate effective communication for people with hearing and speech disabilities in Indonesia. The proposed method integrates a hybrid CNN-LSTM model and Text-to-Speech (gTTS) technology to recognize SIBI gestures. The CNN-LSTM model processes spatial and temporal information from the data, while gTTS provides feedback on the recognized SIBI gestures in audio form. This study compares the performance of the model on two types of SIBI datasets: an image sequence dataset and a numpy sequence dataset resulting from key point feature extraction. Training is conducted with various parameters such as batch size, learning rate, and epochs. The model is evaluated using metrics such as accuracy, precision, recall, and F1-score. The test results show that the model with the image sequence dataset achieved a maximum accuracy of 1.00 at 50 epochs, while the model with the numpy sequence dataset achieved a highest accuracy of 0.98 at 50 epochs. In real-time detection tests, the numpy model could accurately detect SIBI gestures without being affected by environmental and object variations. The real-time detection program produces predictions of 25 SIBI gestures in text and voice forms.

Item Type:

Thesis (Undergraduate)

Contributors:

Contribution	Contributors	NIDN/NIDK	Email
Thesis advisor	Via, Yisti Vita	NIDN0025048602	yistivia.if@upnjatim.ac.id
Thesis advisor	Mandyartha, Eka Prakarsa	NIDN0725058805	eka_prakarsa.fik@upnjatim.ac.id

Subjects:

P Language and Literature > P Philology. Linguistics > P99 Semiotics. Signs and Symbols
P Language and Literature > P Philology. Linguistics > P99.5 Non Verbal communication
Q Science > Q Science (General)
Q Science > QM Human anatomy
T Technology > T Technology (General)
P Language and Literature > P Philology. Linguistics > P93.5 Visual Communication

Divisions:

Faculty of Computer Science > Departemen of Informatics

Depositing User:

Syahrul Hidayat

Date Deposited:

17 Jul 2024 02:37

Last Modified:

17 Jul 2024 02:37

URI:

https://repository.upnjatim.ac.id/id/eprint/26280