# A custom ASR model to transcribe the speech of Kabjye Dilgo Khyentse Rinpoche **URL:** \ **Category:** 🔊 ASR Speech Recognition SIG\ **Created:** [December 20, 2024, 4:26pm UTC](https://forum.webuddhist.com/t/a-custom-asr-model-to-transcribe-the-speech-of-kabjye-dilgo-khyentse-rinpoche/170 "2024-12-20T16:26:48Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ganga\_Gyatso](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.webuddhist.com/ganga_gyatso/32/166_2.png) [@Ganga\_Gyatso](https://forum.webuddhist.com/u/Ganga_Gyatso)\ **Post date:** [December 20, 2024, 4:26pm UTC](https://forum.webuddhist.com/t/a-custom-asr-model-to-transcribe-the-speech-of-kabjye-dilgo-khyentse-rinpoche/170/1 "2024-12-20T16:26:48Z")
**Title:** A Custom ASR Model to Transcribe the Speech of Kabjye Dilgo Khyentse Rinpoche ### Introduction We have a base Speech-to-Text (STT) model trained on a diverse range of audio data, including general conversations, podcasts, and other varied speech recordings. While the base model provides good transcription results across these general domains, it faces challenges when transcribing audio from specific speakers, especially those with unique accents or speech patterns, like Kabjye Dilgo Khyentse Rinpoche. In this experiment, we aim to fine-tune the base model on a small set of annotated speech data from Dilgo Khyentse Rinpoche and evaluate its performance on his recordings. Our hypothesis is that this fine-tuning will improve transcription accuracy for his voice compared to the baseline model. ### Background General-purpose ASR models are typically trained on diverse data to handle a wide range of speakers and contexts. However, these models tend to underperform on single-speaker datasets, especially those with distinct accents, speech cadences, or pronunciations. Fine-tuning the model on a speaker’s specific data can help the model adapt to those particularities, improving performance significantly. This experiment explores the impact of fine-tuning the base model on the unique speech patterns of Kabjye Dilgo Khyentse Rinpoche, a Tibetan spiritual leader. ### Methodology 1. **Dataset Preparation:** 2. **Data Preprocessing:** 3. **Dataset Splitting:** 4. **Model Fine-Tuning:** 5. **Evaluation:** ### Training Details - **Training Dataset:** - **Training Parameters:** - **Hardware and Duration:** ### Results - **Base Model Performance:** - **Fine-Tuned Model Performance:** ### Discussion Fine-tuning the base model on Dilgo Khyentse Rinpoche’s speech data resulted in a substantial improvement in the Character Error Rate (CER). The fine-tuned model performed nearly 27% better than the base model, showcasing how speaker-specific fine-tuning can help tailor ASR models to unique speech patterns. The results underscore the importance of using a speaker’s own data to improve transcription accuracy, especially when working with spiritual or historical figures whose voices and speech patterns differ significantly from those in a general training corpus. While the results are promising, further performance improvements can be achieved by incorporating additional data from Kabjye Dilgo Khyentse Rinpoche or similar speakers. The model could also be fine-tuned with more advanced techniques to achieve even better results in real-world applications, such as transcription of historical teachings and Dharma talks. ### Conclusion This experiment demonstrates the effectiveness of fine-tuning a base STT model on a speaker-specific dataset, leading to significant improvements in transcription accuracy for Kabjye Dilgo Khyentse Rinpoche’s speech. Our hypothesis that fine-tuning the model on this specific dataset would improve transcription accuracy was validated, as evidenced by the reduction in the Character Error Rate (CER). The base model achieved a CER of **0.4754** , while the fine-tuned model reduced the CER to **0.2043** , resulting in a **27% improvement**. This substantial reduction confirms that speaker-specific fine-tuning is highly beneficial for improving transcription accuracy, particularly for unique speech patterns such as those of Kabjye Dilgo Khyentse Rinpoche. ### Resources - [Fine-Tuned Model (Checkpoint 58000)](https://huggingface.co/ganga4364/mms_300_khentse_Rinpoche-Checkpoint-58000) - [Training Dataset](https://huggingface.co/datasets/ganga4364/Dilgo-Khyentse-Rinpoche-dataset) - [Base Model](https://huggingface.co/ganga4364/mms_300_v4.96000) ---
**Author:** ![Trinley](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.webuddhist.com/trinley/32/3_2.png) [@Trinley](https://forum.webuddhist.com/u/Trinley)\ **Post date:** [December 23, 2024, 8:00am UTC](https://forum.webuddhist.com/t/a-custom-asr-model-to-transcribe-the-speech-of-kabjye-dilgo-khyentse-rinpoche/170/2 "2024-12-23T08:00:39Z")
> [@Ganga\_Gyatso](#): > > This experiment demonstrates the effectiveness of fine-tuning a base STT model on a speaker-specific dataset, leading to significant improvements in transcription accuracy for Kabjye Dilgo Khyentse Rinpoche’s speech. This approach is not only applicable to this particular case but can also be extended to other spiritual figures and single-speaker datasets. In future work, we plan to expand the dataset and explore additional fine-tuning techniques to further enhance the model’s performance and generalizability. Could you give figures in your conclusion? Your hypothesis in the introduction is “Our hypothesis is that this fine-tuning will improve transcription accuracy for his voice compared to the baseline model.” so I expect to learn more about it in the conclusion. ---
**Author:** ![openpecha](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.webuddhist.com/openpecha/32/78_2.png) [@openpecha](https://forum.webuddhist.com/u/openpecha)\ **Post date:** [December 23, 2024, 11:41am UTC](https://forum.webuddhist.com/t/a-custom-asr-model-to-transcribe-the-speech-of-kabjye-dilgo-khyentse-rinpoche/170/3 "2024-12-23T11:41:30Z")
yes sure genla. updated the conclusion. ---
**Author:** ![Kaldan](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.webuddhist.com/kaldan/32/43_2.png) [@Kaldan](https://forum.webuddhist.com/u/Kaldan)\ **Post date:** [January 6, 2025, 4:07am UTC](https://forum.webuddhist.com/t/a-custom-asr-model-to-transcribe-the-speech-of-kabjye-dilgo-khyentse-rinpoche/170/4 "2025-01-06T04:07:01Z")
@Ganga_Gyatso if we get more data of rinpoche in future, how shall we proceed the training. Should we training from the latest checkpoint of the base model with old data and new data combined or should we train with new data on fine tuned checkpoint. ---
**Author:** ![Ganga\_Gyatso](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.webuddhist.com/ganga_gyatso/32/166_2.png) [@Ganga\_Gyatso](https://forum.webuddhist.com/u/Ganga_Gyatso)\ **Post date:** [January 6, 2025, 5:06am UTC](https://forum.webuddhist.com/t/a-custom-asr-model-to-transcribe-the-speech-of-kabjye-dilgo-khyentse-rinpoche/170/5 "2025-01-06T05:06:16Z")
Training from scratch with the combined old and new datasets ensures balanced learning and avoids overfitting to patterns from the previously fine-tuned checkpoint. It leverages the base model’s generalizability, treats all data equally, and eliminates biases toward earlier fine-tuned data. Since the dataset is small (\<50 hours), training from scratch is cost-effective, quick (under $10), and provides a fresh optimization process for better convergence and flexibility for future fine-tuning.