# \#data-cleaning
**URL:** https://forum.webuddhist.com/tag/data-cleaning/89.md
[Latest](https://forum.webuddhist.com/latest.md) · [Categories](https://forum.webuddhist.com/categories.md) · [Tags](https://forum.webuddhist.com/tags.md)
---
## [Domain Tagging With Supervised Classification for Retrieval Augmented Translation](https://forum.webuddhist.com/t/domain-tagging-with-supervised-classification-for-retrieval-augmented-translation/188)
**Author:** [@billingsmoore](https://forum.webuddhist.com/u/billingsmoore)\
**Replies:** 0\
**Last updated:** [January 17, 2025, 2:09am UTC](https://forum.webuddhist.com/t/domain-tagging-with-supervised-classification-for-retrieval-augmented-translation/188 "2025-01-17T02:09:12Z")
Domain Tagging With Supervised Clustering for Retrieval Augmented Translation Introduction In a previous post, Domain Tagging With Unsupervised Clustering for Retrieval Augmented Translation, I created domain tags for th…
---
## [Using Topic Modeling to Cluster User Translation (BERTopic)](https://forum.webuddhist.com/t/using-topic-modeling-to-cluster-user-translation-bertopic/111)
**Author:** [@uchiha\_tashi](https://forum.webuddhist.com/u/uchiha_tashi)\
**Replies:** 0\
**Last updated:** [November 28, 2024, 6:57am UTC](https://forum.webuddhist.com/t/using-topic-modeling-to-cluster-user-translation-bertopic/111 "2024-11-28T06:57:20Z")
Using Topic Modeling to Cluster User Translation (BERTopic) Introduction We live in a world overflowing with unstructured text—customer emails, online reviews, tweets, academic papers, and so much more. Believe it or not…
---
## [Sentence Length Proportions As Data Cleaning Heuristic](https://forum.webuddhist.com/t/sentence-length-proportions-as-data-cleaning-heuristic/169)
**Author:** [@billingsmoore](https://forum.webuddhist.com/u/billingsmoore)\
**Replies:** 3\
**Last updated:** [January 7, 2025, 4:01am UTC](https://forum.webuddhist.com/t/sentence-length-proportions-as-data-cleaning-heuristic/169 "2025-01-07T04:01:56Z")
Sentence Length Proportions As Data Cleaning Heuristic Introduction In conversation, Sebastian Nehrdich suggested that sentence lengths may useful as a heuristic for detecting poor quality sentence pairs in the translati…
---
## [Validating Data Cleaning for Translation Model Training](https://forum.webuddhist.com/t/validating-data-cleaning-for-translation-model-training/149)
**Author:** [@billingsmoore](https://forum.webuddhist.com/u/billingsmoore)\
**Replies:** 0\
**Last updated:** [December 7, 2024, 8:58pm UTC](https://forum.webuddhist.com/t/validating-data-cleaning-for-translation-model-training/149 "2024-12-07T20:58:52Z")
Validating Data Cleaning for Translation Model Training openpecha/cleaned\_MT\_v1.0.3 (hereafter “1.0.3”) was recently created as an iterative improvement on openpecha/cleaned\_MT\_v1.0.2 (hereafter “1.0.2”) with respect to…
---
## [Creating openpecha/cleaned\_MT\_v1.0.3](https://forum.webuddhist.com/t/creating-openpecha-cleaned-mt-v1-0-3/142)
**Author:** [@billingsmoore](https://forum.webuddhist.com/u/billingsmoore)\
**Replies:** 0\
**Last updated:** [December 5, 2024, 9:31pm UTC](https://forum.webuddhist.com/t/creating-openpecha-cleaned-mt-v1-0-3/142 "2024-12-05T21:31:05Z")
Creating openpecha/cleaned\_MT\_v1.0.3 The cleaner version of the machine translation dataset openpecha/cleaned\_MT\_v1.0.3 was created by cleaning openpecha/cleaned\_MT\_v1.0.2 with the steps listed below. Each step is accomp…
---
## [Toward a Cleaner Translation Dataset](https://forum.webuddhist.com/t/toward-a-cleaner-translation-dataset/72)
**Author:** [@billingsmoore](https://forum.webuddhist.com/u/billingsmoore)\
**Replies:** 0\
**Last updated:** [November 3, 2024, 4:38pm UTC](https://forum.webuddhist.com/t/toward-a-cleaner-translation-dataset/72 "2024-11-03T16:38:01Z")
Toward a Cleaner Translation Dataset Introduction While performing topic modeling on the translation dataset ‘openpecha/cleaned\_MT\_v1.0.2’ it was discovered that some elements (‘sentence pairs’) of the dataset may presen…
---
## [Topic Modeling Buddhist Material in the Translation Dataset](https://forum.webuddhist.com/t/topic-modeling-buddhist-material-in-the-translation-dataset/91)
**Author:** [@billingsmoore](https://forum.webuddhist.com/u/billingsmoore)\
**Replies:** 0\
**Last updated:** [November 21, 2024, 9:49pm UTC](https://forum.webuddhist.com/t/topic-modeling-buddhist-material-in-the-translation-dataset/91 "2024-11-21T21:49:06Z")
Topic Modeling Buddhist Material in the Translation Dataset Following results from Modeling the Full Translation Dataset, obviously non-Buddhist materials were excluded from the dataset and topic modeling was conducted o…