# Scraping Tibetan News Websites

**URL:** <https://forum.webuddhist.com/t/scraping-tibetan-news-websites/268>\
**Category:** 🛒 Data Collection SIG\
**Tags:** web-scraping, dataset, docs\
**Created:** [May 30, 2025, 9:29am UTC](https://forum.webuddhist.com/t/scraping-tibetan-news-websites/268 "2025-05-30T09:29:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![uchiha\_tashi](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.webuddhist.com/uchiha_tashi/32/45_2.png) [@uchiha\_tashi](https://forum.webuddhist.com/u/uchiha_tashi)\
**Post date:** [May 30, 2025, 9:29am UTC](https://forum.webuddhist.com/t/scraping-tibetan-news-websites/268/1 "2025-05-30T09:29:18Z")

</div>

### Why We’re Doing This

To build a strong Tibetan-to-English translation model, we need lots of real Tibetan text (and audio when available). Our goal is to gather articles from many Tibetan news sites, organize them well, and store them safely. That way, our machine translation systems can learn from a rich dataset of real-world content or for other use cases.

### What We’ve Built

We made a set of Python scripts that:

1. **Collect article links** from key Tibetan news websites.
2. **Download and parse** the articles’ text.
3. **Fetch audio files** when the site offers them.
4. **Save everything** in a clear structure, both on our Git repository (code) and in AWS S3(data).

Every article we scrape—whether it’s just text or text plus audio—gets a standardized `.json` and `.mp3` file.

### Where the Data Lives

- **In the repo** under `News_Articles/`, sorted by region (India or Tibet) and by website name.

- **On S3** , split across two buckets:

We use `.json` files for article text and metadata, and `.mp3` for audio.

### How the Repository Is Organized

```auto
tibetan-news-article-scraping/
├── News_Articles/
│ ├── in_india_website/ ← Indian-based Tibetan news
│ └── in_tibet_website/ ← Mainland Tibet news
├── Web Translation/ ← Glossaries from Glosbe & LinguaTools
├── wikipedia/ ← Wikipedia dumps
├── test code/ ← Notebooks for conversion & audio tests
└── README.md

```

Repo link: [GitHub - OpenPecha/tibetan-news-article-scraping](https://github.com/OpenPecha/tibetan-news-article-scraping)

Inside each site folder you’ll find:

- A folder for raw data
- An `analysis` file with stats on total articles, audio status, failures, and more

### Our Scraping Pipeline

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/openpecha/original/1X/188d45bfb55378ce3271cce9f549ae361cb0b2a0.png)

1. **URL Collection** : Crawl site sections, gather links, remove duplicates, and check they work.
2. **Content Extraction** : Parse HTML for text, download audio if it exists, and grab metadata (author, date, tags).
3. **Data Processing** : (In progress) We will later add cleaning, language checks, and formatting steps.
4. **Storage** : Save JSON, push to S3, commit to Git, and verify backups.

### Keeping Track of Progress

Each site folder has an `analysis` file. It shows:

- How many articles we downloaded
- Which audio files succeeded or failed
- Word-count and title statistics
- Any errors and why they happened  
 ![image](https://canada1.discourse-cdn.com/flex003/uploads/openpecha/original/1X/1c6dd60325e4976462e647c9b9ba7fe8bd58a6be.png)

These stats help us spot problems and measure our coverage.

### What We’ve Collected So Far

- **All “new\_news\_Articles”** (sites under `in_tibet_website/`) on S3:

- **All “News Article”** (sites under `in_india_website/`) on S3:

- **Radio Free Asia (RFA)** data in `s3://voa-rfa-data/RFA_Tibetan/`:

- **VOA Tibetan** in `s3://voa-rfa-data/VOA_Tibetan/` after cleanup:

### For more information on Data Structure and Scraped News Sources, please see this link: [OpenPecha/tibetan-news-article-scraping](https://github.com/OpenPecha/tibetan-news-article-scraping#readme)