Monitoring Foreign-Language News with Local Models

The Shape of the Answer

Before any code, it’s worth being precise about what makes this hard, because each property drives a design decision.

The clock. Asian outlets publish during their working day, which is overnight in North America. This is a batch problem, not a streaming one. You do not need a live feed; you need three runs while you sleep and a digest at breakfast. That single realization removes an enormous amount of complexity, because nothing has to be fast.

The language. Titles and leads arrive in Chinese, Japanese, and Korean. Translation is the only stage in this pipeline with a real per-item cost, whether that cost is money on an API or seconds of GPU time locally.

The ratio. A broad set of tech sources produces hundreds of items a night, and for a narrow beat, single digits are worth reading. This is a filtering problem wearing a translation problem’s clothing.

Put those together and the architecture follows. Batch, so nothing needs to be fast. Filter hard and early, because most items are junk. Translate late, because it’s the expensive step. Score before translating, because you cannot afford to translate everything you might want to score.

Local versus API models is the other early decision. A hosted model is better and easier, and for a low-volume job it’s cheap. It is also metered, which changes how you design: if every translation costs money you become reluctant to over-fetch, and reluctance is the wrong instinct for a collector. Running locally means the marginal cost of an extra source is zero, and you tune for quality instead of invoice. The tradeoff is real: setup, a machine that stays on, and models that are good rather than excellent. For translating hardware headlines, good is enough.

Declaring Sources

Every source lives in one YAML file, and adding or removing one never touches code:

sources:
  - id: ithome
    name: ITHome
    type: rss
    url: https://www.ithome.com/rss/
    tier: a
    lang: cn
    polling: each
    enabled: true

  - id: expreview-news
    name: Expreview news index
    type: scrape
    fetch_via: scrapling
    url: https://www.expreview.com/portal.php?mod=indexNew&contentType=news&page=1
    selector: 'a[href*="expreview.com/"][href$=".html"]'
    tier: a
    lang: cn
    polling: each
    enabled: true

The fields earn their place. type selects a fetcher: a normal feed, a scrape, or a syndication endpoint. fetch_via picks the backend when scraping. lang tells the translate stage what it’s dealing with and lets English and German sources skip translation entirely. tier feeds the scoring rubric, so a first-party outlet outranks an aggregator on identical text. And enabled is the pruning mechanism: when a source breaks, you set it to false with a comment explaining why, and the entry stays as documentation rather than disappearing into git history.

The working set is thirteen sources: mainland Chinese outlets, Taiwanese press, Japanese retail and news blogs, Korean hardware forums, and one English benchmark database. Breadth matters more than you’d think, because the same leak surfacing on four sites is itself a signal, which Step 4 exploits.

Three fetch backends, chosen per source. Most feeds are fine with plain HTTP and a feed parser. Some sit behind a WAF that rejects anything that looks automated, and those go through Scrapling locally. A cloud option (Firecrawl) covers anything Scrapling can’t handle. The abstraction is what matters: the fetcher is a per-source declaration, so a site changing its posture is a one-line config edit rather than a code change.

When RSS Lies

This is where the actual time goes, and it is worth documenting properly because “the feed doesn’t work” turns out to be four distinct problems that look identical from the outside.

The feed exists and is broken. One mainland outlet’s site-wide RSS returns a 302 to an error page. Not a 404, not a block, just a redirect into nothing. The fix was per-category feeds, which are alive and well at a different subdomain, and are arguably better since they let the beat filter start at the source.

The feed exists and is defended. Another outlet’s feed sits behind a WAF that returns a challenge instead of XML. The feed itself is fine; you simply cannot reach it with a plain HTTP client. The fix was to stop trying and scrape the news index page instead, with a CSS selector pulling article links, routed through Scrapling.

The block is cleared and it still doesn’t work. One forum was the instructive case. Scrapling gets past Cloudflare cleanly, the page arrives, and the selector returns zero items. That is not a blocking problem at all, it’s a page-structure problem: the forum runs software whose markup doesn’t match the assumed selector. If you conflate the two, you spend an evening tuning fetch behavior that already works. Always confirm you got the page before blaming the fetch.

The endpoint is simply gone. A syndication endpoint that used to return posts now returns nothing parseable. Self-hosting RSSHub is the fallback, but that path needs an auth token that is fragile and 403s. That source stays disabled.

A fifth, milder case: one Japanese source publishes news and daily sales posts through the same feed, so it needs a URL filter dropping specific path prefixes before anything else looks at it.

The lesson is procedural rather than technical. When a source fails, establish which failure it is before reaching for a tool: did the request succeed, did you get HTML or a challenge page, and did the parser find nodes? Three checks, and they point at three different fixes. Then, if the answer is “this needs more work than it’s worth today,” set enabled: false with a note and move on. A collector with eleven working sources beats one with thirteen sources and two that throw exceptions every night.

Filter Before You Spend

Everything in this step is free, so it all happens before anything expensive.

Age. Anything with a timestamp older than a cutoff (36 hours here) gets dropped. Undated items from scraped index pages pass through rather than being discarded, since absence of a date is not evidence of staleness.

Deduplication, in three layers. A canonicalized URL hash catches the identical link. A normalized title hash (lowercased, punctuation stripped, whitespace collapsed) catches the same story posted at two URLs. Then fuzzy matching with rapidfuzz’s token_set_ratio at a threshold of 85 catches the same story with a reworded headline.

Two details make this work better than it sounds. The comparison runs against a 30-day SQLite window of items already sent, so a story that resurfaces a week later doesn’t come back. And it also runs within the current batch, which matters enormously with thirteen sources on one beat: a genuine leak gets posted by four outlets inside an hour, and without in-batch dedupe your eight-item digest is the same story four times.

A pre-translate content filter. This is the subtle one. Some junk is only recognizable in the source language, because a digest post is titled 早报 or 汇总 and there is no English text yet to match against. So the filter carries source-language patterns for roundups, daily-briefing posts, and tutorial content, and runs before translation. Anything it catches is junk you never paid to translate.

On a real run these stages took 165 fetched items down to 143 before anything expensive happened.

Score Before You Translate

This is the centerpiece, and it rests on one trick that sounds obvious once stated and is easy to miss: make the keyword lists bilingual, and the scoring rubric works on untranslated text.

The beat here is CPUs, GPUs, memory, and storage, plus motherboards and chipsets. So the beat vocabulary carries both sides:

NARROW_BEAT = {
    "ryzen", "epyc", "threadripper", "xeon", "core ultra",
    "zen 5", "zen 6", "x3d",
    "锐龙", "至强",
    "rtx", "radeon rx", "geforce", "intel arc",
    "显卡", "显存",
    "ddr5", "gddr7", "hbm",
    "内存条", "内存模组",
    "nvme", "nand", "ssd controller",
    "phison", "silicon motion", "maxio", "kioxia",
    "固态硬盘", "主控", "闪存",
    "x870", "z890", "am5", "lga1851",
    "主板", "芯片组",
}

锐龙 is Ryzen. 显卡 is graphics card. 固态硬盘 is SSD, 主控 is controller, 闪存 is flash. Part numbers and vendor names are already language-neutral, which does a lot of the work for free. Put those together and a Chinese headline about an SSD controller scores correctly without a single token of translation.

The rubric is a hundred points across seven weighted components:

Before scoring, a hard-exclude list drops entire categories outright: phones, EVs, consoles, and AI-industry personality news. These are legitimate tech stories that are simply not this beat, and dropping them at zero cost is better than ranking them low. Regex patterns catch digest posts and tutorial or explainer content, which are formats rather than topics and never belong in a news digest.

Then the gate: only the top K by score are translated, K being 40 here. On the run cited earlier, 143 items were scored and 40 went to the model. 103 items, 72% of the batch, were sorted and discarded without the GPU ever waking up.

The Local Model Stages

Two models, both local, doing two different jobs.

Translation runs a mid-size multilingual model, and the prompt does most of the work:

Translate the following hardware-news headline and lead from {lang} to English.
Preserve product names, part numbers, units (MB/s, GHz, V, W, $), percentages,
and benchmark scores exactly. Do not paraphrase. Do not add commentary.

Return ONLY a JSON object: {"title": "...", "lead": "..."}

That instruction to preserve part numbers and units is not decoration. For hardware news the numbers are the story, and a translator doing its normal job of producing fluent readable prose will happily round “7,450 MB/s” into “around 7.5 GB/s” or turn a specific SKU into a generic product description. Fluency is the enemy here. You want a literal, boring translation that keeps every figure intact.

Two implementation notes worth stealing. Ask for JSON and request it via the API’s JSON mode, but retry once without JSON mode if parsing fails, because some models ignore the flag and return prose that happens to contain the answer. And skip translation entirely for languages you can already read, via a passthrough list, rather than paying to translate English into English.

A second filter runs after translation, and it catches a different class of junk than the first one. Once text is in English, an entirely new set of exclusions becomes matchable: AI-executive news, phone launches, and consumer-electronics stories that carried no matching keyword in their original language. Filtering twice, once on each side of the translation, catches things neither pass would alone.

Then re-score. The first scoring pass ran on original-language text and did well, but now there is English text contributing too, and the beat vocabulary matches more of it. Items shuffle, which is the point.

Drafting uses a larger model on only the top handful, generating a skeleton outline for each story against a voice guide describing tone and structure. This is the one stage where a bigger, slower model is worth it, precisely because it runs on three items rather than forty. The output is an outline to work from, not copy to publish.

Delivery, and What’s Still Open

Delivery is a plain HTTP POST to Resend, rendering the digest as markdown and converting to HTML. The email has two sections: fully drafted items with their outlines, and a watchlist of everything else that cleared the bar, each with its score and source.

Two separate floors, and the distinction is deliberate. One score threshold governs inclusion in the email at all, a higher one governs whether an item gets a drafted outline. That lets a quiet night still produce a short watchlist without wasting model time drafting weak stories. And if nothing clears the lower floor, the pipeline sends nothing and logs why. A digest that arrives every morning regardless of whether anything happened trains you to ignore it; silence should mean silence.

The runtime knobs all live in environment variables, so tuning never means editing code: digest size, drafting count, both score floors, the age cutoff, the translate cap, the dedupe window, and the fuzzy-match threshold.

Where this actually stands. It runs end to end. Two logged runs completed and sent digests of eight items each, and the dry-run dump holds real graded output with scores from 65 to 80 and outlines attached. The funnel numbers quoted throughout this guide are measurements from those runs, not estimates.

What is not finished: the cron schedule is written but not installed, so runs are currently manual. And the sender domain is not verified, which on Resend means delivery is limited to the account holder’s own address; sending to anyone else needs the domain’s SPF and DKIM records in place first. Both are small, and both are the kind of thing that is easy to describe as done when it isn’t.

Deliberately out of scope: full-article extraction (titles and leads are enough to decide whether to go read the original, and storing more raises questions this project has no reason to raise), images, any kind of web UI, and the RSSHub path for social sources, which was tried and shelved.

What You Spent

Almost nothing recurring, which is the entire argument for the local-model approach.

• Ollama and both models: $0
• Python, SQLite, the libraries: $0
• Resend: free tier covers a few digests a night comfortably
• Firecrawl: optional, only if a source needs the cloud backend
• Hardware: a machine you already have that stays on overnight, with enough GPU memory for a mid-size model

The comparison that matters is against doing the same thing on a hosted translation or LLM API. Translating a few dozen items a night is not expensive in absolute terms, but it is metered, and metering changes behavior. Once every fetch has a downstream cost you start trimming sources, raising thresholds, and translating less, which is exactly backwards for a tool whose job is to not miss things. Running locally, adding a fourteenth source costs nothing, and the scoring gate exists to save time rather than money.

The honest counterargument: a hosted frontier model translates better than a local mid-size one, and for prose where nuance matters that gap is real. For hardware headlines, where the requirement is preserving part numbers and not paraphrasing figures, the local model is entirely sufficient, and the prompt matters more than the model does.