{"id":802,"date":"2026-04-27T06:58:28","date_gmt":"2026-04-26T22:58:28","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=802"},"modified":"2026-04-27T06:58:28","modified_gmt":"2026-04-26T22:58:28","slug":"how-to-build-smarter-multilingual-text-wrapping-with-budoux-through-parsing-html-rendering-model-introspection-and-toy-training","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=802","title":{"rendered":"How to Build Smarter Multilingual Text Wrapping with BudouX Through Parsing, HTML Rendering, Model Introspection, and Toy Training"},"content":{"rendered":"

In this tutorial, we explore how we use BudouX to bring intelligent, phrase-aware line breaking to languages where whitespace is not naturally present, such as Japanese, Chinese, and Thai. We begin by setting up the library and working with its default parsers to understand how raw text is segmented into meaningful chunks. We then move into HTML transformation, where we visually see how BudouX improves readability in constrained layouts by inserting invisible breakpoints. As we progress, we dive deeper into the underlying model, inspecting its learned features and weights to understand how decisions are made. We also experiment with custom model manipulation, integrate BudouX into practical workflows like line wrapping and JSON-based pipelines, and evaluate its performance. Also, we build a minimal end-to-end training pipeline to gain intuition about how such lightweight ML models are constructed.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import subprocess, sys\ndef pip(*pkgs):\n   subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *pkgs])\npip(\"budoux\")\n\n\nimport json, time, textwrap, html, random, re, os, tempfile\nfrom pathlib import Path\nimport budoux\nfrom IPython.display import HTML, display, Markdown\n\n\nprint(f\"\"\u2705\" BudouX version: {budoux.__version__ if hasattr(budoux,'__version__') else 'installed'}\")\n\n\ndef header(title):\n   display(Markdown(f\"## {title}\"))\n\n\nheader(\"1\u20e3 Default parsers \u2014 Japanese \/ Chinese (Simplified & Traditional) \/ Thai\")\n\n\nsamples = {\n   \"Japanese (ja)\":           (\"\u4eca\u65e5\u306f\u5929\u6c17\u3067\u3059\u3002BudouX\u306f\u6a5f\u68b0\u5b66\u7fd2\u3092\u7528\u3044\u305f\u6539\u884c\u6574\u5f62\u30c4\u30fc\u30eb\u3067\u3059\u3002\",\n                               budoux.load_default_japanese_parser()),\n   \"Simplified Chinese\":      (\"\u4eca\u5929\u662f\u6674\u5929\u3002BudouX \u662f\u4e00\u4e2a\u4f7f\u7528\u673a\u5668\u5b66\u4e60\u7684\u6362\u884c\u6574\u7406\u5de5\u5177\u3002\",\n                               budoux.load_default_simplified_chinese_parser()),\n   \"Traditional Chinese\":     (\"\u4eca\u5929\u662f\u6674\u5929\u3002BudouX \u662f\u4e00\u500b\u4f7f\u7528\u6a5f\u5668\u5b78\u7fd2\u7684\u63db\u884c\u6574\u7406\u5de5\u5177\u3002\",\n                               budoux.load_default_traditional_chinese_parser()),\n   \"Thai (th)\":               (\"\u0e27\u0e31\u0e19\u0e19\u0e35\u0e49\u0e2d\u0e32\u0e01\u0e32\u0e28\u0e14\u0e35\u0e21\u0e32\u0e01\u0e41\u0e25\u0e30\u0e09\u0e31\u0e19\u0e2d\u0e22\u0e32\u0e01\u0e2d\u0e2d\u0e01\u0e44\u0e1b\u0e40\u0e14\u0e34\u0e19\u0e40\u0e25\u0e48\u0e19\u0e17\u0e35\u0e48\u0e2a\u0e27\u0e19\u0e2a\u0e32\u0e18\u0e32\u0e23\u0e13\u0e30\",\n                               budoux.load_default_thai_parser()),\n}\nfor name, (text, parser) in samples.items():\n   chunks = parser.parse(text)\n   print(f\"n\u2022 {name}\")\n   print(f\"  raw   : {text}\")\n   print(f\"  parsed: {' | '.join(chunks)}    ({len(chunks)} phrases)\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We install BudouX and set up all required imports to begin working with the library. We load default parsers for multiple languages and pass sample sentences through them to observe how the text is segmented into meaningful phrases. This helps us understand the core functionality of BudouX and how it handles different linguistic structures out of the box.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
header(\"2\u20e3 HTML translation with `translate_html_string`\")\n\n\nja_parser = budoux.load_default_japanese_parser()\nhtml_in = \"\u4eca\u65e5\u306f<b>\u3068\u3066\u3082\u5929\u6c17<\/b>\u3067\u3059\u3002\"\nhtml_out = ja_parser.translate_html_string(html_in)\nvisible = html_out.replace(\"u200b\", \"\u00b7\")\nprint(\"Input  HTML :\", html_in)\nprint(\"Output HTML :\", html_out)\nprint(\"Visualised  :\", visible)\n\n\ndemo_text = (\"BudouX\u306f\u6a5f\u68b0\u5b66\u7fd2\u3092\u7528\u3044\u3066\u3001CJK\u8a00\u8a9e\u306e\u6587\u7ae0\u3092\u610f\u5473\u306e\u3042\u308b\"\n            \"\u30d5\u30ec\u30fc\u30ba\u306b\u5206\u5272\u3057\u3001\u81ea\u7136\u306a\u4f4d\u7f6e\u3067\u6539\u884c\u3067\u304d\u308b\u3088\u3046\u306b\u3057\u307e\u3059\u3002\")\ndemo_html = ja_parser.translate_html_string(demo_text)\ndisplay(HTML(f\"\"\"\n<div style=\"display:flex; gap:16px; font-family:'Hiragino Sans',sans-serif;\">\n <div style=\"width:140px; border:2px solid #c33; padding:8px;\">\n    <b style=\"color:#c33;\">\"\u274c\" Plain<\/b><br>{demo_text}\n <\/div>\n <div style=\"width:140px; border:2px solid #2a8; padding:8px;\">\n    <b style=\"color:#2a8;\">\"\u2705\" BudouX<\/b><br>{demo_html}\n <\/div>\n<\/div>\n\"\"\"))\n\n\nheader(\"3\u20e3 Model introspection \u2014 features & weights\")\n\n\nmodel_dir = Path(budoux.__file__).parent \/ \"models\"\nprint(\"Bundled models:\", [p.name for p in model_dir.glob(\"*.json\")])\n\n\nwith open(model_dir \/ \"ja.json\", encoding=\"utf-8\") as f:\n   ja_model = json.load(f)\n\n\nprint(f\"nFeature categories in ja.json: {list(ja_model.keys())}\")\ntotal = sum(len(v) for v in ja_model.values())\nprint(f\"Total learned features: {total:,}\")\nfor cat, feats in ja_model.items():\n   print(f\"  \u2022 {cat:5s}  \u2192 {len(feats):,} features\")\n\n\nflat = [(cat, feat, w) for cat, d in ja_model.items() for feat, w in d.items()]\nflat.sort(key=lambda x: x[2], reverse=True)\nprint(\"nTop 5 features that vote 'BREAK HERE':\")\nfor cat, feat, w in flat[:5]:\n   print(f\"  [{cat}] {feat!r}  \u2192 weight={w}\")\nprint(\"nTop 5 features that vote 'DO NOT BREAK':\")\nfor cat, feat, w in flat[-5:]:\n   print(f\"  [{cat}] {feat!r}  \u2192 weight={w}\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We use BudouX to transform HTML strings by inserting invisible breakpoints that improve text wrapping. We visualize the effect by comparing plain text rendering with BudouX-enhanced output in a constrained layout. We also inspect the internal model structure, exploring feature categories and weights to understand how the segmentation decisions are learned.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
header(\"4\u20e3 Loading a custom model with `budoux.Parser(model)`\")\n\n\nneutered = {cat: {k: 0 for k in d} for cat, d in ja_model.items()}\nflat_parser = budoux.Parser(neutered)\nprint(\"All-zero model output :\", flat_parser.parse(\"\u4eca\u65e5\u306f\u5929\u6c17\u3067\u3059\u3002\"))\nprint(\"Default model output  :\", ja_parser.parse(\"\u4eca\u65e5\u306f\u5929\u6c17\u3067\u3059\u3002\"))\n\n\nheader(\"5\u20e3 Practical: custom separators, line-wrapping, JSON export\")\n\n\ndef wrap_with_budoux(text, parser, max_width=12, sep=\"n\"):\n   lines, current = [], \"\"\n   for phrase in parser.parse(text):\n       if len(current) + len(phrase) > max_width and current:\n           lines.append(current); current = phrase\n       else:\n           current += phrase\n   if current: lines.append(current)\n   return sep.join(lines)\n\n\nnovel = (\"\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\u3002\u540d\u524d\u306f\u307e\u3060\u7121\u3044\u3002\u3069\u3053\u3067\u751f\u308c\u305f\u304b\u3068\u3093\u3068\u898b\u5f53\u304c\u3064\u304b\u306c\u3002\"\n        \"\u4f55\u3067\u3082\u8584\u6697\u3044\u3058\u3081\u3058\u3081\u3057\u305f\u6240\u3067\u30cb\u30e3\u30fc\u30cb\u30e3\u30fc\u6ce3\u3044\u3066\u3044\u305f\u4e8b\u3060\u3051\u306f\u8a18\u61b6\u3057\u3066\u3044\u308b\u3002\")\nprint(\"Wrapped at width 12:\")\nprint(wrap_with_budoux(novel, ja_parser, max_width=12))\n\n\nseg = {\"text\": novel, \"phrases\": ja_parser.parse(novel)}\nprint(\"nJSON payload (first 120 chars):\", json.dumps(seg, ensure_ascii=False)[:120], \"...\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We experiment with a custom model by modifying all feature weights to zero and observing how segmentation behavior changes. We then implement a practical text-wrapping function that respects BudouX phrase boundaries for better readability. Finally, we export the segmented output as JSON, making it easy to integrate into downstream systems or front-end applications.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
header(\"6\u20e3 Performance benchmark\")\n\n\nbig_text = novel * 200\nt0 = time.perf_counter()\nphrases = ja_parser.parse(big_text)\nelapsed = time.perf_counter() - t0\nprint(f\"Parsed {len(big_text):,} chars \u2192 {len(phrases):,} phrases \"\n     f\"in {elapsed*1000:.1f} ms  ({len(big_text)\/elapsed\/1000:.0f}k chars\/sec)\")\n\n\nheader(\"7\u20e3 Mini end-to-end trainer (toy demo)\")\n\n\ntraining_lines = [\n   \"\u79c1\u306f\u2581\u9045\u523b\u9b54\u3067\u3001\u2581\u5f85\u3061\u5408\u308f\u305b\u306b\u2581\u3044\u3064\u3082\u2581\u9045\u523b\u3057\u3066\u2581\u3057\u307e\u3044\u307e\u3059\u3002\",\n   \"\u30e1\u30fc\u30eb\u3067\u2581\u5f85\u3061\u5408\u308f\u305b\u2581\u76f8\u624b\u306b\u2581\u4e00\u8a00\u3001\u2581\u300c\u3054\u3081\u3093\u306d\u300d\u3068\u2581\u8b1d\u308c\u3070\u2581\u3069\u3046\u306b\u304b\u2581\u306a\u308b\u3068\u2581\u601d\u3063\u3066\u2581\u3044\u307e\u3057\u305f\u3002\",\n   \"\u6d77\u5916\u3067\u306f\u2581\u30b1\u30fc\u30bf\u30a4\u3092\u2581\u6301\u3063\u3066\u2581\u3044\u306a\u3044\u3002\",\n   \"\u4eca\u65e5\u306f\u2581\u3068\u3066\u3082\u2581\u3044\u3044\u2581\u5929\u6c17\u3067\u3059\u3002\",\n   \"\u660e\u65e5\u306f\u2581\u96e8\u304c\u2581\u964d\u308b\u2581\u304b\u3082\u2581\u3057\u308c\u307e\u305b\u3093\u3002\",\n   \"\u9031\u672b\u306f\u2581\u53cb\u9054\u3068\u2581\u6620\u753b\u3092\u2581\u898b\u306b\u2581\u884c\u304d\u307e\u3059\u3002\",\n] * 20\n\n\nSEP = \"u2581\"\n\n\ndef extract_features(s, i):\n   def g(idx): return s[idx] if 0 <= idx < len(s) else \"\"\n   feats = []\n   for off in (-3,-2,-1,0,1,2):\n       feats.append(f\"U{off}:{g(i+off)}\")\n   for off in (-2,-1,0,1):\n       feats.append(f\"B{off}:{g(i+off)}{g(i+off+1)}\")\n   for off in (-1,0):\n       feats.append(f\"T{off}:{g(i+off)}{g(i+off+1)}{g(i+off+2)}\")\n   return feats\n\n\ndef make_examples(lines):\n   X, y = [], []\n   for line in lines:\n       clean = line.replace(SEP, \"\")\n       breaks = set()\n       j = 0\n       for ch in line:\n           if ch == SEP: breaks.add(j)\n           else: j += 1\n       for i in range(1, len(clean)):\n           X.append(extract_features(clean, i))\n           y.append(1 if i in breaks else -1)\n   return X, y\n\n\nX, y = make_examples(training_lines)\nprint(f\"Training examples: {len(X)}  (positives: {sum(1 for v in y if v==1)})\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We benchmark BudouX\u2019s performance to evaluate its efficiency in processing large amounts of text. We then begin constructing a minimal training pipeline by preparing labeled data and extracting features around potential breakpoints. This gives us insight into how training data is structured and how features contribute to segmentation decisions.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def adaboost(X, y, rounds=80):\n   n = len(y)\n   w = [1\/n]*n\n   feat_set = sorted({f for fx in X for f in fx})\n   fmap = [set(fx) for fx in X]\n   model_rounds = []\n   for r in range(rounds):\n       best_feat, best_err, best_pol = None, 1.0, 1\n       for f in feat_set:\n           err_pos = sum(w[i] for i in range(n) if (f in fmap[i]) != (y[i]==1))\n           err_neg = 1 - err_pos\n           if err_pos < best_err: best_feat, best_err, best_pol = f, err_pos, +1\n           if err_neg < best_err: best_feat, best_err, best_pol = f, err_neg, -1\n       if best_err >= 0.5 - 1e-9: break\n       eps = max(best_err, 1e-6)\n       alpha = 0.5 * ( (1-eps)\/eps ) ** 0.5\n       new_w = []\n       for i in range(n):\n           pred = best_pol if best_feat in fmap[i] else -best_pol\n           new_w.append(w[i] * (0.5 if pred == y[i] else 2.0))\n       s = sum(new_w); w = [x\/s for x in new_w]\n       model_rounds.append((best_feat, best_pol, alpha))\n   return model_rounds\n\n\nprint(\"Training (this is a toy trainer \u2014 be patient ~10s)...\")\nt0 = time.perf_counter()\nrounds = adaboost(X, y, rounds=60)\nprint(f\"Done in {time.perf_counter()-t0:.1f}s, {len(rounds)} stumps kept.\")\n\n\ncorrect = 0\nfor fx, label in zip(X, y):\n   score = sum(a if (f in fx) == (p==1) else -a for f,p,a in rounds)\n   pred = 1 if score > 0 else -1\n   correct += (pred == label)\nprint(f\"Training accuracy of toy model: {correct\/len(X)*100:.1f}%\")\nprint(\"\"\ud83d\udc49\" For a production model, use `scripts\/train.py` from the BudouX repo with the matching feature extractor \u2014 this section is illustrative.\")\n\n\nheader(\"8\u20e3 Real-world demo \u2014 narrow column comparison\")\n\n\nparagraph = (\"BudouX\u306fGoogle\u304c\u958b\u767a\u3057\u305f\u30aa\u30fc\u30d7\u30f3\u30bd\u30fc\u30b9\u306e\u6539\u884c\u30e9\u30a4\u30d6\u30e9\u30ea\u3067\u3059\u3002\"\n            \"\u6a5f\u68b0\u5b66\u7fd2\u30e2\u30c7\u30eb\u3092\u4f7f\u3063\u3066\u3001\u6587\u7ae0\u3092\u610f\u5473\u306e\u3042\u308b\u30d5\u30ec\u30fc\u30ba\u306b\u5206\u5272\u3057\u3001\"\n            \"\u8aad\u307f\u3084\u3059\u3044\u4f4d\u7f6e\u3067\u306e\u307f\u6539\u884c\u304c\u8d77\u3053\u308b\u3088\u3046\u306b\u3057\u307e\u3059\u3002\"\n            \"\u4f9d\u5b58\u95a2\u4fc2\u304c\u306a\u304f\u8efd\u91cf\u306a\u305f\u3081\u3001\u30a6\u30a7\u30d6\u30b5\u30a4\u30c8\u3084\u30e2\u30d0\u30a4\u30eb\u30a2\u30d7\u30ea\u306b\"\n            \"\u7c21\u5358\u306b\u7d44\u307f\u8fbc\u3080\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002\")\ndisplay(HTML(f\"\"\"\n<div style=\"display:flex; gap:24px; font-family:'Hiragino Sans','Yu Gothic',sans-serif; font-size:15px;\">\n <div style=\"flex:1; border:2px solid #c33; padding:12px; max-width:180px;\">\n   <b style=\"color:#c33;\">Without BudouX<\/b>\n   <p style=\"line-height:1.7;\">{paragraph}<\/p>\n <\/div>\n <div style=\"flex:1; border:2px solid #2a8; padding:12px; max-width:180px;\">\n   <b style=\"color:#2a8;\">With BudouX<\/b>\n   <p style=\"line-height:1.7;\">{ja_parser.translate_html_string(paragraph)}<\/p>\n <\/div>\n<\/div>\n<p style=\"font-size:12px;color:#666;\">Resize the browser\/Colab pane to see the difference more clearly \u2014 BudouX never breaks a phrase mid-word.<\/p>\n\"\"\"))\n\n\nprint(\"n\"\ud83c\udf38\" Tutorial complete. Try plugging BudouX output into your own UI.\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We implement a simple AdaBoost-based training loop to build a toy segmentation model from scratch. We evaluate the model\u2019s accuracy to understand how well it learns phrase boundaries from the data. Finally, we present a real-world comparison that shows how BudouX improves readability in narrow layouts, reinforcing its practical value.<\/p>\n
In conclusion, we developed a comprehensive understanding of how BudouX applies machine learning to solve the nuanced problem of natural line breaking in CJK and similar languages. We saw how it operates efficiently without heavy dependencies, making it ideal for web and mobile integrations. Through hands-on exploration, from parsing and HTML rendering to model introspection, customization, and even training, we learned how to use BudouX and also how to extend and adapt it for our own use cases. This equips us with both the practical tools and conceptual clarity needed to incorporate phrase-aware text segmentation into real-world applications with confidence.<\/p>\n
\n
Check out\u00a0the\u00a0Full Codes here<\/a><\/strong>.\u00a0<\/strong>Find 100s of ML\/Data Science\u00a0Colab Notebooks here<\/a><\/strong>. Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0130k+ ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.?\u00a0Connect with us<\/mark><\/a><\/strong><\/p>\n
The post How to Build Smarter Multilingual Text Wrapping with BudouX Through Parsing, HTML Rendering, Model Introspection, and Toy Training<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
In this tutorial, we explore h…<\/p>\n","protected":false},"author":1,"featured_media":29,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-802","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/802","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=802"}],"version-history":[{"count":0,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/802\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/media\/29"}],"wp:attachment":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=802"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=802"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=802"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}