{"id":976,"date":"2026-05-26T15:25:55","date_gmt":"2026-05-26T07:25:55","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=976"},"modified":"2026-05-26T15:25:55","modified_gmt":"2026-05-26T07:25:55","slug":"design-a-complete-multimodal-rlvr-pipeline-with-open-mm-rl-vision-language-prompting-reward-scoring-and-grpo-export","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=976","title":{"rendered":"Design a Complete Multimodal RLVR Pipeline with Open-MM-RL, Vision-Language Prompting, Reward Scoring, and GRPO Export"},"content":{"rendered":"

In this tutorial, we explore the TuringEnterprises\/Open-MM-RL<\/strong><\/a> <\/strong>dataset as a practical foundation for multimodal reasoning and reinforcement learning with verifiable rewards. We load the dataset, inspect its schema, analyze domains, formats, question lengths, answer types, and image distributions, and visualize representative examples from each domain. We also build a lightweight reward function that checks exact, numeric, fractional, LaTeX, and symbolic answers, giving us a useful way to evaluate model outputs. Finally, we format prompts for vision-language models, optionally test SmolVLM on sample examples, and export the dataset into a GRPO-style structure for future multimodal RL training.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import subprocess, sys\nsubprocess.run([sys.executable, \"-m\", \"pip\", \"-q\", \"install\",\n               \"datasets>=3.0\", \"huggingface_hub>=0.24\", \"transformers>=4.45\",\n               \"Pillow\", \"matplotlib\", \"pandas\", \"numpy\", \"sympy\",\n               \"accelerate\", \"tqdm\"], check=True)\nimport os, re, io, json, math, random, textwrap, hashlib, warnings\nfrom collections import Counter\nfrom pathlib import Path\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nimport sympy as sp\nfrom datasets import load_dataset\nwarnings.filterwarnings(\"ignore\")\nrandom.seed(0); np.random.seed(0)\npd.set_option(\"display.max_colwidth\", 120)\nDS_ID = \"TuringEnterprises\/Open-MM-RL\"\nds = load_dataset(DS_ID, split=\"train\")\nprint(f\"Loaded {DS_ID} \u2014 {len(ds)} rows\")\nprint(\"Features:\", ds.features)\nprint(\"Row 0 keys:\", list(ds[0].keys()))<\/code><\/pre>\n<\/div>\n<\/div>\n
We install all required libraries and import the core tools needed for dataset loading, analysis, visualization, symbolic math, and file handling. We set random seeds for reproducibility and configure pandas so that longer text fields display clearly. We then load the TuringEnterprises\/Open-MM-RL dataset from Hugging Face and inspect its size, features, and first-row structure.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
df = ds.remove_columns([\"images\"]).to_pandas()\ndf[\"n_images\"]    = [len(ex[\"images\"]) for ex in ds]\ndf[\"q_len_chars\"] = df[\"question\"].str.len()\ndf[\"a_len_chars\"] = df[\"answer\"].str.len()\nprint(\"n=== Domain ===\"); print(df[\"domain\"].value_counts())\nprint(\"n=== Format ===\"); print(df[\"format\"].value_counts())\nprint(\"n=== Sub-domain (top by domain) ===\")\nprint(df.groupby(\"domain\")[\"subDomain\"].value_counts().head(15))\nprint(f\"nMean images\/example: {df['n_images'].mean():.2f}   max: {df['n_images'].max()}\")\nprint(f\"Median Q length: {df['q_len_chars'].median():.0f}   \"\n     f\"Median A length: {df['a_len_chars'].median():.0f}\")\nfig, axes = plt.subplots(1, 3, figsize=(15, 4))\ndf[\"domain\"].value_counts().plot.bar(ax=axes[0], color=\"#4C72B0\")\naxes[0].set_title(\"Examples per domain\"); axes[0].set_ylabel(\"count\")\ndf[\"format\"].value_counts().plot.bar(ax=axes[1], color=\"#55A868\")\naxes[1].set_title(\"Image-format type\"); axes[1].tick_params(axis='x', rotation=25)\ndf[\"n_images\"].plot.hist(ax=axes[2], bins=range(1, df[\"n_images\"].max() + 2),\n                        color=\"#C44E52\", edgecolor=\"white\")\naxes[2].set_title(\"Images per example\"); axes[2].set_xlabel(\"n_images\")\nplt.tight_layout(); plt.show()\ndef img_stats(ex):\n   sizes = [im.size for im in ex[\"images\"]]\n   modes = [im.mode for im in ex[\"images\"]]\n   return {\n       \"n_images\": len(sizes),\n       \"min_w\": min(w for w, h in sizes), \"max_w\": max(w for w, h in sizes),\n       \"min_h\": min(h for w, h in sizes), \"max_h\": max(h for w, h in sizes),\n       \"modes\": \"|\".join(sorted(set(modes))),\n       \"total_pixels\": sum(w * h for w, h in sizes),\n   }\nimg_df = pd.DataFrame([img_stats(ex) for ex in ds])\nprint(\"n=== Image resolution stats ===\")\nprint(img_df[[\"min_w\", \"max_w\", \"min_h\", \"max_h\", \"total_pixels\"]].describe().round(0))\nprint(\"nMode mix:\", Counter(\"|\".join(img_df[\"modes\"]).split(\"|\")))<\/code><\/pre>\n<\/div>\n<\/div>\n
We convert the dataset into a DataFrame after removing the image column, then calculate useful fields such as the number of images, question length, and answer length. We analyze domain counts, format distribution, sub-domain breakdowns, and basic text\/image statistics. We also create charts to visualize the number of examples per domain, the image formats, and the distribution of images per example.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def show_example(ex, max_chars=600):\n   print(\"=\" * 80)\n   print(f\"id={ex['conversation_id']}   {ex['domain']} \/ {ex['subDomain']}\")\n   print(f\"format={ex['format']}   n_images={len(ex['images'])}\")\n   print(\"-\" * 80)\n   q = ex[\"question\"][:max_chars] + (\"...\" if len(ex[\"question\"]) > max_chars else \"\")\n   print(\"Q:\", textwrap.fill(q, 100))\n   print(\"-\" * 80)\n   print(\"A (gold):\", ex[\"answer\"])\n   n = len(ex[\"images\"])\n   fig, axes = plt.subplots(1, n, figsize=(5 * n, 5)) if n > 1 \n               else plt.subplots(1, 1, figsize=(6, 6))\n   axes = np.atleast_1d(axes)\n   for ax, im in zip(axes, ex[\"images\"]):\n       ax.imshow(im); ax.set_xticks([]); ax.set_yticks([])\n       ax.set_title(f\"{im.size[0]}\u00d7{im.size[1]}  ({im.mode})\")\n   plt.tight_layout(); plt.show()\nfor dom in df[\"domain\"].unique():\n   idx = int(df[df[\"domain\"] == dom].index[0])\n   show_example(ds[idx])\nLATEX_PAT = re.compile(r\"\\[[sS]+?\\]|\\([sS]+?\\)|$[^$]+$\")\ndf[\"latex_blocks_q\"] = df[\"question\"].apply(lambda s: len(LATEX_PAT.findall(s or \"\")))\ndf[\"latex_blocks_a\"] = df[\"answer\"].apply(lambda s: len(LATEX_PAT.findall(s or \"\")))\nprint(\"n=== LaTeX blocks per field ===\")\nprint(df[[\"latex_blocks_q\", \"latex_blocks_a\"]].describe().round(2))\ndef classify_answer(a):\n   s = (a or \"\").strip().strip(\"$ []\").strip()\n   s_no_dollar = s.replace(\"$\", \"\")\n   if re.fullmatch(r\"-?s*d+(.d+)?s*\", s_no_dollar):       return \"integer\/float\"\n   if any(t in s for t in [\"\\sqrt\", \"\\frac\", \"\\pi\", \"^\", \"_\", \"\\kappa\", \"\\lceil\"]):\n       return \"symbolic\"\n   if re.fullmatch(r\"[-+0-9.\/()s\\a-zA-Z{}]+\", s) and any(c.isdigit() for c in s):\n       return \"numeric_expr\"\n   return \"text\"\ndf[\"answer_type\"] = df[\"answer\"].apply(classify_answer)\nprint(\"n=== Answer-type breakdown ===\"); print(df[\"answer_type\"].value_counts())\nprint(\"n=== Answer-type \u00d7 domain ===\")\nprint(pd.crosstab(df[\"domain\"], df[\"answer_type\"]))<\/code><\/pre>\n<\/div>\n<\/div>\n
We define a helper function to display one representative example from each domain, including its question, gold answer, and associated images. We use this visual inspection step to better understand how multimodal reasoning problems are structured across different domains. We then analyze LaTeX usage in questions and answers, classify answer types, and compare answer-type distributions across domains.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
EXTRACT_PATS = [\n   r\"\\boxed{([^{}]+)}\",\n   r\"finals+answers*[:=]s*([^n]+)\",\n   r\"answers*[:=]s*([^n]+)\",\n]\ndef extract_final(text):\n   if not text: return \"\"\n   for p in EXTRACT_PATS:\n       m = re.search(p, text, flags=re.IGNORECASE)\n       if m: return m.group(1).strip().strip(\".,;\")\n   lines = [l.strip() for l in str(text).strip().splitlines() if l.strip()]\n   return lines[-1] if lines else \"\"\ndef latex_to_sympy(s):\n   s = (s or \"\").strip().strip(\"$\").strip()\n   s = re.sub(r\"^\\[[(]\", \"\", s); s = re.sub(r\"\\[])]$\", \"\", s)\n   s = (s.replace(\"\\pi\", \"pi\").replace(\"\\cdot\", \"*\").replace(\"\\times\", \"*\")\n          .replace(\"\\,\", \"\").replace(\"\\;\", \"\").replace(\"\\!\", \"\"))\n   s = re.sub(r\"\\fracs*{([^{}]+)}s*{([^{}]+)}\", r\"((1)\/(2))\", s)\n   s = re.sub(r\"\\sqrts*{([^{}]+)}\", r\"sqrt(1)\", s)\n   s = s.replace(\"^\", \"**\")\n   s = re.sub(r\"\\[a-zA-Z]+\", \"\", s)\n   s = s.replace(\"{\", \"(\").replace(\"}\", \")\")\n   return s\ndef grade(pred, gold, tol=1e-4):\n   \"\"\"Verifiable reward in [0,1]: exact > numeric > sympy-symbolic > partial.\"\"\"\n   if pred is None or gold is None: return 0.0\n   p = extract_final(str(pred)).strip()\n   g = str(gold).strip()\n   norm = lambda x: re.sub(r\"s+\", \"\", x.lower()).strip(\"$.,;[]()\")\n   if norm(p) == norm(g): return 1.0\n   def to_float(x):\n       try: return float(latex_to_sympy(x))\n       except Exception:\n           try: return float(sp.sympify(latex_to_sympy(x)).evalf())\n           except Exception: return None\n   fp, fg = to_float(p), to_float(g)\n   if fp is not None and fg is not None:\n       if abs(fp - fg) \/ max(1.0, abs(fg)) < tol: return 1.0\n   try:\n       ep = sp.sympify(latex_to_sympy(p)); eg = sp.sympify(latex_to_sympy(g))\n       if sp.simplify(ep - eg) == 0: return 1.0\n   except Exception:\n       pass\n   if norm(g) and norm(g) in norm(p): return 0.5\n   return 0.0\nprint(\"n=== Grader sanity checks ===\")\nfor pred, gold, want in [\n   (\"The answer is \\boxed{120}\",            \"[120]\",            1.0),\n   (\"After computing: 7396 \\pi\",            \"7396\\pi\",         1.0),\n   (\"Final answer: -71\/4\",                   \"-\\frac{71}{4}\",   1.0),\n   (\"Therefore the result is 0.0074\",        \"0.0074\",           1.0),\n   (\"Final answer: nucleus accumbens\",       \"Nucleus accumbens\",1.0),\n   (\"I don't know\",                          \"12\",               0.0),\n]:\n   print(f\"  pred={pred[:38]!r:42s} gold={gold!r:22s} -> r={grade(pred, gold)}  (want {want})\")\nSYSTEM = (\"You are a STEM expert solving multimodal reasoning problems. \"\n         \"You will see a question and one or more figures. \"\n         \"Reason step by step, then end with exactly one line:n\"\n         \"Final answer: <your answer>\")\ndef build_prompt(ex):\n   img_tags = \"n\".join(f\"[Image {i+1}]\" for i in range(len(ex[\"images\"])))\n   return f\"{SYSTEM}nn{img_tags}nnQuestion:n{ex['question']}nnLet's think step by step.\"\nprint(\"n=== Example prompt (truncated) ===\")\nprint(build_prompt(ds[0])[:600], \"...n\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We build a verifiable reward function that extracts final answers and compares predictions against gold answers using exact, numeric, and symbolic matching. We also add a LaTeX-to-SymPy conversion helper, allowing mathematical expressions to be evaluated more reliably. We test the grader with sanity checks and then create a structured prompt format for vision-language model reasoning.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import torch\nUSE_VLM = torch.cuda.is_available()\nprint(f\"CUDA available: {USE_VLM}\")\nif USE_VLM:\n   try:\n       from transformers import AutoProcessor, AutoModelForVision2Seq\n       MODEL_ID = \"HuggingFaceTB\/SmolVLM-Instruct\"\n       print(f\"Loading {MODEL_ID} (this takes ~1 min) ...\")\n       processor = AutoProcessor.from_pretrained(MODEL_ID)\n       model = AutoModelForVision2Seq.from_pretrained(\n           MODEL_ID, torch_dtype=torch.float16, device_map=\"auto\"\n       )\n       def vlm_solve(ex, max_new_tokens=512):\n           imgs = [im.convert(\"RGB\") for im in ex[\"images\"]]\n           content = [{\"type\": \"image\"} for _ in imgs]\n           content.append({\"type\": \"text\", \"text\": build_prompt(ex)})\n           text = processor.apply_chat_template(\n               [{\"role\": \"user\", \"content\": content}], add_generation_prompt=True)\n           inputs = processor(text=text, images=imgs, return_tensors=\"pt\").to(model.device)\n           with torch.no_grad():\n               out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)\n           return processor.batch_decode(\n               out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]\n       rows, sample_idx = [], random.sample(range(len(ds)), 6)\n       for i in sample_idx:\n           ex = ds[i]\n           try:\n               pred = vlm_solve(ex)\n               r = grade(pred, ex[\"answer\"])\n           except Exception as e:\n               pred, r = f\"<error: {e}>\", 0.0\n           rows.append({\"id\": ex[\"conversation_id\"], \"domain\": ex[\"domain\"],\n                        \"reward\": r, \"pred_tail\": pred[-200:]})\n           print(f\"  id={ex['conversation_id']}  {ex['domain']:9s}  r={r:.2f}\")\n       res = pd.DataFrame(rows)\n       print(f\"nMean reward over {len(res)} samples: {res['reward'].mean():.3f}\")\n       print(res.groupby(\"domain\")[\"reward\"].mean().rename(\"avg_reward\"))\n   except Exception as e:\n       print(f\"VLM run failed ({e}); reward & data pipeline remain usable.\")\nelse:\n   print(\"No GPU detected \u2014 skipping live VLM inference (Runtime \u2192 Change runtime type \u2192 GPU).\")\nout_dir = Path(\"\/content\/open_mm_rl_processed\"); out_dir.mkdir(exist_ok=True, parents=True)\nimg_dir = out_dir \/ \"images\"; img_dir.mkdir(exist_ok=True)\nrecords = []\nfor ex in ds:\n   paths = []\n   for j, im in enumerate(ex[\"images\"]):\n       p = img_dir \/ f\"{ex['conversation_id']}_{j}.png\"\n       im.convert(\"RGB\").save(p)\n       paths.append(str(p))\n   records.append({\n       \"id\":         ex[\"conversation_id\"],\n       \"domain\":     ex[\"domain\"],\n       \"subDomain\":  ex[\"subDomain\"],\n       \"format\":     ex[\"format\"],\n       \"prompt\":     build_prompt(ex),\n       \"gold\":       ex[\"answer\"],\n       \"image_paths\": paths,\n   })\njsonl_path = out_dir \/ \"data.jsonl\"\nwith open(jsonl_path, \"w\") as f:\n   for r in records: f.write(json.dumps(r) + \"n\")\nprint(f\"nWrote {len(records)} records \u2192 {jsonl_path}\")\nprint(f\"Saved {sum(len(r['image_paths']) for r in records)} images under {img_dir}\")\ndef mock_policy_samples(gold, K=4):\n   \"\"\"Stand-in for K policy rollouts. Replace with model.generate(do_sample=True).\"\"\"\n   return [gold,\n           \"Final answer: 0\",\n           f\"Final answer: {gold} (\u2248)\",\n           \"I think the answer is unclear.\"][:K]\ndef grpo_advantages(rewards):\n   r = np.asarray(rewards, dtype=float)\n   return (r - r.mean()) \/ (r.std() + 1e-6)\nprint(\"n=== Mock GRPO rollouts for example 0 ===\")\ngold0 = ds[0][\"answer\"]\ncands = mock_policy_samples(gold0, K=4)\nrewards = [grade(c, gold0) for c in cands]\nadv = grpo_advantages(rewards)\nfor c, r, a in zip(cands, rewards, adv):\n   print(f\"  r={r:.2f}  adv={a:+.2f}   cand={c!r}\")\nprint(\"nDone. To turn this into real training:\")\nprint(\"  1. Replace mock_policy_samples with vlm_solve(..., do_sample=True, num_return_sequences=K).\")\nprint(\"  2. Feed (prompt, K rollouts, K rewards) into TRL's GRPOTrainer or verl.\")\nprint(\"  3. Curriculum: start with examples where rewards have non-zero variance.\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We check whether CUDA is available and, optionally, run SmolVLM on a few examples to generate predictions, then score them using our reward function. We then export the dataset to a GRPO-style JSONL format, saving all images to disk for future multimodal RL experiments. Finally, we demonstrate mock GRPO rollouts, calculate group-relative advantages, and outline how this can be replaced with real model-generated samples.<\/p>\n
In conclusion, we built a complete workflow for understanding, evaluating, and preparing the Open-MM-RL dataset for multimodal reasoning experiments. We moved from dataset loading and exploratory analysis to image inspection, LaTeX-aware answer classification, reward scoring, prompt construction, optional VLM inference, and GRPO-style rollout preparation. It provides a strong starting point for training and evaluating vision-language models with verifiable rewards, while also helping us understand how to transform multimodal datasets into practical reinforcement learning pipelines.<\/p>\n
\n
\n<\/p>
\n<\/p>
Check out\u00a0the\u00a0Full Codes with Notebook here<\/a>.\u00a0<\/strong>Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0150k+ ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.?\u00a0Connect with us<\/mark><\/a><\/strong><\/p>\n
The post Design a Complete Multimodal RLVR Pipeline with Open-MM-RL, Vision-Language Prompting, Reward Scoring, and GRPO Export<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
In this tutorial, we explore t…<\/p>\n","protected":false},"author":1,"featured_media":29,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-976","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/976","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=976"}],"version-history":[{"count":0,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/976\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/media\/29"}],"wp:attachment":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=976"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=976"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=976"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}