{"id":846,"date":"2026-05-04T05:26:42","date_gmt":"2026-05-03T21:26:42","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=846"},"modified":"2026-05-04T05:26:42","modified_gmt":"2026-05-03T21:26:42","slug":"a-coding-implementation-to-explore-and-analyze-the-tasktrove-dataset-with-streaming-parsing-visualization-and-verifier-detection","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=846","title":{"rendered":"A Coding Implementation to Explore and Analyze the TaskTrove Dataset with Streaming Parsing Visualization and Verifier Detection"},"content":{"rendered":"

In this tutorial, we take a deep dive into the TaskTrove<\/strong><\/a> dataset on Hugging Face and build a complete, practical workflow to efficiently explore it. Instead of downloading the full multi-gigabyte dataset, we stream it directly and work with individual samples in real time. We begin by setting up the environment and inspecting the raw structure of the dataset, focusing on how each task is stored as a compressed binary blob. We then implement robust parsing logic to decode these binaries into meaningful formats such as tar archives, zip files, JSON, or plain text. Along the way, we analyze file structures, inspect metadata, and build utilities to better understand the contents of each task.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import subprocess, sys\nsubprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-U\",\n                      \"datasets\", \"huggingface_hub\", \"polars\", \"pandas\",\n                      \"matplotlib\", \"seaborn\", \"tqdm\", \"pyarrow\"])\n\n\nimport os, io, gzip, json, tarfile, zipfile, base64, re, warnings\nfrom pathlib import Path\nfrom collections import Counter, defaultdict\nfrom typing import Any, Dict, Iterator, List, Optional, Union\n\n\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm.auto import tqdm\nfrom datasets import load_dataset\nfrom huggingface_hub import HfApi\n\n\nwarnings.filterwarnings(\"ignore\")\nplt.rcParams[\"figure.dpi\"] = 110\nsns.set_style(\"whitegrid\")\nsns.set_palette(\"mako_r\")\n\n\nDATASET_ID = \"open-thoughts\/TaskTrove\"\nprint(\"\u2713 environment ready\")\n\n\nds_test       = load_dataset(DATASET_ID, split=\"test\",       streaming=True)\nds_validation = load_dataset(DATASET_ID, split=\"validation\", streaming=True)\n\n\nfirst = next(iter(ds_test))\nprint(\"Keys              :\", list(first.keys()))\nprint(\"path              :\", first[\"path\"])\nprint(\"task_binary type  :\", type(first[\"task_binary\"]).__name__)\nprint(\"task_binary length:\", len(first[\"task_binary\"]), \"bytes\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We set up the entire environment by installing all required libraries and importing the necessary modules. We configure visualization settings and initialize the dataset streaming pipeline to reduce download sizes. We also inspect the first sample to understand the dataset\u2019s structure and key fields.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def to_bytes(blob) -> bytes:\n   \"\"\"Coerce whatever `datasets` gives us into raw bytes.\"\"\"\n   if isinstance(blob, (bytes, bytearray)):\n       return bytes(blob)\n   if isinstance(blob, list):\n       return bytes(blob)\n   if isinstance(blob, str):\n       try:\n           return base64.b64decode(blob)\n       except Exception:\n           return blob.encode(\"utf-8\", errors=\"replace\")\n   return bytes(blob)\n\n\n\n\ndef parse_task(blob) -> Dict[str, Any]:\n   \"\"\"gunzip + auto-detect tar \/ zip \/ json \/ jsonl \/ text \/ binary.\"\"\"\n   raw = to_bytes(blob)\n   compressed_size = len(raw)\n   data = gzip.decompress(raw) if raw[:2] == b\"x1fx8b\" else raw\n   raw_size = len(data)\n   bio = io.BytesIO(data)\n\n\n   try:\n       with tarfile.open(fileobj=bio) as tar:\n           files: Dict[str, Union[str, bytes]] = {}\n           for m in tar.getmembers():\n               if not m.isfile():\n                   continue\n               f = tar.extractfile(m)\n               if f is None:\n                   continue\n               content = f.read()\n               try:\n                   files[m.name] = content.decode(\"utf-8\")\n               except UnicodeDecodeError:\n                   files[m.name] = content\n           if files:\n               return {\"format\": \"tar\", \"files\": files,\n                       \"raw_size\": raw_size, \"compressed_size\": compressed_size}\n   except tarfile.TarError:\n       pass\n\n\n   bio.seek(0)\n   try:\n       with zipfile.ZipFile(bio) as zf:\n           files = {}\n           for name in zf.namelist():\n               if name.endswith(\"\/\"):\n                   continue\n               with zf.open(name) as zh:\n                   content = zh.read()\n                   try:\n                       files[name] = content.decode(\"utf-8\")\n                   except UnicodeDecodeError:\n                       files[name] = content\n           return {\"format\": \"zip\", \"files\": files,\n                   \"raw_size\": raw_size, \"compressed_size\": compressed_size}\n   except zipfile.BadZipFile:\n       pass\n\n\n   try:\n       text = data.decode(\"utf-8\")\n       try:\n           return {\"format\": \"json\", \"content\": json.loads(text),\n                   \"raw_size\": raw_size, \"compressed_size\": compressed_size}\n       except json.JSONDecodeError:\n           try:\n               items = [json.loads(l) for l in text.splitlines() if l.strip()]\n               return {\"format\": \"jsonl\", \"content\": items,\n                       \"raw_size\": raw_size, \"compressed_size\": compressed_size}\n           except json.JSONDecodeError:\n               return {\"format\": \"text\", \"content\": text,\n                       \"raw_size\": raw_size, \"compressed_size\": compressed_size}\n   except UnicodeDecodeError:\n       return {\"format\": \"binary\", \"content\": data,\n               \"raw_size\": raw_size, \"compressed_size\": compressed_size}\n\n\n\n\nraw = to_bytes(first[\"task_binary\"])\nprint(\"First 16 bytes (hex):\", raw[:16].hex(\" \"))\ntask = parse_task(first[\"task_binary\"])\nprint(f\"Format            : {task['format']}\")\nprint(f\"Compressed size   : {task['compressed_size']:>10,} bytes\")\nprint(f\"Decompressed size : {task['raw_size']:>10,} bytes\")\nif task[\"format\"] in (\"tar\", \"zip\"):\n   print(f\"Members           : {len(task['files'])}\")\n   for name in list(task[\"files\"])[:10]:\n       body = task[\"files\"][name]\n       size = len(body) if isinstance(body, (str, bytes)) else 0\n       print(f\"  {name:<60} {size:>8} bytes\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We build robust utilities to convert raw task binaries into usable byte formats and parse them intelligently. We handle multiple formats like tar, zip, JSON, JSONL, and plain text using a unified parsing function. We then decode and inspect a sample task to understand its structure and size characteristics.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def show_task(task: Dict[str, Any], json_chars: int = 1500, code_chars: int = 600) -> None:\n   print(\"\u2550\" * 70)\n   print(f\"FORMAT: {task['format']}   |   compressed {task['compressed_size']:,} \u2192 \"\n         f\"raw {task['raw_size']:,} bytes\")\n   print(\"\u2550\" * 70)\n   if task[\"format\"] not in (\"tar\", \"zip\"):\n       print(task.get(\"content\", \"<binary>\"))\n       return\n   files = task[\"files\"]\n   by_ext: Dict[str, List[str]] = defaultdict(list)\n   for name in files:\n       by_ext[Path(name).suffix.lower() or \"<no-ext>\"].append(name)\n   print(\"nFile-type breakdown:\")\n   for ext, names in sorted(by_ext.items(), key=lambda x: -len(x[1])):\n       print(f\"  {ext:<10} {len(names):>4} file(s)\")\n\n\n   meta = [n for n in files if n.lower().endswith((\".json\", \".yaml\", \".yml\", \".toml\"))]\n   code = [n for n in files if n.endswith(\".py\")]\n   for name in meta[:3]:\n       print(f\"n--- {name} ---\")\n       body = files[name]\n       if isinstance(body, str):\n           try:\n               pretty = json.dumps(json.loads(body), indent=2)[:json_chars]\n           except json.JSONDecodeError:\n               pretty = body[:json_chars]\n           print(pretty)\n           if len(body) > json_chars:\n               print(f\"\u2026 ({len(body)-json_chars:,} more chars)\")\n   for name in code[:2]:\n       print(f\"n--- {name} ---\")\n       body = files[name]\n       if isinstance(body, str):\n           print(body[:code_chars])\n           if len(body) > code_chars:\n               print(f\"\u2026 ({len(body)-code_chars:,} more chars)\")\n\n\n\n\nshow_task(task)\n\n\n\n\ndef source_of(path: str) -> str:\n   return path.rsplit(\"-\", 1)[0] if \"-\" in path else path\n\n\n\n\nsource_counts: Counter = Counter()\ncompressed_sizes: List[int] = []\nfor row in tqdm(ds_test, desc=\"counting paths\"):\n   source_counts[source_of(row[\"path\"])] += 1\n   compressed_sizes.append(len(row[\"task_binary\"]))\n\n\nprint(f\"nUnique source prefixes: {len(source_counts)}\")\nprint(\"Top 15 sources:\")\nfor src, n in source_counts.most_common(15):\n   print(f\"  {n:>6}  {src}\")\n\n\nfig, axes = plt.subplots(1, 2, figsize=(14, 6))\nTOP_N = 15\ntop = source_counts.most_common(TOP_N)\nlabels = [s for s, _ in top]\nvalues = [n for _, n in top]\naxes[0].barh(range(len(labels)), values, color=sns.color_palette(\"mako_r\", len(labels)))\naxes[0].set_yticks(range(len(labels)))\naxes[0].set_yticklabels(labels, fontsize=9)\naxes[0].invert_yaxis()\naxes[0].set_xlabel(\"number of tasks\")\naxes[0].set_title(f\"Top {TOP_N} sources in test split\", fontweight=\"bold\")\nfor i, v in enumerate(values):\n   axes[0].text(v, i, f\" {v:,}\", va=\"center\", fontsize=8)\n\n\naxes[1].hist(np.array(compressed_sizes) \/ 1024, bins=50,\n            color=sns.color_palette(\"mako_r\")[2], edgecolor=\"white\")\naxes[1].set_xscale(\"log\")\naxes[1].set_xlabel(\"compressed size (KB, log scale)\")\naxes[1].set_ylabel(\"# tasks\")\naxes[1].set_title(\"Distribution of compressed task sizes\", fontweight=\"bold\")\np50 = np.median(compressed_sizes) \/ 1024\np95 = np.percentile(compressed_sizes, 95) \/ 1024\naxes[1].axvline(p50, color=\"crimson\", linestyle=\"--\", alpha=0.7, label=f\"median = {p50:.1f} KB\")\naxes[1].axvline(p95, color=\"orange\",  linestyle=\"--\", alpha=0.7, label=f\"p95    = {p95:.1f} KB\")\naxes[1].legend()\nplt.tight_layout()\nplt.show()<\/code><\/pre>\n<\/div>\n<\/div>\n
We create a detailed visualization of each task by printing structured file breakdowns and previews. We analyze the dataset distribution by counting source prefixes and measuring compressed task sizes. We also generate plots to better understand the dataset composition and size distribution.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
filename_counter: Counter = Counter()\nall_json_keys:    Counter = Counter()\nsamples_for_show: List = []\n\n\nfor i, row in enumerate(tqdm(ds_test, desc=\"inspecting structure\", total=200)):\n   if i >= 200:\n       break\n   p = parse_task(row[\"task_binary\"])\n   if p[\"format\"] in (\"tar\", \"zip\"):\n       for name, body in p[\"files\"].items():\n           filename_counter[name] += 1\n           if name.endswith(\".json\") and isinstance(body, str):\n               try:\n                   obj = json.loads(body)\n                   if isinstance(obj, dict):\n                       for k in obj.keys():\n                           all_json_keys[k] += 1\n               except Exception:\n                   pass\n       if len(samples_for_show) < 2:\n           samples_for_show.append((row[\"path\"], p))\n\n\nprint(\"nMost common filenames inside task archives:\")\nfor name, n in filename_counter.most_common(15):\n   print(f\"  {n:>4}  {name}\")\n\n\nprint(\"nMost common top-level JSON keys (across any *.json):\")\nfor k, n in all_json_keys.most_common(20):\n   print(f\"  {n:>4}  {k}\")\n\n\nif samples_for_show:\n   print(f\"nFull file listing for one sample task ({samples_for_show[0][0]}):\")\n   for name, body in samples_for_show[0][1][\"files\"].items():\n       sz = len(body) if isinstance(body, (str, bytes)) else 0\n       print(f\"  {name}  ({sz:,} B)\")\n\n\n\n\nVERIFIER_FILE_PATTERNS = (\"verifier\", \"verify\", \"grader\", \"judge\", \"score\", \"eval\")\nVERIFIER_JSON_KEYS     = (\"verifier\", \"verifier_config\", \"judge\", \"grader\",\n                         \"rubric\", \"test_patch\", \"FAIL_TO_PASS\", \"tests\")\n\n\n\n\ndef has_verifier(parsed: Dict[str, Any]) -> bool:\n   \"\"\"Detect verifiers via filename, JSON content, or both.\"\"\"\n   if parsed[\"format\"] not in (\"tar\", \"zip\"):\n       c = parsed.get(\"content\")\n       if isinstance(c, dict):\n           return any(k in c for k in VERIFIER_JSON_KEYS)\n       return False\n\n\n   files = parsed[\"files\"]\n\n\n   for name in files:\n       low = name.lower()\n       if any(pat in low for pat in VERIFIER_FILE_PATTERNS):\n           return True\n\n\n   for name, body in files.items():\n       if name.endswith((\".json\", \".yaml\", \".yml\")) and isinstance(body, str):\n           try:\n               obj = json.loads(body)\n               if isinstance(obj, dict) and any(k in obj for k in VERIFIER_JSON_KEYS):\n                   return True\n           except Exception:\n               pass\n           low = body.lower()\n           if \"verifier\" in low or \"test_patch\" in low:\n               return True\n\n\n   return False\n\n\n\n\nclass TaskTroveExplorer:\n   \"\"\"High-level interface to the open-thoughts\/TaskTrove dataset.\"\"\"\n\n\n   def __init__(self, split: str = \"test\", dataset_id: str = DATASET_ID):\n       self.dataset_id = dataset_id\n       self.split = split\n       self._ds = load_dataset(dataset_id, split=split, streaming=True)\n\n\n   def iter(self, limit: Optional[int] = None,\n            source_filter: Optional[str] = None) -> Iterator[Dict[str, Any]]:\n       rx = re.compile(source_filter) if source_filter else None\n       n = 0\n       for row in self._ds:\n           if rx and not rx.search(source_of(row[\"path\"])):\n               continue\n           yield row\n           n += 1\n           if limit is not None and n >= limit:\n               return\n\n\n   def sample(self, n: int = 5,\n              source_filter: Optional[str] = None) -> List[Dict[str, Any]]:\n       out = []\n       for row in self.iter(limit=n, source_filter=source_filter):\n           parsed = parse_task(row[\"task_binary\"])\n           parsed[\"path\"] = row[\"path\"]\n           parsed[\"source\"] = source_of(row[\"path\"])\n           out.append(parsed)\n       return out\n\n\n   def summary(self, limit: int = 1000,\n               source_filter: Optional[str] = None) -> pd.DataFrame:\n       rows = []\n       for row in self.iter(limit=limit, source_filter=source_filter):\n           parsed = parse_task(row[\"task_binary\"])\n           rows.append({\n               \"source\": source_of(row[\"path\"]),\n               \"compressed\": parsed[\"compressed_size\"],\n               \"raw\": parsed[\"raw_size\"],\n               \"format\": parsed[\"format\"],\n               \"n_files\": len(parsed.get(\"files\", {})),\n               \"has_verifier\": has_verifier(parsed),\n           })\n       df = pd.DataFrame(rows)\n       if df.empty:\n           return df\n       return (df.groupby(\"source\")\n                 .agg(n=(\"compressed\", \"count\"),\n                      mean_compressed_kb=(\"compressed\", lambda s: s.mean()\/1024),\n                      mean_raw_kb=(\"raw\",                lambda s: s.mean()\/1024),\n                      mean_n_files=(\"n_files\", \"mean\"),\n                      verifier_rate=(\"has_verifier\", \"mean\"))\n                 .round(2)\n                 .sort_values(\"n\", ascending=False))\n\n\n   @staticmethod\n   def has_verifier(parsed: Dict[str, Any]) -> bool:\n       return has_verifier(parsed)\n\n\n   def export(self, output_dir: Union[str, Path], n: int = 10,\n              source_filter: Optional[str] = None) -> Path:\n       output_dir = Path(output_dir)\n       output_dir.mkdir(parents=True, exist_ok=True)\n       for parsed in self.sample(n=n, source_filter=source_filter):\n           slug = parsed[\"path\"].replace(\"\/\", \"_\")\n           tdir = output_dir \/ slug\n           tdir.mkdir(exist_ok=True)\n           if parsed[\"format\"] in (\"tar\", \"zip\"):\n               for name, body in parsed[\"files\"].items():\n                   out = tdir \/ name\n                   out.parent.mkdir(parents=True, exist_ok=True)\n                   if isinstance(body, str):\n                       out.write_text(body, encoding=\"utf-8\")\n                   else:\n                       out.write_bytes(body)\n           else:\n               content = parsed.get(\"content\", b\"\")\n               if isinstance(content, (dict, list)):\n                   (tdir \/ \"task.json\").write_text(json.dumps(content, indent=2))\n               elif isinstance(content, str):\n                   (tdir \/ \"task.txt\").write_text(content)\n               else:\n                   (tdir \/ \"task.bin\").write_bytes(content)\n       print(f\"\u2713 exported tasks to {output_dir.resolve()}\")\n       return output_dir\n\n\n\n\nexplorer = TaskTroveExplorer(split=\"test\")\n\n\nprint(\"nSample of 3 parsed tasks:\")\nfor s in explorer.sample(n=3):\n   print(f\"path: {s['path']} | source: {s['source']} | format: {s['format']} | \"\n         f\"files: {len(s.get('files', {}))} | verifier: {has_verifier(s)}\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We deeply inspect the internal structure of tasks by analyzing filenames and extracting common JSON keys. We implement a multi-signal verifier detection system to identify tasks suitable for evaluation or RL workflows. We also build a reusable explorer class that allows us to sample, summarize, and export tasks efficiently.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
summary = explorer.summary(limit=1000)\nprint(f\"nSummary across {len(summary)} sources (1000 sampled rows):\")\nprint(summary.head(20))\n\n\nif not summary.empty:\n   top_sources = summary.head(12)\n   fig, ax = plt.subplots(figsize=(11, 6))\n   x = np.arange(len(top_sources))\n   w = 0.4\n   ax.bar(x - w\/2, top_sources[\"mean_compressed_kb\"], w, label=\"compressed (KB)\",\n          color=sns.color_palette(\"mako_r\")[2])\n   ax.bar(x + w\/2, top_sources[\"mean_raw_kb\"], w, label=\"decompressed (KB)\",\n          color=sns.color_palette(\"mako_r\")[5])\n   ax.set_xticks(x)\n   ax.set_xticklabels(top_sources.index, rotation=40, ha=\"right\", fontsize=9)\n   ax.set_ylabel(\"size (KB)\")\n   ax.set_yscale(\"log\")\n   ax.set_title(\"Mean task size by source (top 12 by row count)\", fontweight=\"bold\")\n   ax.legend()\n   plt.tight_layout()\n   plt.show()\n\n\n   fig, ax = plt.subplots(figsize=(11, 5))\n   vs = summary.head(15)[\"verifier_rate\"].sort_values()\n   colors = sns.color_palette(\"RdYlGn\", as_cmap=True)(vs.values)\n   ax.barh(range(len(vs)), vs.values, color=colors)\n   ax.set_yticks(range(len(vs)))\n   ax.set_yticklabels(vs.index, fontsize=9)\n   ax.set_xlabel(\"fraction of tasks with verifier signal\")\n   ax.set_xlim(0, 1)\n   ax.set_title(\"Verifier presence by sourcen(green = verified \u21d2 usable for RL)\",\n                fontweight=\"bold\")\n   for i, v in enumerate(vs.values):\n       ax.text(min(v + 0.01, 0.97), i, f\"{v:.0%}\", va=\"center\", fontsize=9)\n   plt.tight_layout()\n   plt.show()\n\n\n\n\nverified_task = None\nfor row in tqdm(ds_test, desc=\"hunting for a verified task\"):\n   parsed = parse_task(row[\"task_binary\"])\n   if has_verifier(parsed):\n       parsed[\"path\"] = row[\"path\"]\n       parsed[\"source\"] = source_of(row[\"path\"])\n       verified_task = parsed\n       break\n\n\nif verified_task is None:\n   print(\"No verified task found in test split \u2014 try the validation split.\")\nelse:\n   print(f\"Found verified task: {verified_task['path']}\")\n   print(f\"Source             : {verified_task['source']}\")\n   if verified_task[\"format\"] in (\"tar\", \"zip\"):\n       candidates = []\n       for n in verified_task[\"files\"]:\n           low = n.lower()\n           score = sum(p in low for p in VERIFIER_FILE_PATTERNS)\n           if n.endswith((\".json\", \".yaml\", \".yml\", \".py\")):\n               score += 1\n           candidates.append((score, n))\n       candidates.sort(reverse=True)\n       for _, name in candidates[:2]:\n           body = verified_task[\"files\"][name]\n           if isinstance(body, str):\n               print(f\"n--- {name} ({len(body):,} chars) ---\")\n               print(body[:2000])\n               if len(body) > 2000:\n                   print(f\"\u2026 ({len(body)-2000:,} more chars)\")\n\n\n\n\nEXPORT_DIR = Path(\"\/content\/tasktrove_export\") if Path(\"\/content\").exists() \n            else Path(\".\/tasktrove_export\")\nEXPORT_DIR.mkdir(exist_ok=True)\nexplorer.export(EXPORT_DIR, n=5)\n\n\nfor task_dir in sorted(EXPORT_DIR.iterdir())[:3]:\n   print(\"\u2500\" * 60)\n   print(task_dir.name)\n   for sub in sorted(task_dir.rglob(\"*\"))[:8]:\n       if sub.is_file():\n           print(f\"  {sub.relative_to(task_dir)}  ({sub.stat().st_size:,} B)\")\n\n\n\n\nrows: List[Dict[str, Any]] = []\nMAX_TASKS = 500\nn_seen = 0\n\n\nfor row in tqdm(ds_test, desc=\"building slice\", total=MAX_TASKS):\n   parsed = parse_task(row[\"task_binary\"])\n   n_seen += 1\n   src = source_of(row[\"path\"])\n   is_verified = has_verifier(parsed) or \"verifier\" in src.lower()\n\n\n   files = parsed.get(\"files\", {})\n   instruction = \"\"\n   for name in files:\n       if name.endswith((\".json\", \".md\", \".txt\")) and isinstance(files[name], str):\n           if len(files[name]) > len(instruction):\n               instruction = files[name]\n\n\n   rows.append({\n       \"path\": row[\"path\"],\n       \"source\": src,\n       \"is_verified\": bool(is_verified),\n       \"n_files\": len(files),\n       \"compressed_kb\": parsed[\"compressed_size\"] \/ 1024,\n       \"raw_kb\": parsed[\"raw_size\"] \/ 1024,\n       \"instruction_preview\": instruction[:300],\n   })\n   if len(rows) >= MAX_TASKS:\n       break\n\n\ndf = pl.DataFrame(rows)\nprint(f\"nInspected {n_seen} rows, kept {len(df)} total \"\n     f\"({df['is_verified'].sum() if len(df) else 0} flagged verified)\")\nif len(df):\n   print(df.head(5))\n\n\nif len(df) == 0:\n   print(\"Empty slice \u2014 nothing to aggregate or save.\")\nelse:\n   grouped = (df.group_by(\"source\")\n                .agg([pl.len().alias(\"n\"),\n                      pl.col(\"is_verified\").sum().alias(\"n_verified\"),\n                      pl.col(\"raw_kb\").mean().round(1).alias(\"mean_raw_kb\"),\n                      pl.col(\"n_files\").mean().round(1).alias(\"mean_n_files\")])\n                .sort(\"n\", descending=True))\n   print(\"nSlice composition by source:\")\n   print(grouped)\n\n\n   out_path = (Path(\"\/content\") if Path(\"\/content\").exists() else Path(\".\")) \n              \/ \"tasktrove_slice.parquet\"\n   df.write_parquet(out_path)\n   print(f\"n\u2713 wrote {len(df)} rows to {out_path} \"\n         f\"({out_path.stat().st_size\/1024:.1f} KB)\")\n\n\n\n\napi = HfApi()\nfiles = api.list_repo_files(repo_id=DATASET_ID, repo_type=\"dataset\")\nsubdirs = sorted({f.split(\"\/\", 1)[0] for f in files\n                 if \"\/\" in f and \"__\" in f.split(\"\/\", 1)[0]})\nprint(f\"nFound {len(subdirs)} source-dataset subdirectories. First 25:\")\nfor s in subdirs[:25]:\n   print(\" \", s)<\/code><\/pre>\n<\/div>\n<\/div>\n
We aggregate statistics across sources and visualize key metrics, such as task size and verifier presence. We identify and inspect a verified task to understand how evaluation signals are structured. Finally, we build a clean dataset slice, export it, and prepare it for downstream analysis or modeling workflows.<\/p>\n
In conclusion, we constructed a comprehensive pipeline to explore, analyze, and extract value from the TaskTrove dataset. We generated insights into source distributions, task sizes, and internal file patterns, and built mechanisms to detect verifier signals indicating high-quality, evaluation-ready tasks. We also created reusable tools, such as the TaskTroveExplorer class, to sample, summarize, and export tasks for downstream use. Also, we produced a clean, structured dataset slice that can be directly used for research, benchmarking, or reinforcement learning workflows. Through this process, we learn how to handle complex dataset formats efficiently and also establish a scalable approach to working with large, structured AI datasets in real-world scenarios.<\/p>\n
\n
Check out\u00a0the\u00a0Full Codes with Notebook here<\/a><\/strong>.\u00a0<\/strong>Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0130k+ ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.?\u00a0Connect with us<\/mark><\/a><\/strong><\/p>\n
The post A Coding Implementation to Explore and Analyze the TaskTrove Dataset with Streaming Parsing Visualization and Verifier Detection<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
In this tutorial, we take a de…<\/p>\n","protected":false},"author":1,"featured_media":29,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-846","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/846","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=846"}],"version-history":[{"count":0,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/846\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/media\/29"}],"wp:attachment":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=846"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=846"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=846"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}