{"id":1027,"date":"2026-06-03T01:57:41","date_gmt":"2026-06-02T17:57:41","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=1027"},"modified":"2026-06-03T01:57:41","modified_gmt":"2026-06-02T17:57:41","slug":"tinyfish-launches-bigset-an-open-source-multi-agent-system-that-builds-structured-live-datasets-from-plain-english-descriptions","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=1027","title":{"rendered":"TinyFish Launches BigSet: An Open-Source Multi-Agent System That Builds Structured Live Datasets from Plain-English Descriptions"},"content":{"rendered":"

Building a structured dataset from the web is still a pipeline problem. You identify a data source, write or configure a scraper, design a schema, handle deduplication, schedule refreshes, and fix breakage when upstream sites change. That process stays roughly the same whether you do it once or a hundred times.<\/p>\n

TinyFish is releasing BigSet <\/a><\/strong>to address that workflow directly. Bigset is an open-source multi-agent system licensed under AGPL-3.0. It takes a natural-language description as input and returns a structured, exportable dataset built from live web data. The full codebase is available on GitHub.<\/p>\n

What is BigSet<\/strong><\/h1>\n

Bigset<\/a><\/strong> positions itself as the layer between a data requirement and a usable table. You describe what you want in a sentence. The system infers the schema, dispatches agents to gather data, deduplicates results, and produces a downloadable CSV or XLSX file.<\/p>\n

A practical example: you type \u201cYC companies that are currently hiring engineers, with their funding stage, location, and number of open roles.\u201d<\/em> Bigset infers what columns that implies, finds the relevant entities on the web, and fills in the rows. You don\u2019t specify a URL. You don\u2019t configure selectors. You describe the data.<\/p>\n

A scheduled refresh feature lets datasets update automatically. You set a cadence \u2014 30 minutes, 6 hours, 12 hours, daily, weekly \u2014 and the agents re-run on that schedule. The table stays current without re-running the task manually.<\/p>\n

One practical note: dataset generation takes 2\u20135 minutes. The agents are doing real web research \u2014 searching, fetching pages, and verifying data. It is not an instant result.<\/p>\n

How the Multi-Agent Architecture Works<\/strong><\/h1>\n

The architecture here is worth understanding concretely. BigSet<\/a><\/strong> is not a single LLM call with a web search tool attached. It runs a structured two-tier agent system.<\/p>\n

Step 1 \u2014 Schema Inference:\u00a0 <\/strong>When you submit a description, Claude Sonnet (accessed via OpenRouter) infers the dataset schema. This includes column names, data types, primary keys, and where to look for the data. This happens before any web access. The default is anthropic\/claude-sonnet-4.6, but it is set by the SCHEMA_INFERENCE_MODEL env var and can be pointed at any OpenRouter model slug.<\/p>\n

Step 2 \u2014 Orchestrator Agent:\u00a0 <\/strong>A separate orchestrator agent runs broad discovery using TinyFish Search. It identifies which entities match your description and where to find them. The model defaults to Qwen (qwen\/qwen3.7-max, via OpenRouter), configurable through POPULATE_ORCHESTRATOR_MODEL.<\/p>\n

Step 3 \u2014 Sub-Agent Fan-Out:\u00a0 <\/strong>The orchestrator dispatches sub-agents in parallel. Each sub-agent handles exactly one entity \u2014 one row in the final table. Each agent has a tool budget capped at 6 calls. It uses TinyFish Fetch to retrieve real page content, extracts the relevant fields, and inserts a row.<\/p>\n

Step 4 \u2014 Deduplication and Source Attribution:\u00a0 <\/strong>The system applies primary key deduplication. Each row carries source attribution \u2014 a traceable link to the web page the data came from. Quota enforcement per user is also applied at this stage.<\/p>\n

Step 5 \u2014 Export:\u00a0 <\/strong>The final result is a structured table available as CSV or XLSX download.<\/p>\n

Tech Stack<\/strong><\/h1>\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Layer<\/strong><\/td>\nTechnology<\/strong><\/td>\n<\/tr>\n
Frontend<\/td>\nNext.js 16, React 19, Tailwind 4<\/td>\n<\/tr>\n
Backend<\/td>\nFastify, TypeScript<\/td>\n<\/tr>\n
Auth<\/td>\nClerk<\/td>\n<\/tr>\n
Database<\/td>\nConvex (self-hosted)<\/td>\n<\/tr>\n
AI Orchestration<\/td>\nMastra workflows + Vercel AI SDK + OpenRouter<\/td>\n<\/tr>\n
LLM \u2014 Schema Inference<\/td>\nClaude Sonnet via OpenRouter<\/td>\n<\/tr>\n
LLM \u2014 Orchestrator Agent<\/td>\nQwen via OpenRouter<\/td>\n<\/tr>\n
Data Collection<\/td>\nTinyFish Search, TinyFish Fetch, TinyFish Browser<\/td>\n<\/tr>\n
Table View<\/td>\nTanStack Table + react-window virtualization<\/td>\n<\/tr>\n
Exports<\/td>\nCSV (built-in) + XLSX via SheetJS<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

How to Set It Up and Use It<\/strong><\/h1>\n

Bigset <\/a>is self-hosted. You run it on your own infrastructure using Docker. Below is a complete walkthrough from clone to first dataset.<\/p>\n