{"id":312,"date":"2026-01-24T05:30:26","date_gmt":"2026-01-23T21:30:26","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=312"},"modified":"2026-01-24T05:30:26","modified_gmt":"2026-01-23T21:30:26","slug":"how-an-ai-agent-chooses-what-to-do-under-tokens-latency-and-tool-call-budget-constraints","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=312","title":{"rendered":"How an AI Agent Chooses What to Do Under Tokens, Latency, and Tool-Call Budget Constraints?"},"content":{"rendered":"

In this tutorial, we build a cost-aware planning agent that deliberately balances output quality against real-world constraints such as token usage, latency, and tool-call budgets. We design the agent to generate multiple candidate actions, estimate their expected costs and benefits, and then select an execution plan that maximizes value while staying within strict budgets. With this, we demonstrate how agentic systems can move beyond \u201calways use the LLM\u201d behavior and instead reason explicitly about trade-offs, efficiency, and resource awareness, which is critical for deploying agents reliably in constrained environments. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
import os, time, math, json, random\nfrom dataclasses import dataclass, field\nfrom typing import List, Dict, Optional, Tuple, Any\nfrom getpass import getpass\n\n\nUSE_OPENAI = True\n\n\nif USE_OPENAI:\n   if not os.getenv(\"OPENAI_API_KEY\"):\n       os.environ[\"OPENAI_API_KEY\"] = getpass(\"Enter OPENAI_API_KEY (hidden): \").strip()\n   try:\n       from openai import OpenAI\n       client = OpenAI()\n   except Exception as e:\n       print(\"OpenAI SDK import failed. Falling back to offline mode.nError:\", e)\n       USE_OPENAI = False<\/code><\/pre>\n<\/div>\n<\/div>\n
We set up the execution environment and securely load the OpenAI API key at runtime without hardcoding it. We also initialize the client so the agent gracefully falls back to offline mode if the API is unavailable. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def approx_tokens(text: str) -> int:\n   return max(1, math.ceil(len(text) \/ 4))\n\n\n@dataclass\nclass Budget:\n   max_tokens: int\n   max_latency_ms: int\n   max_tool_calls: int\n\n\n@dataclass\nclass Spend:\n   tokens: int = 0\n   latency_ms: int = 0\n   tool_calls: int = 0\n\n\n   def within(self, b: Budget) -> bool:\n       return (self.tokens <= b.max_tokens and\n               self.latency_ms <= b.max_latency_ms and\n               self.tool_calls <= b.max_tool_calls)\n\n\n   def add(self, other: \"Spend\") -> \"Spend\":\n       return Spend(\n           tokens=self.tokens + other.tokens,\n           latency_ms=self.latency_ms + other.latency_ms,\n           tool_calls=self.tool_calls + other.tool_calls\n       )<\/code><\/pre>\n<\/div>\n<\/div>\n
We define the core budgeting abstractions that enable the agent to reason explicitly about costs. We model token usage, latency, and tool calls as first-class quantities and provide utility methods to accumulate and validate spend. It gives us a clean foundation for enforcing constraints throughout planning and execution. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
@dataclass\nclass StepOption:\n   name: str\n   description: str\n   est_spend: Spend\n   est_value: float\n   executor: str\n   payload: Dict[str, Any] = field(default_factory=dict)\n\n\n@dataclass\nclass PlanCandidate:\n   steps: List[StepOption]\n   spend: Spend\n   value: float\n   rationale: str = \"\"\n\n\ndef llm_text(prompt: str, *, model: str = \"gpt-5\", effort: str = \"low\") -> str:\n   if not USE_OPENAI:\n       return \"\"\n   t0 = time.time()\n   resp = client.responses.create(\n       model=model,\n       reasoning={\"effort\": effort},\n       input=prompt,\n   )\n   _ = (time.time() - t0)\n   return resp.output_text or \"\"<\/code><\/pre>\n<\/div>\n<\/div>\n
We introduce the data structures that represent individual action choices and full plan candidates. We also define a lightweight LLM wrapper that standardizes how text is generated and measured. This separation allows the planner to reason about actions abstractly without being tightly coupled to execution details. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def generate_step_options(task: str) -> List[StepOption]:\n   base = [\n       StepOption(\n           name=\"Clarify deliverables (local)\",\n           description=\"Extract deliverable checklist + acceptance criteria from the task.\",\n           est_spend=Spend(tokens=60, latency_ms=20, tool_calls=0),\n           est_value=6.0,\n           executor=\"local\",\n       ),\n       StepOption(\n           name=\"Outline plan (LLM)\",\n           description=\"Create a structured outline with sections, constraints, and assumptions.\",\n           est_spend=Spend(tokens=600, latency_ms=1200, tool_calls=1),\n           est_value=10.0,\n           executor=\"llm\",\n           payload={\"prompt_kind\":\"outline\"}\n       ),\n       StepOption(\n           name=\"Outline plan (local)\",\n           description=\"Create a rough outline using templates (no LLM).\",\n           est_spend=Spend(tokens=120, latency_ms=40, tool_calls=0),\n           est_value=5.5,\n           executor=\"local\",\n       ),\n       StepOption(\n           name=\"Risk register (LLM)\",\n           description=\"Generate risks, mitigations, owners, and severity.\",\n           est_spend=Spend(tokens=700, latency_ms=1400, tool_calls=1),\n           est_value=9.0,\n           executor=\"llm\",\n           payload={\"prompt_kind\":\"risks\"}\n       ),\n       StepOption(\n           name=\"Risk register (local)\",\n           description=\"Generate a standard risk register from a reusable template.\",\n           est_spend=Spend(tokens=160, latency_ms=60, tool_calls=0),\n           est_value=5.0,\n           executor=\"local\",\n       ),\n       StepOption(\n           name=\"Timeline (LLM)\",\n           description=\"Draft a realistic milestone timeline with dependencies.\",\n           est_spend=Spend(tokens=650, latency_ms=1300, tool_calls=1),\n           est_value=8.5,\n           executor=\"llm\",\n           payload={\"prompt_kind\":\"timeline\"}\n       ),\n       StepOption(\n           name=\"Timeline (local)\",\n           description=\"Draft a simple timeline from a generic milestone template.\",\n           est_spend=Spend(tokens=150, latency_ms=60, tool_calls=0),\n           est_value=4.8,\n           executor=\"local\",\n       ),\n       StepOption(\n           name=\"Quality pass (LLM)\",\n           description=\"Rewrite for clarity, consistency, and formatting.\",\n           est_spend=Spend(tokens=900, latency_ms=1600, tool_calls=1),\n           est_value=8.0,\n           executor=\"llm\",\n           payload={\"prompt_kind\":\"polish\"}\n       ),\n       StepOption(\n           name=\"Quality pass (local)\",\n           description=\"Light formatting + consistency checks without LLM.\",\n           est_spend=Spend(tokens=120, latency_ms=50, tool_calls=0),\n           est_value=3.5,\n           executor=\"local\",\n       ),\n   ]\n\n\n   if USE_OPENAI:\n       meta_prompt = f\"\"\"\nYou are a planning assistant. For the task below, propose 3-5 OPTIONAL extra steps that improve quality,\nlike checks, validations, or stakeholder tailoring. Keep each step short.\n\n\nTASK:\n{task}\n\n\nReturn JSON list with fields: name, description, est_value(1-10).\n\"\"\"\n       txt = llm_text(meta_prompt, model=\"gpt-5\", effort=\"low\")\n       try:\n           items = json.loads(txt.strip())\n           for it in items[:5]:\n               base.append(\n                   StepOption(\n                       name=str(it.get(\"name\",\"Extra step (local)\"))[:60],\n                       description=str(it.get(\"description\",\"\"))[:200],\n                       est_spend=Spend(tokens=120, latency_ms=60, tool_calls=0),\n                       est_value=float(it.get(\"est_value\", 5.0)),\n                       executor=\"local\",\n                   )\n               )\n       except Exception:\n           pass\n\n\n   return base<\/code><\/pre>\n<\/div>\n<\/div>\n
We focus on generating a diverse set of candidate steps, including both LLM-based and local alternatives with different cost\u2013quality trade-offs. We optionally use the model itself to suggest additional low-cost improvements while still controlling their impact on the budget. By doing so, we enrich the action space without losing efficiency. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def plan_under_budget(\n   options: List[StepOption],\n   budget: Budget,\n   *,\n   max_steps: int = 6,\n   beam_width: int = 12,\n   diversity_penalty: float = 0.2\n) -> PlanCandidate:\n   def redundancy_cost(chosen: List[StepOption], new: StepOption) -> float:\n       key_new = new.name.split(\"(\")[0].strip().lower()\n       overlap = 0\n       for s in chosen:\n           key_s = s.name.split(\"(\")[0].strip().lower()\n           if key_s == key_new:\n               overlap += 1\n       return overlap * diversity_penalty\n\n\n   beams: List[PlanCandidate] = [PlanCandidate(steps=[], spend=Spend(), value=0.0, rationale=\"\")]\n\n\n   for _ in range(max_steps):\n       expanded: List[PlanCandidate] = []\n       for cand in beams:\n           for opt in options:\n               if opt in cand.steps:\n                   continue\n               new_spend = cand.spend.add(opt.est_spend)\n               if not new_spend.within(budget):\n                   continue\n               new_value = cand.value + opt.est_value - redundancy_cost(cand.steps, opt)\n               expanded.append(\n                   PlanCandidate(\n                       steps=cand.steps + [opt],\n                       spend=new_spend,\n                       value=new_value,\n                       rationale=cand.rationale\n                   )\n               )\n       if not expanded:\n           break\n       expanded.sort(key=lambda c: c.value, reverse=True)\n       beams = expanded[:beam_width]\n\n\n   best = max(beams, key=lambda c: c.value)\n   return best<\/code><\/pre>\n<\/div>\n<\/div>\n
We implement the budget-constrained planning logic that searches for the highest-value combination of steps under strict limits. We apply a beam-style search with redundancy penalties to avoid wasteful action overlap. This is where the agent truly becomes cost-aware by optimizing value subject to constraints. Check out the\u00a0FULL CODES here<\/a><\/strong>.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def run_local_step(task: str, step: StepOption, working: Dict[str, Any]) -> str:\n   name = step.name.lower()\n   if \"clarify deliverables\" in name:\n       return (\n           \"Deliverables checklist:n\"\n           \"- Executive summaryn- Scope & assumptionsn- Workplan + milestonesn\"\n           \"- Risk register (risk, impact, likelihood, mitigation, owner)n\"\n           \"- Next steps + data neededn\"\n       )\n   if \"outline plan\" in name:\n       return (\n           \"Outline:n1) Context & objectiven2) Scopen3) Approachn4) Timelinen5) Risksn6) Next stepsn\"\n       )\n   if \"risk register\" in name:\n       return (\n           \"Risk register (template):n\"\n           \"1) Data access delays | High | Mitigation: agree data list + ownersn\"\n           \"2) Stakeholder alignment | Med | Mitigation: weekly reviewn\"\n           \"3) Tooling constraints | Med | Mitigation: phased rolloutn\"\n       )\n   if \"timeline\" in name:\n       return (\n           \"Timeline (template):n\"\n           \"Week 1: discovery + requirementsnWeek 2: prototype + feedbackn\"\n           \"Week 3: pilot + metricsnWeek 4: rollout + handovern\"\n       )\n   if \"quality pass\" in name:\n       draft = working.get(\"draft\", \"\")\n       return \"Light quality pass done (headings normalized, bullets aligned).n\" + draft\n   return f\"Completed: {step.name}n\"\n\n\ndef run_llm_step(task: str, step: StepOption, working: Dict[str, Any]) -> str:\n   kind = step.payload.get(\"prompt_kind\", \"generic\")\n   context = working.get(\"draft\", \"\")\n   prompts = {\n       \"outline\": f\"Create a crisp, structured outline for the task below.nTASK:n{task}nReturn a numbered outline.\",\n       \"risks\": f\"Create a risk register for the task below. Include: Risk | Impact | Likelihood | Mitigation | Owner.nTASK:n{task}\",\n       \"timeline\": f\"Create a realistic milestone timeline with dependencies for the task below.nTASK:n{task}\",\n       \"polish\": f\"Rewrite and polish the following draft for clarity and consistency.nDRAFT:n{context}\",\n       \"generic\": f\"Help with this step: {step.description}nTASK:n{task}nCURRENT:n{context}\",\n   }\n   return llm_text(prompts.get(kind, prompts[\"generic\"]), model=\"gpt-5\", effort=\"low\")\n\n\ndef execute_plan(task: str, plan: PlanCandidate) -> Tuple[str, Spend]:\n   working = {\"draft\": \"\"}\n   actual = Spend()\n\n\n   for i, step in enumerate(plan.steps, 1):\n       t0 = time.time()\n       if step.executor == \"llm\" and USE_OPENAI:\n           out = run_llm_step(task, step, working)\n           tool_calls = 1\n       else:\n           out = run_local_step(task, step, working)\n           tool_calls = 0\n\n\n       dt_ms = int((time.time() - t0) * 1000)\n       tok = approx_tokens(out)\n\n\n       actual = actual.add(Spend(tokens=tok, latency_ms=dt_ms, tool_calls=tool_calls))\n       working[\"draft\"] += f\"nn### Step {i}: {step.name}n{out}n\"\n\n\n   return working[\"draft\"].strip(), actual\n\n\nTASK = \"Draft a 1-page project proposal for a logistics dashboard + fleet optimization pilot, including scope, timeline, and risks.\"\nBUDGET = Budget(\n   max_tokens=2200,\n   max_latency_ms=3500,\n   max_tool_calls=2\n)\n\n\noptions = generate_step_options(TASK)\nbest_plan = plan_under_budget(options, BUDGET, max_steps=6, beam_width=14)\n\n\nprint(\"=== SELECTED PLAN (budget-aware) ===\")\nfor s in best_plan.steps:\n   print(f\"- {s.name} | est_spend={s.est_spend} | est_value={s.est_value}\")\nprint(\"nEstimated spend:\", best_plan.spend)\nprint(\"Budget:\", BUDGET)\n\n\nprint(\"n=== EXECUTING PLAN ===\")\ndraft, actual = execute_plan(TASK, best_plan)\n\n\nprint(\"n=== OUTPUT DRAFT ===n\")\nprint(draft[:6000])\n\n\nprint(\"n=== ACTUAL SPEND (approx) ===\")\nprint(actual)\nprint(\"nWithin budget?\", actual.within(BUDGET))<\/code><\/pre>\n<\/div>\n<\/div>\n
We execute the selected plan and track actual resource usage step by step. We dynamically choose between local and LLM execution paths and aggregate the final output into a coherent draft. By comparing estimated and actual spend, we demonstrate how planning assumptions can be validated and refined in practice.<\/p>\n
In conclusion, we demonstrated how a cost-aware planning agent can reason about its resource consumption and adapt its behavior in real time. We executed only the steps that fit within predefined budgets and tracked actual spend to validate the planning assumptions, closing the loop between estimation and execution. Also, we highlighted how agentic AI systems can become more practical, controllable, and scalable by treating cost, latency, and tool usage as first-class decision variables rather than afterthoughts.<\/p>\n
\n
Check out the\u00a0FULL CODES here<\/a><\/strong>.\u00a0Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0100k+ ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
The post How an AI Agent Chooses What to Do Under Tokens, Latency, and Tool-Call Budget Constraints?<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
In this tutorial, we build a c…<\/p>\n","protected":false},"author":1,"featured_media":29,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-312","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/312","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=312"}],"version-history":[{"count":0,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/312\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/media\/29"}],"wp:attachment":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=312"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=312"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=312"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}