The first problem I wanted to solve was narrow: let an agent run product research on its own.

It needed to search keywords, read competitors' pricing pages, find candid discussions on Reddit, pull App Store reviews, watch what competitors had changed, and turn the evidence into a report I could use to make a decision.

I assumed one search API and a general-purpose crawler would be enough.

The split happened almost immediately. Google results include People Also Ask and Related Searches, while semantic search surfaces a different set of pages. A plain web page can be reduced to its main text, but Reddit and LinkedIn depend on a logged-in session. App Store reviews have their own data model. Some sites require JavaScript. A competitor's pricing page needs a recurring diff, not a one-time read.

"Web access" turned into search, reading, browsing, authentication, extraction, monitoring, and verification. Each one fails differently.

I ended up with two layers. The first gets the data. The second decides which credentials an agent may use, where it may run, and whether it has permission to build, sign, or release anything.

## What you can take from this article

This article includes five reusable pieces:

- A routing diagram for choosing a web-access path by task type.
- A decision order for official APIs, open-source tools, browser sessions, and managed services.
- A control-plane design that separates credential metadata, secret values, and permission to use them.
- A product-and-environment isolation matrix that keeps test jobs away from production credentials.
- An end-to-end flow from task intent and evidence collection to decisions and audit records.

If you maintain one product, one environment, and a few low-risk API keys, `.env` may still be enough. Once local agents, GitHub Actions, production services, and signing workflows start sharing credentials, these diagrams are a practical place to begin splitting permissions.

## One agent needs several kinds of web access

I no longer treat "can open a web page" as a complete capability. A useful research workflow has to handle several distinct jobs.

![Figure 1: The UpWeb Web Access Router. Start with the task intent, then choose the narrowest verifiable data path.](https://img.upweb.dev/media/b197278e-2400x1350-UpWeb-Web-Access-Router-Diagram.png)

### Search and discovery

Standard search is good at finding explicit matches. It can also return People Also Ask, Related Searches, video results, and spelling corrections. That information matters for brand checks, keyword research, and demand scans.

Semantic search solves a different problem. It finds pages that are conceptually close even when they use different language, and it is useful for adding candidate sources during a research session.

I currently use Serper for standard search and Exa for semantic follow-up. They produce different evidence, so I don't let either one replace the other.

### Plain web content

Competitor homepages, pricing pages, help docs, and blogs usually don't need a full browser. A lightweight reader such as Jina Reader can return clean text quickly, with lower cost and fewer moving parts.

That path works well for one-off reading. It stops being enough when a page relies on client-side rendering, interaction state, or a login wall.

### Platforms that require a logged-in session

Reddit, X, LinkedIn, and Instagram often need an existing account and its cookies. Maintaining a separate login script for every platform increases account risk and breaks whenever the page changes.

After connecting the open-source Agent Reach and OpenCLI projects, my research agents can reuse the current Chrome session for read-only searches, page navigation, and extraction. Cookies stay in the browser and local configuration. They don't enter the repository or the research report.

Agent Reach acts as a router here. It brings page reading, semantic search, community platforms, video transcripts, and developer data sources behind one operational entry point. It does not replace Serper, SEO data services, or store-specific scrapers.

### Store and platform data

The App Store and Google Play make a useful counterexample. A general crawler can open a store page, but it may not reliably return review text, install counts, ratings, or in-app purchase data.

For these jobs, I start with the iTunes Search API, Apple RSS, the open-source `google-play-scraper`, and other platform-specific sources. Their schemas are clear, their cost is low, and changes to the page layout are less likely to break them.

I only reach for a managed collector such as an Apify Actor when I need more volume or fields the free sources don't provide. Every paid run starts with a small sample so I can verify both the schema and the cost.

### Continuous monitoring

Reading a page once and monitoring it over time are separate systems. Pricing, features, terms, and landing pages keep changing. Monitoring means storing snapshots, diffing them, filtering noise, and triggering follow-up work only when a change crosses a threshold.

I'm still evaluating a more stable free foundation built from local fetching, `diff`, and `watch`. The rule for now is simple: design one-time evidence collection and continuous monitoring separately. A repeated read is not automatically a monitoring system.

## The open-source tools I compared

I didn't try to build a crawler leaderboard. I picked a few tools that represent different layers.

- Firecrawl and Crawl4AI both turn websites into material an AI system can use. Firecrawl is closer to a complete web-data service, while Crawl4AI is a good fit for a self-hosted, LLM-friendly crawler.
- browser-use handles clicks, logins, and form interactions in a real browser.
- Crawlee is closer to collection infrastructure, with queues, retries, proxy rotation, and concurrency controls.
- MarkItDown doesn't crawl. It converts PDFs, Office files, and other documents into Markdown, covering the non-web part of the pipeline.

No one tool covers every job. I first decide whether I need page reading, browser interaction, collection infrastructure, or document conversion. Then I compare the maintenance cost of open source with the price of a managed service.

## How I choose between open source and paid services

"Open source or paid" wasn't specific enough to guide a build. I now start by identifying the interface the task needs and the maintenance work I'm willing to own.

My order looks roughly like this:

1. Use the official API when the data source provides one.
2. If there is no official API, look for a platform-specific open-source tool.
3. Use a lightweight reader for public page content.
4. Use a real browser session when the page depends on JavaScript, interaction, or authentication.
5. Consider a managed web API when the job needs stable structure, concurrency, a proxy network, or ongoing maintenance.
6. Put a sample size, budget, and stop condition around every paid call.

![Figure 2: The build-or-buy decision. Free software still has a maintenance cost, and paid services need explicit triggers.](https://img.upweb.dev/media/0348c6cc-2400x1350-UpWeb-Build-vs-Buy-Web-Data-Diagram.png)

Official APIs are usually the most stable and the narrowest. A specialized scraper can take advantage of a platform's structure, but it needs to track page changes. A browser can handle difficult pages, at the cost of speed, concurrency, and reproducibility. A managed API moves proxies, anti-bot work, structured output, and maintenance to a vendor, in exchange for ongoing spend and vendor dependence.

At the managed layer, I'm testing this article's sponsor, [Context.dev](https://t.upweb.dev/c/context?c=article). It puts scraping, crawling, structured extraction, and retry handling behind one web-data API, which matches step five above. I apply the same rule I use for any paid service: run a small sample, verify the fields and pricing, then decide whether to scale it up.

I compare two bills: the API charge and the time I spend repairing the collection path. Once maintenance time costs more than the service, insisting on a free option is not saving money.

A unified API is not a reason to move every source. When an official endpoint solves the job reliably, I keep the narrower path. One entry point should reduce maintenance, not erase the differences between data sources.

## Once the data worked, credentials became the next problem

As the research agents gained access to more sources, a second set of questions kept coming up.

Which agent may use which API key? Is it running locally, in GitHub Actions, or on a production server? Can development and production credentials be mixed? Does the agent need to see the secret itself? Who may build the app, who may sign it, and who has permission to upload or release it?

If those answers live across `.env` files, GitHub Secrets, and chat history, I can't tell how far a credential has spread. I also can't revoke it quickly after a leak.

So I built an environment, credential, and release control plane.

![Figure 3: The Credential Control Plane. The catalog returns metadata, OpenBao stores secrets, and the Use Broker grants short-lived, restricted capabilities.](https://img.upweb.dev/media/22dfc51b-2400x1350-UpWeb-Credential-Control-Plane-Diagram.png)

### When a control plane starts paying for itself

Copying `.env` files becomes riskier than building a control plane when two or three of these conditions appear together:

- More than one product uses the same third-party services.
- A product has development, staging, and production environments.
- Local agents, CI, and production runtimes all need credentials.
- Build jobs begin touching signing, uploads, or production releases.
- API calls need their own budgets, stop conditions, and audit records.
- After a credential leaks, you need to identify every consumer and workflow that used it.
- You want GitHub Actions to use short-lived identities instead of long-lived cloud keys.

My setup now meets most of those conditions. A self-hosted control plane lets me keep an open-source secret engine while retaining the product rules, release gates, and recovery process in my own system.

### The catalog stores identity and policy, not secret values

The Credential Catalog stores a credential's name, owning product, environment, purpose, risk level, and rotation state. It does not store the secret value.

An agent can learn that a capability exists, which product owns it, and whether it is currently available. Reading the catalog does not reveal the credential. Tool selection and secret storage can change independently.

### Secret values stay in OpenBao

The production secret engine is the open-source [OpenBao](https://openbao.org/docs/). It stores long-lived credentials, while callers use a controlled process to obtain short-lived access. The Credential Catalog and Use Broker are control-plane components I built around OpenBao. They are not OpenBao modules.

Credentials use vendor-independent references such as `credential://<product>/<environment>/<purpose>`. Application configuration doesn't need to know whether the secret ultimately lives in OpenBao, a manual vault, or a recovery vault. Moving the storage layer does not force every caller to change.

### Agents request a capability, not the entire vault

The Use Broker sits between agents and credentials. It checks the caller's identity, product, environment, purpose, and requested scope before granting short-lived access.

That lets me separate research API access, application builds, signing material, test uploads, and production releases. Permission to run research does not imply permission to ship an app.

### GitHub Actions uses short-lived identities

Cloud builds prefer GitHub Actions OIDC or Google Cloud Workload Identity Federation instead of long-lived cloud keys stored in a repository. A workflow proves its repository, branch, environment, and task identity before receiving time-limited permissions.

Signing is tighter. A build exchanges a fresh OIDC identity for a non-renewable OpenBao session with a limited number of calls, retrieves the signing material, and revokes the session. Signed artifacts are stored by their content digest (`sha256`). Promotion accepts the digest, not a mutable tag.

Five products now have separate Android build identities and signing material. One unsigned canary path has already completed successfully. Signing, uploading, and releasing remain separate gates. A successful build does not grant release permission.

### Products and environments must stay separate

The same third-party service may support several products, each with development, staging, and production settings. The catalog and broker both validate the product and environment. A test workflow cannot pick up a production credential, and one product cannot read another product's secret.

The control plane also covers budgets and audit records. I need structured answers to questions such as how many paid searches an agent ran, which workflow requested a production capability, and which credential needs rotation.

![Figure 4: The product-by-environment isolation matrix. Every cell has its own identity and credential reference.](https://img.upweb.dev/media/62b53f1c-2400x1350-UpWeb-Product-Environment-Isolation-Diagram.png)

## How the two layers work together

A research task now follows this path:

1. The agent determines whether it needs search, page content, a logged-in browser, store data, or continuous monitoring.
2. The router selects the narrowest verifiable data interface.
3. The agent asks the broker for the required capability and declares the product, environment, and purpose.
4. The broker checks authorization and returns a short-lived identity or a controlled call path.
5. When data comes back, the system records the source URL, capture time, tool, and task ID.
6. The result enters a report or product decision only after corroboration across sources.
7. Budgets, failures, and permission denials enter the audit record for the next review.

![Figure 5: The end-to-end agent research flow. It starts with declared intent and ends with traceable results, costs, and denials.](https://img.upweb.dev/media/7b19e559-2400x1350-UpWeb-End-to-End-Agent-Research-Flow.png)

This involves more work than simply letting an agent open a page. It also prevents several common failures: committing cookies, printing API keys in logs, letting test jobs reach production, paying for data that a free source already provides, and producing a confident report without enough evidence.

## The rules I kept

Seven operating rules survived this build.

First, use the narrowest stable interface. If an official API can solve the problem, don't start a browser.

Second, reserve browser sessions for authenticated or interactive pages. They should not be the default backend for every collection job.

Third, include maintenance time in the cost of a free option. Repairing a parser several times a month can cost more than a managed API.

Fourth, a unified interface reduces maintenance but does not make data sources semantically equivalent. People Also Ask, semantic search, community posts, and store reviews still need separate verification.

Fifth, an agent receives a limited capability. Secret values, builds, signing, uploads, and releases each have their own boundary.

Sixth, every paid data source needs a budget, a stop condition, and a fallback path.

Seventh, stop when the source, timestamp, or authorization scope cannot be verified. An agent should not guess its way around missing evidence or permissions.

## A checklist you can use directly

Before giving an agent a new web-data capability, I check:

- Does the task need search, reading, authenticated browsing, structured collection, or continuous monitoring?
- Is there an official API or a well-maintained platform-specific tool?
- Do browser cookies stay in the browser, outside repositories, prompts, and logs?
- Has the caller declared the product, environment, purpose, and budget?
- Does the agent receive a limited capability, or can it read the secret itself?
- Do development, staging, and production use different identities and credential references?
- Are build, signing, upload, and release protected by separate gates?
- Does every piece of evidence retain its URL, timestamp, tool, and task ID?
- Can the workflow stop when data or authorization is unclear?
- Does every paid service have a stop condition and a fallback path?

You don't need to copy the entire system. Start by constraining one agent, one CI workflow, and one paid API. Split more permissions as your products and environments multiply.

I started by trying to help an agent read a few more websites. The system now manages data paths, authenticated browser sessions, secrets, short-lived identities, build signing, and audit records.

Giving an agent a browser answers whether it can see a page. Reliable operation needs three more answers: what it is looking at, why it is allowed to look, and what it may do with the result.