# A deep dive into OpenSEO > Explore OpenSEO's architecture: SEO data APIs, Cloudflare Workers, its audit crawler, MCP tools, AI agents, storage, and self-hosting trade-offs. Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-10-16 | Topics: [AI](https://flaviocopes.com/tags/ai/) | Canonical: https://flaviocopes.com/openseo/ [OpenSEO](https://openseo.so/) brings keyword research, competitor analysis, backlinks, site audits, and search visibility into one application. You can use its dashboard, connect an AI agent through MCP, or run the open-source application yourself. The part I find useful is the connection between those workflows. Research can live beside the project it belongs to. An agent can retrieve data, remember the business context, and turn the findings into a report you can read later. The [open-source repository](https://github.com/every-app/open-seo) also gives us a substantial application to learn from. It combines a React interface with server functions, a relational database, object storage, durable background workflows, and stateful agents. Those pieces have different jobs. Understanding the boundaries explains both what OpenSEO can do and what you take responsibility for when you self-host it. I haven't used OpenSEO to run an SEO campaign. This guide is based on its public source and documentation, checked in October 2026. The setup walkthroughs below describe the supported paths; they aren't a claim that I've deployed either one or benchmarked its SEO results. ## What I would use it for Suppose you run a small software business. You know your product well, but you don't know which searches lead people to competing products. OpenSEO can help you explore that market. Start with a competitor's domain, inspect the keywords and pages associated with it, and save the relevant ideas into your own project. Then look at the search results for those ideas and decide whether you have something useful to offer. For an existing publication, I would start with Google Search Console. Which pages already receive impressions? Which queries appear beside them? Is an article losing visibility, or does it still appear frequently but attract fewer clicks? Competitor data can add context to those questions. A technical crawl can reveal a broken canonical or an internal link pointing at a redirect. Rank tracking can follow a small set of searches after you make a change. An agency could use the same project structure to keep client research separate. However, the deployment's permission model matters: the Cloudflare self-hosted workspace is shared among everyone you allow into it. It isn't a set of isolated client portals. I would also consider it for a developer building a focused SEO tool. The MIT-licensed code supplies an interface, data integrations, and useful application patterns. You still need to understand the external providers and their terms before turning a fork into a product. It won't tell you what your customers want without judgment. Search volume can suggest demand. It cannot explain why someone abandons a checkout or whether your product solves their problem. ## Open source application, external data services OpenSEO's application is open source. Its market data comes largely from [DataForSEO](https://dataforseo.com/apis), a separate commercial service. That distinction affects almost every other decision. When you self-host, you control the application and its stored state. You bring DataForSEO credentials and pay that provider for the requests you make. Cloning the repository doesn't download a global keyword database or a backlink index onto your machine. OpenSEO adds the workflow around that data: project defaults, validation, saved research, tracking history, reports, agent tools, and the dashboard. Some data follows other paths. Search Console uses Google's API and the property you authorize. The built-in agent uses OpenRouter for model access. A rendered audit can use Cloudflare Browser Run, with Context as an optional rendering fallback. A normal HTML crawl also fetches pages from the site being audited. It doesn't need to ask an LLM to decide whether a response contains a title element. | Capability | Main source or execution path | | --- | --- | | Keyword and competitor research | DataForSEO APIs | | Backlink research | DataForSEO's backlink data | | Rank observations | DataForSEO search-result collection | | Search Console performance | Google's API for an authorized property | | Technical HTML checks | OpenSEO's crawler and analyzers | | JavaScript-rendered pages | Browser Run or Context, depending on deployment | | Lighthouse checks | DataForSEO integration | | In-app agent reasoning | OpenRouter model access | | AI visibility collection | DataForSEO collection methods that differ by engine | The hosted application adds its own subscription and usage accounting. The [current pricing page](https://openseo.so/pricing) lists a $10 monthly base plan with $10 of included usage, plus additional usage purchases. Those figures were checked in October 2026; check the current terms before comparing costs. Self-hosting removes that hosted application subscription. Provider charges and your infrastructure costs remain. ## The architecture from the outside OpenSEO has several entry points into the same product. A person opens the dashboard. An external agent connects to `/mcp`. The built-in SAM agent uses its own chat connection. Scheduled work enters through Cloudflare's scheduled handler. These entry points converge on server-side features and repositories. They don't all have identical authentication or identical tools. The main application uses **TanStack Start**, React, and TanStack Router. TanStack Query handles client-side server state, while Tailwind supplies the styling system. TypeScript runs through the interface and server code. TanStack Start gives the project a full-stack application structure. Server functions can validate input and call a feature service without exposing provider credentials to the browser. At the runtime layer, the main entry point is [`src/server.ts`](https://github.com/every-app/open-seo/blob/main/src/server.ts). It dispatches ordinary application requests, MCP traffic, agent connections, and scheduled work. The web application is largely organized by feature. For example, `src/server/features/` contains separate areas for keywords, domains, backlinks, audits, projects, reports, integrations, and agents. Services implement product operations. Repositories handle stored application data. Shared provider clients handle the details of talking to DataForSEO. This is a useful structure for a growing application. A keyword tool and an MCP tool can reach the same feature behavior while retaining different entry-point contracts. The site audit engine has a separate runtime boundary. It lives in an auxiliary Worker, rather than loading its full crawl machinery into the main application Worker. We'll follow that path shortly. ## A map of the implementation If you want to follow these mechanisms in the repository, these are useful starting points. The directory boundaries also show where the application keeps different responsibilities. | Area | Source entry point | What to follow | | --- | --- | --- | | Request dispatch | [`src/server.ts`](https://github.com/every-app/open-seo/blob/main/src/server.ts) | App, MCP, chat, and scheduled entry points | | Research operations | [`src/server/features`](https://github.com/every-app/open-seo/tree/main/src/server/features) | Product services and repositories | | External SEO data | [`src/server/lib/dataforseo`](https://github.com/every-app/open-seo/tree/main/src/server/lib/dataforseo) | Provider methods, envelopes, pricing, and retries | | Durable execution | [`src/server/workflows`](https://github.com/every-app/open-seo/tree/main/src/server/workflows) | Audit, ranking, and visibility orchestration | | Audit state | [`AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts) | Frontier ownership and temporary SQLite state | | MCP | [`src/server/mcp`](https://github.com/every-app/open-seo/tree/main/src/server/mcp) | Authentication, schemas, transport, and tool wrappers | | In-app agent | [`src/server/features/sam`](https://github.com/every-app/open-seo/tree/main/src/server/features/sam) | Chat sessions, memory, tools, and model execution | | SQL backend | [`src/db`](https://github.com/every-app/open-seo/tree/main/src/db) | Schemas, provider selection, transactions, migrations | | Infrastructure | [`deploy/alchemy`](https://github.com/every-app/open-seo/tree/main/deploy/alchemy) | Resource provisioning and deployment modes | ## Why Cloudflare appears throughout the code The application is built around Cloudflare bindings. A **binding** gives a Worker access to a configured resource, such as a database or bucket. Application code receives the capability through its environment instead of discovering it from a network address and password. OpenSEO uses D1 for the default relational database, R2 for object storage, KV for selected lightweight state, Cloudflare Workflows for durable background orchestration, and SQLite-backed Durable Objects for specific stateful jobs. That is why a generic Node server plus a Postgres URL isn't a complete deployment recipe. The application expects more runtime services than an HTTP listener and SQL connection. Docker runs the Cloudflare development and preview runtime locally. It provides local implementations of those bindings and persists their state in the container's mounted data directory. This makes the same application approachable on a laptop. It also means Docker is running a local version of an edge application, with its own runtime constraints and startup machinery. If those concepts are new, my free [Cloudflare Workers course](https://flaviocopes.com/courses/cloudflare-workers/) explains the execution model and bindings. The distinction between a short request and a durable job becomes especially useful in this codebase. ## Follow a keyword request Let's use a concrete question: which searches relate to a waiting-list tool? In OpenSEO, the project supplies context such as a domain, location, and language. The person doing the research chooses a seed or another research operation, then applies result limits and filters. The server first needs a valid user context and access to the relevant project. A browser choosing a project ID doesn't establish ownership by itself. The feature service turns the request into the appropriate provider operation. Related keywords, domain keywords, search volume, and live search results are different requests. A page that displays several of them can require several provider calls. The provider client adds authentication on the server, sends the request, and interprets the response. A response has an HTTP status and a provider-specific result envelope. Checking only for HTTP 200 would miss task-level failures. After that, the feature can map provider-specific fields into rows the interface understands. Saving a keyword is another operation: it adds the idea to the project's canonical saved list. A transient research response and a saved keyword therefore have different lifetimes. Closing a tab shouldn't erase a saved keyword, and saving one doesn't mean every possible metric has been copied into that row forever. The same distinction applies when an agent does the research. It can retrieve results, decide which ideas matter, and explicitly save the selected ones. An MCP response by itself isn't an editorial decision. ### Research is a composed operation The keyword service does more than forward one seed to one endpoint. It normalizes and deduplicates seeds, selects a supported provider path for the market, and combines results according to the requested mode. Its automatic mode can combine keyword suggestions and ideas, interleave them, remove duplicates, and fall back to related keywords when the initial result is thin. Countries outside one provider dataset's coverage can use a Google Ads path instead. Local research needs another distinction. The service can start with national ideas and then replace selected metrics with local Google Ads metrics. It avoids overwriting the national metric records with those local values. That behavior explains why a research request needs a richer cache key than its keyword string. Market, language, requested limit, research mode, and local settings can all change the answer. The domain and backlink features have their own scope rules. A hostname overview and a subfolder report aren't automatically the same provider operation with a different label. Some upstream summary or history endpoints cannot represent every narrower scope. The application must label or omit data rather than pretend the provider applied an unsupported filter. This is where a product earns its place above an API client. It needs to preserve the meaning of each field while combining sources into one interface. ### Location and language are part of the question The keyword alone doesn't identify an observation. A search in one country can have a different result set from the same search elsewhere. Language, device, result limits, and filters can change both the returned data and the work required to collect it. OpenSEO stores project market defaults and also records market information alongside saved or tracked data. That prevents a list of identical keyword strings from pretending to describe one uniform market. For a useful comparison, keep the market settings consistent. If you change the country halfway through an investigation, label the new results accordingly. I would rather track a few searches in the market I can serve than collect a huge international list with no clear use. ## Caching, saved data, and freshness A client cache helps the dashboard avoid unnecessary loading. A provider-result cache helps avoid repeating external work. Saved project data gives the user a durable record. OpenSEO's [`r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts) stores cached provider responses in R2. It builds deterministic keys from the operation's parameters and puts an expiration timestamp in object metadata. On a read, an expired object is treated as a miss. That is a **soft TTL**: the application refuses to reuse the object after its expiration. The timestamp doesn't itself prove the underlying object has been physically removed from storage. Research paths can also validate cached data against a schema. That matters after a deployment changes the expected shape. A valid JSON document can still be the wrong kind of result. Caching policies belong to individual operations. Don't assume every endpoint has the same lifetime or every dashboard action is free because something similar was fetched earlier. The product trade-off is freshness against repeated work. A cached keyword response can be useful for planning. A scheduled rank observation needs a new measurement at the intended time. The database stores the parts that must survive as product history: the project, saved list, tracking configuration, and collected observations. An object cache isn't a substitute for that history. When interpreting any number, ask when it was collected and which operation produced it. A smooth interface can otherwise make yesterday's research look like a live measurement. ## Where the data lives OpenSEO has several storage systems because its data has different shapes and lifetimes. | Storage | Responsibility | | --- | --- | | D1, or the optional Postgres backend | Users and organizations, projects, saved research, tracking configuration and results, integrations, report documents | | R2 | Cached provider payloads, larger audit artifacts, generated exports | | KV | Selected progress and transient state; a separate namespace holds hosted MCP OAuth data | | SAM Durable Object SQLite | Persistent chat messages and agent state for a session | | Audit scratchpad Durable Object SQLite | Temporary crawl frontier, seen URLs, link edges, and slim page state | The distinction between SQL and R2 isn't simply small files versus large files. It is also about access patterns. A project listing needs indexed rows and relationships. A large cached response can be retrieved as an object by a deterministic key. A crawl frontier needs tightly coordinated updates while a job runs. Reports are an easy place to make the wrong assumption. Their metadata and HTML are stored in the relational database. R2 is used for other artifacts, but the presence of a bucket doesn't mean every generated document lives there. ### The relational model follows ownership The central relationship is organization → project → feature data. A project records its organization, name, optional domain, and default market. Saved keywords belong to a project. Audit runs belong to a project, and crawled pages belong to an audit. Tracking settings and observations build their own history beneath a project. This makes a project more than a visual folder. Its ID is part of the authorization and query boundary. Repository methods frequently scope a child lookup by its project as well as its ID. If a caller is authorized for one project, guessing a report ID from another project must not return that report. Projects also support archival. Hiding a project and deleting all its history are different operations, so the application can preserve underlying data when a project is archived. ### D1 and Postgres are two backends D1 is the default self-hosted database. The code also has a Postgres path through Cloudflare Hyperdrive. Drizzle supplies the query layer, with separate schema declarations and migrations for SQLite and Postgres. Schema-parity tests help keep the application-facing structures aligned. The abstraction has to account for real backend differences. OpenSEO's [`runBatch`](https://github.com/every-app/open-seo/blob/main/src/db/runBatch.ts) helper uses D1's batch operation for one backend and a transaction for the other. Statements must be built from the provided transaction handle when using Postgres. Without that detail, a helper could appear transactional while some queries execute outside the transaction. Postgres clients are also scoped to an invocation. The application uses request-local context so feature code can reach the correct client. The Worker doesn't share an open database socket from an unrelated earlier request. Hyperdrive is the configured connection path. Setting a random database URL doesn't replace the required binding. Postgres gives the project another scaling path, but it doesn't remove Cloudflare Workflows, agent storage, object storage, or the audit engine. It changes a database backend within the existing architecture. ## Domain and backlink research Competitor research starts with a different kind of uncertainty from Search Console. You can authorize Search Console access for your own site. You usually cannot authorize access to a competitor's property. Market-research providers therefore estimate visibility from the data they collect. OpenSEO's domain features expose that provider data through project-aware operations and tables. You can examine ranking keywords and pages, apply filters, and move from a domain summary into a more specific investigation. Backlink research asks where links to a domain or page have been observed. Referring domains, individual backlinks, history, and quality indicators answer different questions. A large backlink count alone tells me little. I would inspect the actual source pages, whether the links are relevant, and whether the proposed opportunity makes sense for my audience. There is also a product-design lesson here. Large research tables need server-side pagination and deliberate filters. Fetching a giant dataset into a browser just to hide most rows would waste provider budget, memory, and transfer. Compare data providers with the same domain, market, and date. Their coverage and collection methods differ. An estimate from one index doesn't become a fact about a competitor's actual visitor count because it appears beside a precise number of rows. ## Rank tracking is a durable job Rank tracking records observations over time. It needs more than a button that performs a search. A configuration identifies what to check and under which market settings. A scheduled handler finds due work. A workflow then submits and collects the observations and saves the results. Cloudflare's cron wakes the dispatcher periodically. It doesn't mean every tracked keyword is checked on every tick. The provider also has an asynchronous task system. OpenSEO can submit a DataForSEO task, retain the task ID, wait, and retrieve the result later. That gives us two separate mechanisms: - Cloudflare Workflows preserves the application's progress through a job. - DataForSEO's task queue performs the external collection. Neither should be confused with Cloudflare Queues. A variable named `queued` can describe a provider task without the application having a Queue binding. Workflows are useful here because waiting for a search-result task shouldn't require keeping the original browser request open. The dashboard can show progress and retrieve the completed observation later. The operational questions are more interesting than the timer expression. What happens when a job is dispatched twice? What happens when a provider task completes after the application loses its connection? How does the next run avoid charging for an unnecessary replacement task? OpenSEO has run guards, stored task state, and reconciliation paths around its background work. Those reduce duplicate work and help recover from interruptions. They don't make every external paid operation part of one atomic transaction. There is a current availability caveat. The October 2026 source contains a hosted-only pause flag for scheduled rank tracking. Manual checks remain available in that code, while self-hosted schedules bypass that hosted pause. Check the hosted service's current status before choosing it for an unattended tracking requirement. ### Claiming a schedule without creating a storm The scheduler reads due configurations in order and limits how much work it admits in one tick. Its budget accounts for keyword and device combinations, because tracking desktop and mobile creates separate collection tasks. A compare-and-swap update claims the schedule slot. That means the write succeeds only if the stored state still matches the state the dispatcher examined. Concurrent ticks or a configuration edit cannot silently claim the same old slot through an unconditional overwrite. There are deliberate failure trade-offs. A workflow launch failure doesn't immediately put the configuration back into an unlimited retry loop. Existing active runs and temporary plan-lookup errors take different paths so the next tick can make a controlled decision. The workflow writes observations incrementally. If a later batch fails, earlier persisted observations remain available. Finalization can then describe the run from stored results rather than depend on one enormous response surviving in memory. Scheduled checks generally use queued provider tasks; manual checks begin with live collection. The implementation includes fallbacks for missing or failed tasks. A queued task followed by a live fallback can incur both charges, so durable orchestration doesn't imply exactly-once provider spending. A missing result also isn't automatically a lost ranking. The code has paths that preserve the previous observation rather than invent a new not-found position from an unsuccessful fallback. The UI needs to distinguish old evidence, fresh absence, and collection failure. ## The site audit engine An audit begins with a URL and a bounded scope. It discovers pages, fetches them, analyzes them, and builds results across the crawl. That sounds like a loop. In a durable application, the hard part is deciding which state survives between iterations and what happens when the work stops halfway through. OpenSEO separates the audit engine into an auxiliary Worker. The main application starts its `SiteAuditWorkflow` through a cross-script binding and later reads status or issues control operations. The source documents memory pressure as the reason for this split. In-flight HTML and large Lighthouse payloads can consume enough memory to interfere with the main application. A separate Worker keeps that workload out of the main request handler's heap. It remains one feature in the product, with shared permanent storage. This is a runtime isolation decision rather than a requirement to turn every feature into a separate service. ### Discovering the next page The crawler needs a **frontier**: URLs it knows about but hasn't finished processing. Discovery can draw from the starting page, sitemap information, and links found while crawling. URL policy and normalization keep the work inside the intended scope and reduce duplicate fetches. Without those rules, fragments, tracking parameters, alternative spellings, and links outside the intended site can make a crawl much larger than expected. OpenSEO keeps a per-audit scratchpad in a SQLite-backed Durable Object. It holds the frontier, seen-set, link edges, and a slim mirror of page information needed during analysis. That object gives one audit a coordinated state owner. Workers can advance the crawl without copying the whole graph through every workflow step or repeatedly scanning permanent application tables. The scratchpad is temporary. It is destroyed when the audit successfully finalizes, and a cleanup alarm handles abandoned state. Permanent results need to be written elsewhere before that cleanup. ### Small checkpoints and controlled memory The workflow processes bounded chunks of leased URLs. A chunk receives a stable identity, so retrying it can recover the same lease rather than quietly claim another unrelated set of pages. The scratchpad's writes use stable keys and idempotent insert or replacement behavior. Completed chunks don't become permission to crawl a new chunk under the same identifier. Workflow checkpoints return small counters and status rather than the whole URL frontier or every downloaded document. The frontier stays in its state owner, and larger artifacts live in storage designed for them. The crawler also applies backpressure. When page persistence falls behind fetching, it pauses launching more work instead of accumulating an unbounded pile of HTML in memory. Its adaptive fetch window is conservative; the existence of a concurrency controller doesn't mean the current implementation launches hundreds of fetches at once. A `429` response can pause new fetches using the target's retry guidance or a fallback delay. Throttle state carries across chunks, so starting the next durable step doesn't immediately forget that the site asked the crawler to slow down. There are still bounded-analysis limits. The link graph has a storage ceiling, and rendered-page analysis truncates very large HTML. A crawl can keep producing useful findings without containing a complete representation of every page and edge. That changes how I would read an orphan-page or internal-link report. At a page cap, after blocked responses, or after graph truncation, the report describes the crawl's observed graph. It cannot prove a page has no inbound link anywhere on the site. ### Fetching and classifying responses Fetching a page doesn't always produce analyzable HTML. The server can redirect. A CDN can send a challenge. A rate limit can return an error document. The response can be an image or another non-HTML resource. OpenSEO's audit implementation separates fetch classification from page analysis. That prevents a blocked request from being interpreted as a normal page with no heading and very little text. Robots policy, crawl throttling, scope checks, and bounded concurrency are also part of the job. Increasing concurrency can make a crawl faster until it causes the target to throttle or block the crawler. For a site you control, start with a small page budget and inspect the response classifications. A report full of access failures needs a crawler-access diagnosis before it needs a content rewrite. The default Worker configuration also uses Cloudflare's public-only fetch compatibility behavior. That is a useful network boundary, but an internet crawler still needs URL validation and deliberate redirect handling. It shouldn't become a general-purpose proxy into networks it wasn't intended to inspect. ### Page checks and cross-page checks Some checks are local to one response: title, description, headings, canonical URL, indexability directives, and selected structured data or social metadata. Others need the crawl's wider picture. Duplicate titles require comparing pages. Link relationships require knowing which page pointed where. A canonical target can make sense in isolation while conflicting with other signals across the site. That is why the frontier and link graph are useful beyond downloading pages. They support analysis that a single-page parser cannot perform. Audit severity still needs interpretation. A missing description on an unimportant utility page and an accidental `noindex` on a product page deserve different attention. The application can organize findings; you decide which pages matter to the business. ### Rendering JavaScript changes the workload Raw HTML can be enough for a server-rendered site. A client-rendered site may expose its content and navigation only after JavaScript runs. OpenSEO supports a rendered audit path. On Cloudflare, Browser Run is the first renderer, with Context available as a fallback for pages the browser cannot render. Docker doesn't include that browser binding; its documented rendering path uses Context credentials. Rendering adds browser time, resource requests, and more ways for a page to fail. A page can have valid HTML but fail to hydrate, depend on a blocked third-party script, or render different content after a delay. The self-hosting operations guide currently requires a paid Workers plan for the supported Browser Run crawl path. A basic self-hosted app fitting within free allowances doesn't make every optional execution path free. Lighthouse is another workload again. OpenSEO integrates DataForSEO's Lighthouse service. A Lighthouse result is a lab observation under that test's conditions, rather than a measurement of every real user's experience. For actual field performance, I would keep real-user measurements alongside the lab results. One score shouldn't become a substitute for investigating the requests and scripts on a page. ### Recovering a stuck audit A durable workflow can still fail. Runtime limits, provider failures, invalid responses, and deployment changes can leave a run without its intended final write. The main application's scheduled handler runs a stale-audit reconciler. It looks for audit state whose execution has stopped making progress and updates the product's view of that run. This is an important pattern: the job writes its normal outcome, while an independent watchdog handles jobs that never reach their normal cleanup code. The scratchpad's abandonment cleanup solves another problem. Marking an audit failed in the database doesn't automatically remove every temporary object it created. If you're designing a background feature yourself, write down the permanent result, temporary state, and recovery owner separately. OpenSEO needs all three because a crawl crosses multiple storage and execution boundaries. ## MCP is another application interface OpenSEO exposes tools through [MCP](https://flaviocopes.com/what-is-mcp/), so an external agent can work with the same projects and data. A useful request might be: ```text Use the project for my waiting-list tool. Read its context first. Find a small set of relevant search opportunities in the project's market. Reuse existing research where possible. Tell me which fresh requests would cost money before expanding the investigation. Save the selected keywords and a report containing the evidence and the next actions. ``` This is a workflow request. The actual work happens through tool calls, each with an input schema and server-side behavior. OpenSEO's MCP implementation includes transport handling, authentication, project authorization, response formatting, output schemas, and individual tools. It is more than a collection of prompt snippets around an API key. A skill tells the agent how to approach a task. A tool gives it an operation it can execute. Neither should be treated as proof that the model's interpretation is correct. The distinction is useful when debugging. If a tool returns the wrong data, inspect the parameters and server operation. If the data is right but the recommendation is poor, inspect the agent's reasoning and instructions. ### Identity must reach every tool call In hosted mode, the MCP connection can use the application's OAuth system. The server also supports API-key authentication. The credential establishes a user identity; project access is checked for the requested project. That user can belong to more than one organization. The server derives the organization from the authorized project for the operation, instead of trusting a stale active-workspace choice from an earlier browser session. This matters for both data isolation and billing. A paid request must belong to the organization whose project is being researched. The project guard also protects operations on child records. A tool can't safely accept a report ID, tracking configuration ID, or connection ID and assume it belongs to the supplied project. On a Cloudflare Access deployment, authentication follows a different path. Access protects the application and the server validates its assertions. Managed OAuth needs additional configuration for MCP clients and their callback addresses; a browser login alone doesn't guarantee an MCP client can complete its connection. The local Docker mode has no application authentication. An MCP endpoint reachable through that local mode inherits that trust assumption. Changing the hostname doesn't create a permission system. My free [MCP course](https://flaviocopes.com/courses/mcp/) explains the protocol pieces. OpenSEO is a useful larger example because its tools reach paid operations, persistent data, and several organizations. ### Stateless transport, persistent product state The MCP transport is intentionally stateless between tool requests. The product's durable state lives in projects, repositories, provider tasks, and other explicit storage owners. This is different from SAM's persistent chat object. An external agent can keep its own conversation while making independent MCP requests. OpenSEO doesn't need a long-lived MCP conversation object to remember the user's saved keyword list. A user can lose access after obtaining a credential, so tool calls still need current membership checks. The tool's project gate needs to consult the current relationship rather than trust an organization snapshot indefinitely. An API key accepted for MCP isn't a browser session for arbitrary account-administration endpoints. Keeping those surfaces distinct reduces the amount of authority one integration credential accidentally acquires. The diagram above shows successive calls made by the external agent. A research tool doesn't secretly call a report-writing tool: the agent reads the result, constructs a document, and makes a separate save request. ### Tools need spending boundaries An agent can generate many plausible research calls. That is convenient until each call spends money. A prompt such as “research every competitor and every keyword” doesn't define a budget. It also leaves the agent free to explore directions that may have little business value. I would begin with one project, one market, and a bounded set of questions. Ask the agent to reuse saved data and identify uncertainty before fetching more. The server's validation and metering remain necessary even with careful instructions. A prompt is guidance for the model, while an authorization check and a spending gate run regardless of how the model phrases its request. Third-party pages and search snippets also enter the agent's context. Treat them as evidence, rather than instructions that can change the task or authorize new actions. ## Project context is stored application data OpenSEO can keep business context with a project: what the product does, who it serves, competitors, and other facts that help research stay relevant. This solves a practical problem with occasional agent sessions. Without stored context, each new session starts by guessing the same things about the business. The server provides explicit operations to read or update that context. It is ordinary durable application state that the agent can use later. Project context also includes structured competitors, selected important pages, and bounded research-log summaries. Those can help an agent avoid buying the same research repeatedly, provided it reads and uses the stored context. The memory update path validates operations and groups related writes atomically. A partially applied business-context edit could otherwise leave the next agent with contradictory information. It isn't model training. Saving a business description doesn't change the model's weights or guarantee the model will follow every preference correctly. That distinction also helps with maintenance. If the product changes, update the context. Otherwise, a technically correct keyword search can still be based on an obsolete description of the business. I would keep the context short enough to inspect. A long accumulation of unreviewed agent notes can preserve old assumptions and make them look authoritative. For a tool such as [Waiting Lists](https://waitinglists.dev), I would record the actual self-hosted workflow, double opt-in behavior, and intended users. I would avoid claiming it has integrations or a managed service unless those exist. That gives the research a real boundary before the agent proposes topics. ## SAM, the built-in agent SAM is OpenSEO's in-app assistant. It shares feature services with the rest of the application, but it has a different execution surface from an external MCP client. Each chat session has a SQLite-backed Durable Object. A relational session record ties that session to the user and project. The Worker authorizes the connection before routing chat traffic into the object. That check applies to both the live connection and message-history requests. Knowing a session's identifier must not be enough to read it. The agent can then load project context, call its available tools, and stream responses through the chat connection. Model access goes through OpenRouter; self-hosting requires the relevant key to enable that functionality. The model doesn't get to select an arbitrary project by changing a tool argument. SAM's server-side tool wrappers supply the authorized project context. OpenSEO also bundles public skills into SAM, adapting their instructions to the in-app environment. SAM has no local filesystem, so a workflow that tells an external coding agent to save a file cannot run unchanged. There is a particularly useful boundary: SAM currently has no report tools. Its adapted skills explicitly tell it to return findings in chat and skip report-saving steps. External MCP clients can use the report tools; SAM's chat history is a separate artifact. The conversation belongs to the individual user's session, while project context is shared with authorized project users. Moving shared knowledge into project context therefore changes who can see it compared with leaving it in a private chat. SAM also compacts and clips context as it grows. The Durable Object's memory budget matters alongside the model's context window: retaining a huge tool response in application memory can fail before the model reaches its token limit. Persistent messages don't mean an interrupted inference is automatically replayed. The implementation deliberately avoids automatic recovery that could repeat paid tool calls. Partial conversation state can survive while the user still needs to ask it to continue. SAM has no local shell or filesystem for editing a website. It can explain a problem and use its available data tools. Applying a code change belongs to a separate editing environment or external coding agent with that access. This is a good example of why listing an SDK dependency doesn't explain an agent's capabilities. You need to inspect its actual tools, authorization wrapper, persistent state, and instructions. ## Reports are stored HTML with a security boundary External agent workflows can produce a self-contained HTML report and save it into a project. That gives a person something more useful than a long transcript. The report can organize evidence, explain recommendations, and preserve the result after the agent session ends. The report service validates document structure and byte limits before saving. It also enforces project and organization storage guardrails. An update replaces the stored content; it isn't a full document-versioning system. The application stores report HTML in SQL and retrieves metadata separately for listings. Loading every complete report just to draw a table would needlessly put large documents into the application's memory. HTML generated from crawled pages and model output is untrusted. OpenSEO serves it inside a restrictive sandbox and applies a Content Security Policy to the report response itself. The normal report policy blocks scripts, network connections, and form submission. The sandbox doesn't grant the report the application's origin. That matters even when someone opens the document outside its usual embedded viewer. Printing uses a narrowly allowed application-supplied script. It doesn't generally turn on arbitrary scripts from the report. Hosted public share links are another access mode. The code currently disables report sharing on self-hosted installations, while private reports remain available there. A public link is also a disclosure decision: a beautifully formatted report can still contain commercially sensitive research. The sandbox reduces what an HTML report can execute. It cannot establish whether its conclusions are true, whether an outbound link is a good recommendation, or whether every fact in the report belongs to the right client. ## Search Console and Analytics answer different questions Search Console gives OpenSEO access to performance data for a property you authorize. That includes clicks, impressions, positions, and query or page dimensions supported by Google's API. This is first-party site evidence. It is still subject to Google's reporting rules, aggregation, quotas, and omitted data. An API query isn't a raw dump of every individual search event. Self-hosted Google integration needs a Google OAuth client, an exact callback URL, and a secret used to encrypt stored tokens. OpenSEO stores the grant and refreshes access as needed. Back up the encryption secret with the database. Restoring encrypted tokens without the key doesn't restore a usable integration. Google sign-in and permission to read analytics are separate grants. Logging in with Google doesn't automatically authorize Search Console or GA4 access. The integration code stores data grants against the connecting user and Google account. A project mapping identifies the selected property and the grant used to read it. Other authorized members of the project can consume its data through that connection. That makes the connector's account an operational dependency. Removing the grant, losing Google-side property permissions, or removing the connector can break reads even though the OpenSEO project still exists. A connected Google account and a selected project property are also separate concepts. The account can have access to several properties; the project needs the intended one. The repository additionally has Google Analytics integration code and GA4-specific services. GA4 measures activity on an instrumented site, whereas Search Console describes its appearance in Google Search. Neither replaces competitor research. When troubleshooting, isolate the boundary: missing OAuth configuration, a callback mismatch, an account without property access, and an API quota problem need different fixes. ## AI visibility needs a careful definition AI visibility asks whether a brand appears in answers to selected questions and whether those answers cite its site. That is a useful observation when recorded with its conditions. It becomes misleading if presented as a single universal measure of how often “AI recommends you.” OpenSEO lets a project define prompts and markets, collect answers from supported engines, and preserve mentions and citations over time. The collection method differs by engine. In the current implementation, ChatGPT, Gemini, and Google AI Overviews use DataForSEO's scraper or search-result collection paths. Claude and Perplexity use its model-response APIs with web search. Those latter responses aren't a recording of a signed-in person's consumer app session. Even a collected app answer isn't a guarantee of what another person will see with a different location, account history, time, or prompt wording. ### A mention and a citation are separate signals A **mention** asks whether the brand name appears in the answer. A **citation** asks whether the answer links to an owned domain. OpenSEO's matching code applies text matching for mentions and domain matching for citations. A generic brand word can be ambiguous. A citation to a competitor talking about your product isn't a citation to your own site. The two signals can disagree. An answer can name a product without linking to it. It can also link to a page without repeating the brand in the main answer text. I would inspect the underlying answers before acting on a trend. The aggregate is useful for finding a question worth investigating; the answer provides the context. ### Comparable runs need comparable inputs If one run checks ten prompts and the next checks fifty, the aggregate can move because the sample changed. The same problem appears when the engine, market, or brand definition changes. A before-and-after chart should compare like with like. OpenSEO has explicit comparison logic around matching prompt, engine, market, and brand cells. It also applies answer-coverage thresholds before showing certain changes. Failed collection and missing answers shouldn't automatically count as brand absence. That is a sound reporting idea, but it doesn't turn a chosen prompt list into a representative sample of all searches. You still chose what to measure. I would begin with a few customer questions that matter to the product, keep the conditions stable, and review the actual cited pages. Repeating hundreds of vague prompts can produce a larger bill without a clearer decision. OpenRouter's role is separate here. It can help SAM reason or suggest research prompts. The tracked answer collection uses DataForSEO. A model helping you plan the measurement isn't necessarily the engine being measured. ## Paid requests and failure handling A paid API changes how you design retries. For an ordinary read from your own database, retrying a transient connection failure can be cheap. Submitting an external research task can create billable work even if the response never reaches your application. OpenSEO's provider client distinguishes HTTP failures from task-envelope failures, bounds request time, and treats retry behavior carefully. It doesn't blindly replay every timeout as though nothing happened upstream. Hosted requests add credit accounting through Autumn. The application estimates or reserves usage around operations and later reconciles provider-reported costs through its billing paths. Self-hosted requests bypass that hosted billing layer. DataForSEO still sees your credentials and bills the provider account directly. There is no single transaction spanning the application database, the provider's task system, a workflow, and hosted credit accounting. Recovery therefore depends on identifiers, recorded state, and reconciliation. That is a broader lesson for any app calling a paid external service. “The request failed” can mean validation rejected it before execution, the provider rejected it, work completed but the response was lost, or saving the result failed afterward. Those states have different retry and accounting consequences. A generic “try again” button can hide that distinction from the user, but the server still needs to handle it. For a self-hosted installation, I would use provider spending controls where available and start with small limits. Scheduling turns an occasional expense into recurring work, and agents can expand a research task faster than a person clicking through a dashboard. ## Run it locally with Docker Docker is the documented starting point for trying the self-hosted application on one machine. On macOS, use Docker Desktop or another compatible Docker environment with Compose support. Docker runs Linux containers through a VM, so the mounted volume and port mapping are part of that local setup. Clone the repository and copy its environment template: ```bash git clone https://github.com/every-app/open-seo.git cd open-seo cp .env.example .env ``` In `.env`, set `DATAFORSEO_API_KEY` to the Base64 credentials supplied through DataForSEO's API access page. The value represents its API login and password. Base64 is an encoding, so treat that value as a secret. You can also opt out of the application's anonymous self-hosting telemetry: ```dotenv OPENSEO_TELEMETRY_DISABLED=1 ``` Start the published image: ```bash docker compose up -d docker compose logs -f open-seo ``` The supplied Compose file defaults to `ghcr.io/every-app/open-seo:latest` and maps the app to `127.0.0.1:3001` on the host. Open [localhost:3001](http://localhost:3001). The application doesn't become ready immediately. Startup validates configuration, applies local migrations, and builds the application when the current image or build-relevant environment requires it. Later restarts can reuse an unchanged build. Check its health after startup: ```bash curl --fail http://localhost:3001/api/health ``` A healthy response is a configuration and database check. It doesn't prove that every paid provider operation, Google connection, or optional model integration works. ### Docker's authentication assumption The Compose configuration forces `AUTH_MODE=local_noauth`. The application injects a local administrator identity. There is no app login protecting the dashboard in this mode. The loopback port mapping is therefore significant. Don't change it to a public listener just to make the application reachable from another computer. If you add a reverse proxy or tunnel, it needs an appropriate access boundary. `ALLOWED_HOST` allows a hostname through Vite's host checks; it doesn't authenticate the people using that hostname. TLS also doesn't add a login. For shared internet-facing use, the documented Cloudflare path gives the application the authentication model it expects. ### Persistence and scheduled work The named `open_seo_data` volume mounts `/app/.wrangler`. That directory contains the local runtime's persistent resource state. Removing a container is different from deleting its volume. Be particularly careful with Compose commands that remove volumes: the image can be downloaded again, while your project state cannot be recreated from the image. The container supervisor runs the preview server and a local scheduler together. The scheduler invokes the local scheduled handler periodically because the preview runtime doesn't emit cron events on its own. That means the container must remain running for scheduled checks. A laptop sleeping overnight isn't an always-on tracking server. For an update, back up first, then pull and recreate the service: ```bash docker compose pull docker compose up -d ``` Pin an image tag you have checked when you need a repeatable deployment. Keep the previous image available, but don't assume an older image can read a database after every newer migration. The repository's [Docker guide](https://github.com/every-app/open-seo/blob/main/docs/SELF_HOSTING_DOCKER.md) covers optional variables and operational commands. My free [Docker course](https://flaviocopes.com/courses/docker/) covers volumes, networking, and Compose if those pieces are unfamiliar. ## Deploy it to Cloudflare The supported Cloudflare self-hosting path provisions the application with Alchemy, the infrastructure code in the repository. It creates the storage resources and Workers, applies migrations, and provisions a Cloudflare Access application. This is more than uploading the main JavaScript bundle. You need a Cloudflare account with R2 enabled, a recent supported Node installation, pnpm, and DataForSEO credentials. R2 activation can require a payment method even when your usage stays within free allowances. Start from your clone, then install dependencies: ```bash corepack enable pnpm install ``` The documented login and bootstrap sequence is: ```bash pnpm alchemy login pnpm alchemy cloudflare bootstrap ``` The login needs the Access permissions described in the [self-hosting guide](https://github.com/every-app/open-seo/blob/main/docs/SELF_HOSTING_CLOUDFLARE.md). Bootstrap creates the infrastructure tool's state store in your account. Copy the dedicated template: ```bash cp deploy/.env.selfhost.example .env.selfhost ``` Set `DATAFORSEO_API_KEY` and `ACCESS_ALLOWED_EMAILS` there. The email list defines who can enter through the provisioned Access gate. Keep this file private. Deploy with: ```bash pnpm deploy:selfhost --yes ``` The deployment can also accept an existing Access configuration through `TEAM_DOMAIN` and `POLICY_AUD`. Choose whether infrastructure code or your own Access administration owns that policy; otherwise a later deployment can overwrite a dashboard change you intended to keep. Open the printed Worker URL and authenticate through Access. Then verify the dashboard, health endpoint, and one deliberately small provider request. Everyone allowed through this self-hosted Access deployment works in the same workspace and sees its projects. That is useful for a small trusted team. It is a poor starting point for mutually isolated clients or unrelated tenants. For external MCP clients, configure Access Managed OAuth and the appropriate redirect addresses. Connect them to your deployment's `/mcp` endpoint. Follow the [operations guide](https://github.com/every-app/open-seo/blob/main/docs/SELF_HOSTING_CLOUDFLARE_OPERATIONS.md), because the Access settings and the client's OAuth callback behavior must agree. I would avoid replacing this process with a generic one-command Worker upload. The app and auxiliary audit Worker have cross-script bindings, persistent storage, workflow registrations, and authentication settings that must remain consistent. ## Add integrations gradually I would verify the base app before adding every optional integration. Start with a project and one small research operation. Confirm that the result uses the intended market and that the provider account shows the expected activity. Next, connect Search Console if you have a verified property. Configure the Google OAuth client and callback precisely. Preserve the secret used for token encryption. Add OpenRouter when you want SAM or the prompt-research features that depend on it. Supplying a model key doesn't configure DataForSEO, and supplying DataForSEO credentials doesn't enable every model-powered setup step. Try a bounded HTML audit before enabling browser rendering. That makes it easier to distinguish URL policy, fetch failures, and analyzer behavior from browser-specific problems. Finally, enable schedules for a small set of observations. Check that they finish after the browser closes. A manual button working once doesn't verify the scheduled path. This sequence gives each failure a smaller set of possible causes. ## Backups and recovery A backup plan needs to cover the state you cannot cheaply reconstruct. For Docker, preserve the persistent runtime volume and private configuration. Take a consistent backup with the service stopped or through a storage-aware procedure, rather than copying changing database files and assuming the result is coherent. For Cloudflare, plan separately for relational data, important R2 objects, relevant KV state, and durable chat state. A database export alone doesn't contain every resource in the architecture. The audit scratchpad is intentionally disposable. Saved projects, reports, integration grants, and completed observations have a different recovery value. Secrets matter too. The DataForSEO account can supply new credentials, but losing an encryption key can make stored OAuth tokens unusable. Infrastructure configuration also explains how restored resources should be bound to the application. Keep a deployment revision with your backup. Separate SQLite and Postgres migration histories mean a recovery procedure needs to know which backend it is restoring. I would test recovery into a separate environment and leave schedules disabled during that test. Otherwise, a successful restore can accidentally start paying for duplicate observations. The infrastructure destroy command is especially consequential: it can delete the stage's stored data along with its Workers. Use the project's operational documentation and understand resource retention before treating teardown as a routine restart. ## What will limit it as usage grows? Several bottlenecks can appear, and adding a larger SQL database addresses only some of them. Provider latency and cost affect every fresh research operation. More users can mean more concurrent paid work, even when the application's own CPU usage is modest. Crawl resources grow with pages, link edges, HTML size, rendering, and optional Lighthouse work. A page limit helps bound a job, but not every page costs the same amount to process. Database access patterns determine how project listings, history, and large result tables behave. Pagination and scoped indexes matter before a backend migration becomes the right fix. Persistent objects and artifacts need retention rules. Cached responses with soft expiration can become storage housekeeping work, while saved reports and history need an explicit product policy. Long-running operations also need observability. Look at the main Worker, auxiliary audit Worker, workflow state, and provider errors separately. A responsive dashboard doesn't prove that background collection is healthy. The code's split between app and audit execution is a practical response to one observed class of pressure. It doesn't establish a throughput guarantee for an arbitrary workload. I would measure the intended deployment with real limits before promising it could serve an agency's entire portfolio unattended. ## OpenSEO versus the alternatives The right comparison depends on which job you need done. | Option | Why I would consider it | Main trade-off | | --- | --- | --- | | OpenSEO self-hosted | Custom workflows, owned application state, inspectable code, agent tools | Operating the app and paying providers directly | | OpenSEO hosted | The same product direction without maintaining its infrastructure | Subscription, usage accounting, service availability, less deployment control | | Ahrefs | Established competitor and backlink research workflows backed by its data collection | A different dataset, product model, and subscription to evaluate | | Semrush | A broad marketing and SEO suite with many integrated workflows | Scope and cost can exceed what a small project needs | | Screaming Frog SEO Spider | Detailed technical crawling from a desktop tool | It doesn't replace the whole hosted research and agent workspace | | Search Console | Direct evidence about your own Google Search appearance | Authorized properties rather than competitor-market research | | Direct DataForSEO integration | Full control of the data requests and a narrow custom tool | You build storage, interface, access control, reporting, and recovery | [Ahrefs Site Explorer](https://ahrefs.com/site-explorer) combines competitor research with its own collection and product conventions. Compare specific queries and domains before assuming one dataset is a drop-in replacement for another. [Semrush](https://www.semrush.com/features/) covers a broader set of marketing workflows. Its value may come from capabilities outside the small slice you need today. Conversely, paying for that breadth makes little sense if you only research a few keywords each month. [Screaming Frog](https://www.screamingfrog.co.uk/seo-spider/) is particularly relevant when the central task is a detailed technical crawl. It runs on macOS, Windows, and Linux, with a mature set of crawl controls. I would compare actual audit requirements rather than expect a web app and a desktop crawler to feel interchangeable. [Search Console](https://developers.google.com/search/docs/monitor-debug/search-console-start) is where I would begin for an existing site. It tells me about the site's own search appearance. I would add market data when I have a question Search Console cannot answer. Going directly to DataForSEO is reasonable for a narrow internal tool. If I need one recurring report, a small integration may be enough. If I also need projects, authentication, agents, report viewing, tracking history, and an interface for other people, OpenSEO gives me code for a much larger part of that job. I wouldn't choose solely from the monthly sticker price. Provider coverage, the number of fresh operations, team needs, maintenance time, and recovery requirements can change the comparison substantially. ## What I like about the design The application has a useful separation between product workflows and provider plumbing. People work with projects and research questions, while the server translates those requests into external operations. The audit split is also sensible. The main application doesn't need to carry every large crawl payload in the same execution environment as ordinary dashboard requests. Durable state has explicit owners. A SAM session has its own object. An audit has a temporary scratchpad. Relational records preserve the results users come back to. The report sandbox deserves attention too. An agent-generated HTML document is treated as untrusted even though the user requested it. That is a more realistic boundary than assuming generated output is safe because it came from a helpful workflow. For developers, the database abstraction is another useful example. Supporting two SQL backends means handling transactions and connection lifetimes, not merely renaming a driver in a configuration file. ## What I would be careful with The phrase “self-hosted SEO” can suggest more independence than the architecture provides. You own the app, but market data and optional model features still depend on external services. The Cloudflare dependency is substantial. Docker makes local use accessible; it doesn't turn the app into a conventional portable Node service. A migration to another runtime would involve workflow execution, durable state, bindings, and storage semantics. Authentication also changes the meaning of a deployment. Local no-auth mode, an Access-protected shared workspace, and hosted organizations are distinct models. I would make that choice before inviting other people. The agent surfaces require careful expectations. External MCP clients, SAM, skill instructions, and stored reports don't have identical capabilities. A workflow working in one client doesn't prove every other surface can execute it. Finally, SEO numbers need interpretation. Provider estimates, first-party search reports, crawl findings, and AI-answer samples describe different things. Putting them in one dashboard is useful as long as their origins remain clear. ## How I would use OpenSEO I would start with one small project, probably a tool such as Waiting Lists, and a few explicit questions. First, what searches already bring people to the site? I would use Search Console for that. Then I would inspect a small number of comparable products and search results. I would look for questions I can answer from real product knowledge, rather than ask an agent to produce a long publishing calendar from search volume alone. I would save useful keywords with their market settings and keep the project context factual. An external agent could turn that research into a report with links to the evidence and a short list of proposed actions. After reviewing it, I would make one change: improve a confusing page, fix an indexing issue, or write something a user needs. A small tracking set could help me watch the relevant searches afterward. AI visibility would be a separate experiment with a stable prompt list. I would read the collected answers and inspect citations before treating any aggregate movement as useful evidence. I would try Docker first, keeping it local. If the workflow became useful across devices or for a trusted team, I would evaluate the Cloudflare deployment and test its backups before depending on scheduled work. For a single technical crawl, I might still use a desktop crawler. For a quick check on this blog, Search Console may already answer the question. OpenSEO becomes more useful when I want the research, history, project context, and agent operations to stay together over time.