Two Weeks In: What We've Learned Building an Autonomous Publication

The Honest Version

Open Signal launched two weeks ago with a premise that sounds either exciting or absurd depending on your priors: an AI-powered intelligence publication, free to everyone, that aims to produce analysis worth reading every day. We wrote a detailed account of the architecture and philosophy behind the project. Now it is time to talk about what actually happened when theory met practice.

Two weeks is not a long time. It is long enough, though, to learn where your assumptions were right and where they were quietly, stubbornly wrong.

The Content Pipeline in Practice

The core bet behind Open Signal is that AI can handle the labor-intensive parts of intelligence publishing — sourcing, synthesis, structured output — while maintaining a quality bar high enough that readers find genuine value. After two weeks, here is where that bet stands.

What Works

Consistent output. The pipeline produces content on schedule. Daily briefings, analysis pieces, structured maps — the volume has been sustainable in a way that would be impossible for a solo human operation. This was the first-order requirement, and it holds up. A publication that cannot publish regularly is not a publication.

Breadth of sourcing. AI is exceptionally good at pulling together information from a wide surface area. A single Deep Signal piece might synthesize earnings data, technical documentation, regulatory filings, and industry commentary into a coherent narrative. Doing this manually takes a skilled analyst hours. The pipeline does it in minutes, and it does not get tired or develop blind spots from fatigue.

Speed of synthesis. When news breaks — a major product announcement, an earnings surprise, a policy shift — the pipeline can produce structured analysis fast. Not breaking-news fast, and not with the depth of a seasoned beat reporter’s contextualization, but fast enough to be useful. For a free publication without a newsroom, this is a meaningful capability.

What Is Still Challenging

Voice consistency. This is the subtlest and most persistent problem. The pipeline can produce text that is fluent, structured, and factually grounded. But voice — the particular way a publication sounds, the rhythms and instincts that make a reader feel they are hearing from a consistent editorial perspective — is harder to maintain. Some pieces read as distinctly Open Signal. Others could have come from any competent summary service. Narrowing that gap is an active area of work.

Repetitive framing. AI systems have favorite phrases. They reach for the same transitional structures, the same analytical frameworks, the same ways of introducing uncertainty. After two weeks of daily output, certain patterns have become visible. “This matters because…” shows up too often. “The implications are significant” is a phrase that says nothing. We are building a set of anti-patterns — specific constructions to flag and rephrase — but this is a game of whack-a-mole. Fix one repetition and another emerges.

Depth versus surface. The pipeline’s default mode is competent synthesis: take the available information, organize it clearly, state the obvious implications. That is valuable, but it is not what separates a good publication from a forgettable one. The best intelligence analysis makes connections the reader did not see coming, challenges assumptions, or reframes a familiar situation in a way that changes understanding. Getting the pipeline to consistently produce that level of insight — not occasionally, but reliably — remains the hardest problem we face.

Technical Lessons

Astro as a Publishing Platform

Astro was the right choice, and after two weeks of active use, the conviction has only grown. A few specifics:

Build speed matters more than you expect. When you are publishing daily, you rebuild the site daily. Astro’s build times are measured in seconds, not minutes. This sounds like a small thing until you consider the alternative — waiting three minutes for a Next.js build every time you want to verify a formatting change in a briefing. Fast builds reduce friction, and reducing friction increases output quality because you actually preview things instead of assuming they are fine.

Content collections are the right abstraction. Astro’s content collections system — typed schemas, frontmatter validation, build-time error checking — has caught multiple issues before they reached production. A missing publishDate, a misspelled tag category, a projectArea value that does not match the enum. These are the kinds of errors that, in a less structured system, would result in broken pages or silent data corruption. The schema catches them at build time. For a publication that aspires to automated content pipelines, this is essential infrastructure.

Static is an advantage, not a limitation. Intelligence content does not need real-time interactivity. It needs to load fast, render cleanly on any device, and be indexable by search engines. Astro’s static-first approach delivers all three without configuration overhead. The site loads in under a second on a cold cache. There is no JavaScript bundle to optimize because there is, for most pages, no JavaScript.

MDX: Flexibility at a Cost

MDX lets us embed components in markdown — interactive tables, callout boxes, structured data visualizations. The flexibility is real. But it introduces a category of bugs that plain Markdown avoids. A malformed JSX expression in an MDX file produces an error message that is, charitably, unhelpful. The debugging experience when something goes wrong is meaningfully worse than with plain Markdown.

The tradeoff is worth it for now. Signal Maps in particular benefit from the ability to embed structured components. But we are watching this carefully. If the content pipeline is going to produce MDX autonomously, the error surface needs to be smaller than it is today.

Vercel Deployment

There is not much to say here, and that is the point. Push to the repository, the site deploys. Automatic HTTPS, edge caching, preview deployments for branches. It works. In two weeks of daily deploys, there has been zero downtime and zero deployment failures. For a project where the goal is to minimize operational overhead, Vercel delivers exactly what it promises.

Editorial Calibration

This is the question we think about most: what does “good enough to publish” mean when AI is the primary author?

The honest answer is that we are still calibrating. The initial quality threshold was set conservatively — only publish content that a human editor would consider solid work from a junior analyst. Not brilliant, not groundbreaking, but factually accurate, clearly structured, and analytically coherent. That bar has held up as a reasonable minimum, but it has also revealed a tension.

A publication that consistently publishes B+ work is useful but not compelling. Readers will consult it but not recommend it. The aspiration is to regularly produce A-level analysis — pieces that make people think differently about a topic, not just inform them about it. The pipeline can hit that bar, but not predictably. Maybe one in four Deep Signals pieces reaches it. The rest are solid and informative but not distinctive.

The calibration process involves reviewing every piece against a set of questions. Does this tell the reader something they could not easily find elsewhere? Does the analysis go beyond restating the consensus view? Is there a specific insight or connection that justifies this piece’s existence? When the answer to all three is yes, the piece is worth publishing. When it is not, the piece either gets reworked or shelved.

We do not have this automated yet. The quality scoring rubric exists on paper but has not been implemented as a gate in the pipeline. That is a Phase 2 priority.

Reader Patterns

Two weeks is too early for statistically meaningful data about reader behavior. But some directional patterns are visible.

Briefings get the most consistent readership. This makes intuitive sense — they are short, structured, and designed for daily consumption. People check in, scan the five items, read deeper on what interests them. The format works.

Deep Signals pieces have higher engagement per reader but lower total readership. Fewer people click through, but those who do spend meaningful time with the content. This is the expected pattern for long-form analysis and it is fine. The briefings bring people in; the deep analysis gives them a reason to come back.

The distinction between synthesis and original analysis matters enormously. Our most-read Deep Signal so far was not the one that most comprehensively covered its topic — it was the one that offered a framing the reader had not encountered before. Synthesis is table stakes. It keeps the lights on. But the pieces that generate the most reader response are the ones that say something new, even if that something is a modest reframing rather than a revelatory insight.

Open Source content has a different audience. The meta-content about the project itself attracts a more technical readership — people interested in AI-powered publishing, automation, and the specific question of whether this kind of project can work. That audience is smaller but highly engaged. They want details. They want honesty about what does not work. That is what we are trying to provide.

What Is Next

The near-term roadmap has three priorities.

GitHub Actions for content automation. Right now, the pipeline requires manual triggering. The next step is scheduling — automated daily briefings generated and committed via GitHub Actions, with a quality check gate before merge. This is the infrastructure that moves Open Signal from “a person using AI to publish” to “a publication that publishes itself with human oversight.” The difference is meaningful even if the output looks similar.

Pagefind for search. As the content library grows, discoverability becomes critical. Pagefind builds a search index at build time with zero runtime JavaScript — a perfect match for Astro’s static-first architecture. This is straightforward to implement and high-value for readers who want to find past analysis on a specific company, technology, or topic.

RSS and newsletter infrastructure. A publication that requires readers to check the site manually is fighting against how people actually consume content. RSS is the minimum viable distribution mechanism. A newsletter — likely weekly, summarizing the best pieces — is the next step after that.

Beyond those, the longer-term work is improving the content pipeline itself. Better prompting strategies for analytical depth. Anti-pattern detection for repetitive framing. Automated fact-checking against source material. None of these are solved problems. All of them are tractable.

Where Does This Actually Sit?

Here is the honest assessment, two weeks in.

Open Signal is currently closer to “interesting experiment” than “genuinely useful publication.” The briefings are useful — they provide a structured, free summary of what is happening in technology and AI that a reader can consume in five minutes. Some of the deep analysis is genuinely good. But the overall consistency is not yet at a level where a reader should rely on Open Signal as a primary information source.

The question is whether the trajectory points toward “genuinely useful.” We think it does, for a few reasons.

First, the technical foundation is solid. The stack is fast, reliable, and designed for automation. There are no scaling bottlenecks in the infrastructure — the constraints are all in content quality, which is where we want them to be.

Second, content quality is improvable in specific, measurable ways. We know the weaknesses — voice consistency, analytical depth, repetitive framing — and we have concrete approaches to address each one. These are not mysterious or intractable problems. They are engineering problems with iterative solutions.

Third, the economics work. A free publication needs negligible operating costs, and Open Signal’s infrastructure costs are currently under ten dollars per month. That is not a typo. Static hosting, build-time processing, no database, no server-side runtime. The cost structure means this project can survive indefinitely without revenue, which means it can afford to be patient about quality.

The experiment is worth continuing. Whether it becomes something genuinely indispensable is an open question — one we intend to answer in public, with the same transparency we have tried to bring to everything else. If it works, everyone benefits. If it does not, at least the documentation of the attempt will be useful to whoever tries next.