Skip to main content
Fingers On The Pulse Content Intelligence System
Read the full technical deep-dive on our blog →

​
The Problem

In fast-moving industries like growth marketing, MarTech, and AI, yesterday’s breakthrough is tomorrow’s baseline. Content teams face an impossible challenge: staying current across thousands of information sources while actually creating content. The reality for most content teams:
  • Researchers spend 60% of their time just trying to stay current
  • Manual monitoring limits coverage to 25 channels maximum
  • Different researchers extract different insights from the same content
  • Educational content lags 3-6 months behind cutting-edge developments
  • By the time you’ve researched and created content, the landscape has already shifted
Meanwhile, thousands of videos publish daily across YouTube alone. Important announcements—GPT-4.5 launches, new automation tools, shifting best practices—happen in real-time. Content teams needed to “keep their fingers on the pulse” but couldn’t scale their research capacity fast enough.

​
What This Unlocks

We built an automated content intelligence system that transforms how content teams stay current. Here’s what becomes possible:

​
Instant Knowledge at Your Fingertips

When OpenAI drops a new model announcement, the system detects it within minutes, processes the full transcript, extracts key insights, and notifies your team—all within 20 minutes of publication. You can create response content the same day, not months later.

​
Automated Newsletter Curation

Generate curated newsletters automatically by monitoring specific channels or topics. The system continuously tracks content across 800+ sources, extracts the most relevant insights, and packages them into ready-to-publish digests—weekly, daily, or triggered by emerging trends.

​
Competitive Intelligence

Monitor your competitors’ content strategies in real-time. See what topics they’re covering, which formats perform best, when they publish, and what gaps exist in their coverage. Turn their content strategy into your competitive advantage.

​
Trend Detection Before They Peak

Spot emerging topics as they gain momentum, not after they’ve saturated the market. When a new tool, technique, or concept starts appearing across multiple channels, the system flags it immediately—giving you first-mover advantage on content creation.

​
Data-Driven Content Planning

Shift from “what should we write about?” guesswork to “here’s exactly what the industry is discussing right now” data. Every content decision backed by real-time analysis of thousands of hours of industry content. The transformation: Content research drops from 60% of creators’ time to just 15%. Channel coverage expands from 25 to 800+. Content lag shrinks from months to same-day. Processing speed increases 200x.

​
How It Started

The project came through the Growth Engineering Slack community run by Mike Taylor. Fabian reached out with an ambitious vision for his LMS platform - he needed to leverage AI, automation, and advanced scraping to gain a competitive edge in educational content creation. Fabian knew me from my work in the community. I’m known as one of the more technical members who delivers advanced solutions rather than relying on out-of-the-box tools. When he explained what he wanted to build, we were immediately interested. The appeal was multifaceted: it was a technically challenging problem requiring creative solutions, it had direct application to our own YouTube content creation needs, and it gave me the opportunity to demonstrate how AI tools create real competitive advantages. The scope was ambitious and pushed technical boundaries. It was a great challenge - quite a grand, ambitious plan. We were keen from the start to tackle it.

​
Building the Solution

I built an automated content intelligence system that could monitor 800+ YouTube channels simultaneously, process thousands of hours of video content, extract structured insights using AI, and identify emerging trends in real-time. The goal was to shape the zeitgeist by knowing what’s happening as it happens, enabling the creation of educational content that’s always relevant and current. We needed to reduce content research time by 75% while expanding coverage 32x.

​
The Technical Build

I designed the system around a multi-level batch processing pipeline: 
Channel Discovery → Video Processing → Content Analysis → Insight Storage → Trend Analysis
Content Discovery Flow Diagram I built this using Trigger.dev for job orchestration, Hono as the backend framework, OpenAI GPT-4 for content analysis, and the YouTube Data API for content retrieval.

​
Why We Chose Trigger.dev

I could have gone with traditional BullMQ and it would have worked fine. But Trigger.dev stood out for its impressive UI and user experience for managing job queues. It was killing two birds with one stone. The decision came down to practical considerations: Do I build a UI from scratch to handle job queue work? Trigger.dev gave Fabian something he could interact with immediately - run projects, scrapes, crawls, and see what was happening in real-time. One of the huge benefits is their fantastic UI for visualizing job queues. Throughout the debugging process, we could retry jobs and see what was working or failing without any layer of abstraction. Easy to set up with a free tier, plus the option to self-host if needed. I considered alternatives like Inngest (similar feature set with open source options) and Hatchet.run (great for pure work orchestration with more control over worker relationships). BullMQ would have been traditional but required building custom monitoring. Ultimately, Trigger.dev had the perfect mix of user experience and developer experience to get stuff sorted. Thank god I chose Trigger.dev - it was perfect for this. Most developers have horror stories about production queue systems—Redis running out of memory at 3 AM, jobs silently failing, dead-letter queues not configured. Trigger.dev eliminated these concerns entirely: 
export const processYouTubeChannels = task({
  id: "process-youtube-channels",
  maxDuration: 600,
  run: async (payload: ProcessYouTubeChannelsPayload) => {
    // Create batch payloads for each channel
    const channelPayloads = payload.channels.map((channel) => ({
      payload: { youtubeUrl: channel },
      options: {
        queue: {
          name: "youtube-channels",
          concurrency: 5,
        },
      },
    }));

    // Batch trigger channel scraping
    const batchResult = await scrapeYouTubeProfile.batchTrigger(
      channelPayloads
    );

    return {
      success: true,
      tasksTriggered: batchResult.runs.length,
    };
  },
});

​
How It Works

​
Multi-Level Batch Processing

I designed the system to process content at two levels for maximum efficiency. Level one processes multiple channels concurrently. Level two processes multiple videos within each channel concurrently. This approach enables processing hundreds of videos simultaneously. The math is simple: manual approach would be 200 videos × 30 minutes = 100 hours (2.5 work weeks). My system processes 200 videos in parallel in about 30 minutes total. That’s a 200x speedup.

​
Making AI Work with Raw Transcripts

When processing video transcripts, it really depends on the content type and how the speaker addresses the audience. We used the YouTube API to get transcripts directly, which meant we didn’t have speaker identification - just raw transcript data to work with. The challenges were significant. You’re relying on your inputs to be as clean as possible. If the transcript misspelled something or the LLM didn’t understand a concept, it affected the output. Sometimes videos had sponsors, and halfway through they’d start talking about the sponsor. That could leak into the data output, so I had to prompt specifically to handle this. With this volume, cost and speed become critical. Claude Sonnet was out of the question - too expensive. OpenAI’s GPT-4 had smaller, faster models I could leverage. Where LLMs excel is generalizing lots of content and aggregating many pieces into one coherent format. That’s exactly what we needed - understanding what the video was about, what technologies were discussed, whether it was educational or sales content. We ran evals to find the right mix of speed, cost, and quality. But evaluation doesn’t have to be scary or overly scientific. Sometimes you can just stick your finger in the air and work out which way the wind is blowing. Consistently use your product, see what works and what doesn’t, then tweak it. For each video, the system extracts:
  • Talking Points: Key topics discussed
  • Category: Primary content classification
  • Summary: Concise overview
  • Keywords: Relevant terms and concepts
  • Learnings: Actionable insights
const response = await generateObject({
  model: openai("gpt-4o"),
  schema: z.object({
    talkingPoints: z.array(z.string()),
    category: z.string(),
    summary: z.string(),
    keywords: z.array(z.string()),
    learnings: z.array(z.string()).nullable(),
  }),
  prompt: `Analyze transcript: ${markdown}`,
});

​
Content Analysis Flow

Content Analysis Flow Diagram

​
Building for Scale from Day One

When aggregating content across different platforms, you always need a factory pattern that creates a unified format for your database. We were collecting from YouTube and LinkedIn initially - YouTube through their APIs and LinkedIn through Apify, which is a fantastic marketplace for scraping APIs. I knew from the start that if we were going to add another 50 platforms, we’d need a unified way of defining data collection, piping it into a consistent format, and moving it forward. It’s a tried and tested pattern I’ve used time and again. 
interface ContentAdapter<T, U> {
  extract(source: T): Promise<RawContent>;
  normalize(raw: RawContent): UnifiedContent;
  process(content: UnifiedContent): Promise<ContentInsights>;
  store(insights: ContentInsights, metadata: U): Promise<void>;
}
I made several key architectural decisions upfront. We introduced caching so successful results weren’t being called again unnecessarily. The queue system was essential from day one for handling cron jobs that watch channels and immediately process new content. Each platform has its own quirks - rate limits, data formats - but they all flow through the same pipeline. This pattern makes adding new content sources like Twitter, blogs, or podcasts straightforward. All sources flow through the same processing pipeline while maintaining platform-specific optimizations.

​
How We Built It

I love doing these small passion projects. Being the person who both builds and scopes, talking directly to Fabian to understand exactly what’s needed - there’s no conversation chains or multiple stakeholders slowing things down. I delivered a proof of concept in one week, then completed the full production build in two weeks total. The approach was rapid prototyping with creative freedom. We were able to take creative liberties and make technical decisions on the fly, then pitch them to Fabian after building. We knew he would agree because we were on the same page from the start. We were almost a co-founder on this project. This autonomy enabled immediate technical decisions without committee approval, creative problem-solving without bureaucratic overhead, and rapid iteration based on real results. The one-week POC gave Fabian everything he needed: real data testing the automation concept, validation of AI-generated insight quality, ROI determination before full investment, and the vision brought to life immediately.

​
The Results

Before: Cumbersome, desystematised, dirty

Before: Cumbersome, desystematised, dirty

After: Automation utopia with a 200x increase

After: Automation utopia with a 200x increase

The transformation was dramatic. Content research dropped from 60% of creators’ time to just 15%. Channel coverage expanded from 25 YouTube channels to over 800. Content planning shifted from subjective impressions to data-driven decisions based on trends. Most importantly, content lag went from 3-6 months behind cutting edge to same week or even same day. Processing speed increased by 200x - from 1 audit per day per researcher to hundreds processed simultaneously. Here’s what that looks like in practice. When OpenAI launches a new model, the video gets published on YouTube. My system detects it within minutes, processes the transcript, extracts key insights, notifies the content team, and enables educational material creation within 20 minutes total.

​
Solving the Hard Problems

The technical challenges fell into three main categories. YouTube API Rate Limiting: I implemented channel-based concurrency controls, batch processing optimization, state tracking to avoid redundancy, and caching to prevent unnecessary API calls. Processing Long-Form Content: The system uses chunked transcript processing for memory efficiency, extracts relevant sections to focus on key content, and has an intelligent caching system to avoid reprocessing. Ensuring Reliability: I built comprehensive error handling with graceful fallbacks, detailed logging and monitoring through Trigger.dev’s UI, state tracking for resumable processing after failures, and result caching to maintain progress despite interruptions.

​
What We Learned

The scope was tight, the delivery was painless. My expertise from similar projects meant this was bread and butter for me. We really enjoyed doing it, which always makes things easier. Fabian was great to work with - very knowledgeable and technically minded. It was easy to explain concepts and quickly flesh out ideas together. Focus on core problems, not infrastructure. Trigger.dev eliminated weeks of queue setup work. This is the kind of decision that makes or breaks project timelines. Pipeline architectures provide flexibility. Composable tasks make systems resilient. When one part breaks, the rest keeps working. Smart concurrency is crucial. Understanding constraints enables reliable scaling. You can’t just throw more workers at the problem. Structured analysis yields better results. AI needs structure for consistent insights. Raw text in, structured data out. Balance automation with expertise. Systems augment, don’t replace, human judgment. The AI processes content, but humans still decide what matters. If you’re going to build something like this, plan it out from the start. Get someone involved who understands the engineering patterns needed. Be very specific about what data you want to collect - it’s easy to collect everything and then have to refine it down. When you’re scraping, the more properties you collect, the higher the likelihood of things falling apart with nulls and undefined values. Proper scoping really helped this project succeed.

​
Where This Could Go

I would love to see this evolve beyond just extracting insights. We played around with sentiment analysis and bias checking during development. There’s a ton of potential for content teams using this for strategic analysis of their own channels or competitor research. I explored several possibilities that didn’t make it into the initial build: creating content off the back of insights like synopses or summaries, done-for-you newsletters collecting all YouTube content for a channel weekly, sentiment and bias analysis to understand not just what’s being said but how and why, and deep competitor analysis to inform content strategy. If you’re interested in building something like this, please get in touch - we would love to do it. This system demonstrates how automation can transform content research from a bottleneck into a competitive advantage. By processing the firehose of content automatically, teams can focus on what humans do best—creating engaging, nuanced learning experiences—while the system ensures they’re always working with the latest information. The proof of concept validated everything we needed to know: automated content intelligence is technically feasible, the quality of insights meets educational standards, ROI justifies the investment in automation, and the system scales to handle massive content volumes.

​
Frequently Asked Questions

The key is multi-level batch processing. Instead of analyzing videos sequentially (manual approach: 200 videos × 30 minutes = 100 hours), our system processes hundreds of videos in parallel simultaneously. This reduces the total processing time from weeks to under 30 minutes through concurrent execution.
Trigger.dev provided an exceptional UI for managing and monitoring job queues without building custom dashboards. It eliminated the horror stories of Redis memory issues, silent job failures, and complex dead-letter queue configurations. The visual job management and retry capabilities were crucial for debugging the complex batch processing workflows.
The system balances cost, speed, and quality using OpenAI’s GPT-4o models. We handle common issues like sponsor content pollution and transcript inaccuracies through specific prompting. The evaluation approach focuses on consistent dogfooding - continuously using the product to identify what works and iterating based on real-world performance rather than theoretical metrics.
Yes, it’s designed with a Universal Content Adapter pattern from day one. The system already processes LinkedIn content via Apify and can easily be extended to Twitter, blogs, podcasts, or any content platform. Each platform has its own adapter but flows through the same unified processing pipeline.
We implement channel-based concurrency controls, batch processing optimization, and intelligent caching to avoid redundant API calls. The system tracks state to prevent unnecessary requests and uses result caching so successful operations aren’t repeated, maximizing efficiency within API constraints.
For each video, the system extracts talking points (key topics discussed), category classification, concise summaries, relevant keywords and concepts, and actionable learnings. This structured data enables trend analysis and helps educational content creators understand what’s being discussed across their industry in real-time.
The system monitors 800+ channels continuously and can detect new content within minutes of publication. When major announcements happen (like OpenAI launching a new model), the complete cycle from detection to processed insights takes about 20 minutes, enabling same-day or even same-hour content creation responses.
The proof of concept was built in just 1 week, with the full production system completed in 2 weeks total. The rapid development was possible due to direct client communication, creative technical freedom, and leveraging existing expertise with similar content processing systems.
Manual research consumed 60% of content creators’ time, could only monitor 25 channels, resulted in 3-6 month content lag, and produced inconsistent analysis. The automated system reduced research time to 15% of creators’ workload while monitoring 800+ channels with same-day insights and consistent, structured analysis.
The architecture enables automated content creation, done-for-you newsletters summarizing weekly channel content, sentiment and bias analysis, competitive intelligence deep-dives, and strategic analysis tools. The foundation supports expanding beyond content monitoring into active content strategy and creation assistance.

​
Technologies Used

Trigger.dev

Job orchestration

Hono

Backend framework

OpenAI GPT-4

Content analysis

YouTube API

Content retrieval

TypeScript

Type-safe development

Batch Processing

Parallel execution
⌘I