A one-week proof of concept to turn YouTube research into insight

In short

I built a one-week proof of concept for a learning platform whose content researchers were spending 60% of their time trying to stay current. The system watched YouTube channels, retrieved transcripts and used structured AI analysis to turn videos into searchable content insights. The projected outcome was broader coverage, more consistent planning and a much shorter delay between industry developments and new learning material.

Educational material loses value when the subject moves faster than the research behind it. For technical topics especially, a practice that felt new yesterday can quickly become the expected standard.

The learning platform I worked with needed its educational content to remain relevant and reflect current industry practice. Its team wanted to follow developments across the digital space and use them to support learners' progression.

Manual research consumed time without providing enough coverage

The client's existing process put a heavy burden on its researchers, yet still left the team months behind important developments.

• Research absorbed most of their working time: Staying current took 60% of content researchers' time.
• Findings varied between researchers: Two people could draw different conclusions from the same material.
• Coverage was restricted: The team could regularly monitor only about 25 channels.
• New subjects arrived late: In some sectors, content lagged three to six months behind emerging topics, which was particularly problematic for new LLM tooling and models.
• Priorities depended on personal judgement: Researchers had strong editorial taste, but topic selection relied on what felt important because there was no system for assessing developments at greater speed.

Continuing manually would have meant accepting both the workload and the ongoing risk of outdated content.

Trigger.dev kept the one-week build focused on the client’s problem

Time spent operating queues would have been time not spent proving whether automated analysis could help the content team.

On earlier projects, I had spent countless nights wrestling with job queues. Most developers have stories of production queues failing in the early hours because Redis exhausted its memory, or of jobs disappearing where dead-letter handling had not been configured. Processing thousands of items reliably brings operational work that has little to do with the underlying business problem.

I selected Trigger.dev at the outset to avoid repeating that work. It provided the processing foundation while I concentrated on content analysis:

• Queues were already managed: There was no need for us to operate Redis or build job persistence.
• Batching was built in: Hundreds of items could be processed in parallel.
• Concurrency could be controlled: I could limit activity to avoid API rate limits.
• Failures had a clear handling path: Automatic retries and detailed error reporting were included.
• Development stayed focused: My work centred on business logic instead of infrastructure.

The implementation began with one task receiving the channel list and starting a scraping task for each channel through batchTrigger:

export const processYouTubeChannels = task({
  id: "process-youtube-channels",
  maxDuration: 600,
  run: async (payload: ProcessYouTubeChannelsPayload) => {
    // Create batch payloads for each channel
    const channelPayloads = payload.channels.map((channel) => ({
      payload: { youtubeUrl: channel },
      options: {
        queue: {
          name: "youtube-channels",
          concurrency: 5,
        },
      },
    }));

    // Batch trigger channel scraping
    const batchResult = await scrapeYouTubeProfile.batchTrigger(
      channelPayloads
    );

    return {
      success: true,
      tasksTriggered: batchResult.runs.length,
    };
  },
});

Handing the queue operations to Trigger.dev allowed me to spend the sprint on the client's actual requirement. Its team handled the underlying platform well, and the tool proved a strong fit for this kind of workflow.

A fixed one-week test limited the client’s commitment

The client did not need to commission a complete platform before finding out whether the approach was useful. I structured the work as a one-week proof of concept so its team could test the result rigorously and then decide whether to proceed or change direction.

The sprint delivered the central pipeline rather than every surrounding feature:

1. Channel discovery: Watch selected YouTube channels and identify new videos.
2. Video processing: Retrieve each video's transcript and metadata.
3. Content analysis: Apply AI to turn transcripts into structured findings.
4. Insight storage: Organise and retain the processed results.
5. Content retrieval: Filter the processed material using different criteria.

That scope was enough to test the value of automation without pretending to be a finished end-to-end product. It gave the client's team something concrete to assess while keeping the initial risk contained.

Parallel processing turned weeks of viewing into approximately 30 minutes

The system needed to remove the viewing bottleneck if coverage was going to increase. I used batching at two levels so that neither channels nor their videos had to wait in one long sequence:

• At the first level, several channels were processed concurrently.
• At the second level, several videos within each channel were processed concurrently.

This allowed hundreds of videos to be handled at the same time. The proof-of-concept comparison made the potential scale clear:

• Manual research: 200 videos × 30 minutes per video = 100 hours, or 2.5 working weeks.
• The automated system: 200 videos processed in parallel = approximately 30 minutes in total.

The 200x speedup showed how substantially automation could change the client's research process.

Structured AI output made research consistent enough to use

Faster retrieval would not help if every video still produced vague notes. The pipeline therefore converted each transcript into a consistent set of fields that the content team could compare and filter:

• Talking points: The main subjects discussed.
• Category: The primary area, such as technology or business.
• Summary: A concise account of the material.
• Keywords: Important terms and concepts.
• Learnings: Practical insights that users could apply.

My first prompt requested only a summary and keywords. Its answers were too generic and lacked the technical detail the team needed, so I iterated towards a defined output structure instead of continuing with free-form summaries.

The AI SDK's generateObject passed the transcript to GPT-4o, while a Zod schema determined which answers were acceptable:

const response = await generateObject({
  model: openai("gpt-4o"),
  schema: z.object({
    talkingPoints: z.array(z.string()),
    category: z.string(),
    summary: z.string(),
    keywords: z.array(z.string()),
    learnings: z.array(z.string()).nullable(),
  }),
  prompt: `SECRET PROMPTING TECHNIQUES: ${markdown}`,
});

Other engineers often tell me that this sounds simple, and the basic architecture is deliberately simple. The demanding work sits in operating it: coordinating hundreds of parallel jobs, respecting API limits, recovering from errors and handling gigabytes of transcript data.

Each new video moved through discovery, analysis and storage

The pipeline joined channel monitoring, transcript retrieval and structured analysis into one flow. State tracking prevented known videos from being treated as new, while caching reduced unnecessary processing.

Discovery queued only videos the system had not seen

A scheduler examined every channel's feed at six-hour intervals. When it found an unfamiliar video, it placed that video on the processing queue.

Batching and stored state made this manageable across many channels. Recording the last processed video for each channel gave the system a clear boundary between existing material and new work.

Analysis reused cached results before calling the model

Analysis started once the video and its transcript were available. The system checked for an existing result before spending time and money on another model request.

Where a cached answer existed, the model call could be skipped completely. This reduced processing time and API costs for repeated or similar content, while error handling kept the flow dependable.

Planning could start from evidence rather than scattered notes

Under the manual process, a researcher had to discover and watch every video personally. Their individual notes then carried the findings into the content plan.

The automated flow shifted that repetitive work to the system. In the automated process, the planning meeting starts from a dashboard instead of a collection of separate viewing notes.

This gave the client's team a way to examine more material with greater speed and consistency. It also offered a route to reducing the delay between an industry development and the learning content it informed.

Faster research mattered for subjects such as React

React illustrates why broader monitoring had practical value. Its core features, associated libraries and accepted practices change continually, so a researcher working manually could struggle to track all the relevant developments:

• Updates and new capabilities in core React.
• New patterns and conventions adopted by the community.
• Integrations with frameworks such as Next.js and Remix.
• Approaches to state management.
• Techniques for improving performance.
• Developments in server components.

The automated system could follow these subjects as new videos appeared. Content creators could then plan material around current practice rather than inadvertently teach patterns that had already been superseded.

The main technical risks had explicit controls

YouTube’s limits were managed per channel

Strict limits in the YouTube Data API could have stopped the pipeline if every task ran without restraint. I combined three measures to keep processing within those boundaries:

• Channel-level concurrency controls used a concurrency key for each channel.
• Batch processing made more efficient use of API requests.
• Stored processing state prevented repeated work.

// Channel-based concurrency control
options: {
  concurrencyKey: channelId, // Group by channel
  queue: {
    name: "youtube-channels",
    concurrency: 5, // Process up to 5 channels concurrently
  },
}

Long videos were broken into manageable analysis

Some source videos ran for hours and produced very large transcripts. Sending those through as single units would have conflicted with API limits and made focused analysis harder.

The system divided transcripts into chunks and selected the most relevant sections for examination. Caching also ensured that the same material did not need to be processed repeatedly.

Failures could be inspected and resumed

A pipeline handling thousands of videos has to expect some jobs to fail. I used structured error responses, detailed logs and state tracking so problems could be diagnosed and interrupted work could continue.

Trigger.dev removed much of the usual implementation burden here. That mattered in a short proof of concept, where delivery had to be both quick and accurate within a very narrow window.

The architecture was ready for sources beyond YouTube

The initial build addressed YouTube, but the client was not constrained to that source by the underlying design. I used a “Universal Content Adapter” pattern so each new source could enter the same insight pipeline:

interface ContentAdapter<T, U> {
  // Extract content from source-specific format
  extract(source: T): Promise<RawContent>;

  // Normalize to unified content model
  normalize(raw: RawContent): UnifiedContent;

  // Process content through insight pipeline
  process(content: UnifiedContent): Promise<ContentInsights>;

  // Store results in format suitable for source
  store(insights: ContentInsights, metadata: U): Promise<void>;
}

Extraction, normalisation, processing and storage remained separate responsibilities. As a result, new sites, YouTube channels, articles and Substack posts could be added with a couple of lines, while Trigger.dev's tasks and queues handled the processing.

The projected impact was more coverage with less research time

I used the proof of concept to project how the client's content operation could change. These were forecast outcomes rather than results from a completed production deployment:

The intended business change was to return time to the people creating the content. They could concentrate on nuanced, engaging learning experiences instead of devoting most of their working hours to monitoring their subject.

Production still required the surrounding product

The one-week proof of concept demonstrated the potential of automated content intelligence, but it did not complete everything a production service would need. The remaining work was explicit:

1. Build the content dashboard: Create an interface for examining trends and insights.
2. Add further sources: Include LinkedIn, Twitter and technical blogs.
3. Refine insight extraction: Adapt the AI analysis for particular content types.
4. Introduce feedback loops: Let creators rate insights and improve them.
5. Connect content management systems: Smooth the path from an insight to published material.

Automation handled the watching while experts kept editorial control

The proof of concept produced five lessons that shaped my assessment of the approach:

1. Spend build time on the business problem: Trigger.dev allowed me to work on content analysis rather than queue administration.
2. Divide the pipeline into composable tasks: That structure made the system easier to extend and more resilient.
3. Apply concurrency deliberately: Reliable scaling depended on understanding resource constraints and controlling activity where necessary.
4. Require structured analysis: Defined outputs produced more consistent and usable findings than an unrestricted summary.
5. Use shared adapters for expansion: New sources could pass through the same processor, with Trigger.dev providing the task and queue system.

The central judgement was about responsibility rather than technology. The pipeline could monitor and research every day, but it was not intended to replace the platform's editorial expertise.

Its experts would retain the decisions about analysis, curation and teaching. Automation gave them more material and more consistent evidence without taking those decisions away.

The same pattern could also be applied to competitive intelligence, trend analysis and market research.

Frequently asked questions

What did the one-week proof of concept include?

It included channel discovery, transcript and metadata retrieval, structured AI analysis, insight storage and filtered content retrieval. It deliberately left the production dashboard and other surrounding features for later work.

Where does the 200x speedup come from?

It compares 200 videos at 30 minutes each with approximately 30 minutes of parallel processing. The manual side represented 100 hours of work, or 2.5 working weeks.

Did the system replace the client's editorial judgement?

No. It handled channel monitoring, transcript processing and research so the experts could focus on analysis, curation and teaching. Editorial decisions remained with the client's team.

Could the pipeline analyse sources other than YouTube?

Yes. The Universal Content Adapter pattern allowed new sites, channels, articles and Substack posts to enter the same processing pipeline. LinkedIn, Twitter and technical blogs were identified as possible production additions.