About

We had to thoroughly stress-test our enterprise transcription service, measuring maximum parallel runs, real-time factors (RTF), multi-channel behavior with different diarization modes, and GPU usage on an NVIDIA RTX PRO 6000.

This article is about how we completed around 22 iterations of testing in 1.5 days of elapsed time and <4hrs of manual effort (planning and steering). I’m sure this would have taken at least a week of (semi?)manual planning and implementation in the pre-Claude-Code era.

Here’s exactly how we leveraged Claude to orchestrate the entire testing process.

Target audience

If you’re already using Claude or any coding agent comfortably, you can skim through just the Key choices & Rationale section to get some of the patterns of effective usage that we’re following.

Otherwise, this full blog will be useful to you if:

  • You have solid experience building and testing enterprise systems AND
  • You’re still waiting for your organization to provide a coding agent, OR
  • You’ve just acquired the license and are still fiddling with it, OR
  • You’re not out of touch with code at the moment but are exploring how agents could help you and your team.

Context

We developed and maintain a local transcription implementation for one of our enterprise customers. In addition to transcription, the implementation also handles VAD, alignment, and diarization (so multiple models are involved).

After validating transcription quality, we needed deeper insights into GPU performance on an NVIDIA RTX PRO 6000. Specifically, we wanted clear answers on maximum parallel transcriptions, RTF for video and audio, multi-channel media behavior under different combinations of channel and speaker diarization, and more.

TLDR;

For a complex, one-time benchmarking effort on our enterprise transcription pipeline, we built a Claude Loop (structured prompts + reusable scripts + sub-agents) that let us complete in 1.5 days what would normally take a week or more of manual effort. The results were well-organized and fully verifiable. Here’s the approach.

Details:

Test Plan (Semi-manual)

We prepared structured test plans based on how we wanted to run the tests. Examples include:

  • Parallel transcription of given audio file(s) with 12 runs running concurrently
  • Parallel transcription of the maximum number of media files (pro-rated based on hardware usage from the first test; auto-scaling not applicable)
  • Parallel transcription of multi-channel media with channel diarization, speaker diarization, and no diarization
  • (Plus several more variations)

We identified and stored appropriate test media files for each scenario.

Necessary Scripts (Fully Claude)

To avoid the need for Claude to generate commands from scratch every time and save tokens, we prepared a few reusable scripts that Claude could orchestrate.

For our use case, we needed:

  • A script that takes a list of files, generates signed URLs, and triggers the jobs
  • A script that streams pod logs to the local machine while also capturing nvidia-smi logs in a multi-threaded manner

Execution Plan (Claude + Loop Engineering)

With data and scripts ready, we created a strong initiating prompt to let Claude run the full loop.

Here’s the tentative prompt we used:


## Goal:

The goal is to run each item mentioned in the test suite using the mentioned strategy.

## Test Suite:

<The list of batches that we have to execute transcripts on and the size of the parallel experiments>

## Cluster Details:

<K8s cluster information so Claude can scale pods and troubleshoot if needed>


## Strategies for Running Experiments and Collecting Logs:

- Use a sub-agent to run each experiment.
- Add a summary before and after every experiment with columns: “folder”, “experiment summary”, “status”. Keep the “experiment summary” to one line.
- Use only the provided scripts to initiate experiments and capture logs.
- Before starting, ensure the previous experiments are complete/concluded.
- Before starting, ensure that you have the necessary number of pods running.

## Scripts:

<Links to the scripts with descriptions>
<Curl command to check status>

## Relevant Info:

<Source code of the service for troubleshooting>

Testing (Claude independently executing the plan with occasional interruptions)

We first ran one complete experiment and manually verified the loop, then refined the prompt and fixed a few gaps.

And let it run with occasional steering.

If you see the prompt structuring, this is how the overall execution looked like,

flowchart TD
    A[Test Suite] --> B[Initiating Prompt]
    B --> C[Main Claude<br/>orchestrator]
    C --> D[Per experiment]
    D --> E[Pre-checks:<br/>previous experiment concluded?<br/>required pods running?]
    E --> F[Spawn sub-agent]
    F --> G[Submit jobs]
    F --> H[Collect logs<br/>+ nvidia-smi]
    G --> I[Write to results folder]
    H --> I
    I --> J[Summary before / after]
    J --> K{More experiments?}
    K -->|yes| D
    K -->|no| L[Results Folder<br/>all runs]
    L --> M[Human + separate Claude session<br/>analysis]

Key choices & Rationale:

Sufficient details in the logs:

We had necessary details available in logs sufficient to answer the questions that will be needed. For example the memory profiling, starting and ending of each section of transcription, etc.

Owning the strategy - Decoupling data collection from getting insights:

If you have noticed the Test Suite section of the prompt, you might see that the actual goal/questions to be answered is not mentioned. Rather we asked it to run the jobs and collect the logs. (Remember Sherlock Holmes’s “Data! Data! Data”? It’s always fool-proof to decouple the data collection from analysis)

This process of decoupling the grunt work and getting the answers helps in few ways,

  1. This reduces the probability of Claude hallucinating the answers and decoupling the process of getting the insights as a separate step.
  2. You can do manual verification based on the data collected to cross verify the insights that you get from Claude. Once we had the data, getting insight was an interactive process with a separate claude code session.

Scripts:

Anything that can be done with simple scripts shouldn’t be done with LLMs.

We had scripts prepared for the steps like submitting jobs for a given set of files, log collection from pods, etc. So that Claude would take care of the stuff it is good at, which is dealing with the not-so-simple decision makings and orchestration.

You might ask, “Why not have a script for every step in this experiment?”. Yes that is ideal, but the agent is there to solve the “unexpected scenarios” that the script is not designed for. For example, it took care of dangling pods, prompted me to ensure the VPN and EKS session when it went off, and so on.

So a “good enough script” + “agent” is a great combo for occasional troubleshooting and testing.

Use of sub-agents:

So that each experiment is exclusively handled by the sub-agent and main Claude that orchestrates stays away from the nitty-grittys.

This avoids the problem of context pollution in the main agent.

Summary:

10+ hours in, Claude was fully steering the process on its own. We completed around 22 iterations of the experiment, with only occasional fixes and troubleshooting in between.

What would have taken at least a week in the pre-Claude-Code period was completed in just 1.5 days (including preparation and troubleshooting).

At the end of it, we got answers for all the questions that we started with,

  • How many parallel transcriptions can be executed for a media on NVIDIA RTX PRO 6000?
  • Is there really a difference between the video and audio transcription?
  • What’s the cost of speaker diarization?
  • What factors affect RTF?
  • Do different media formats have effects on the transcription?
  • What’s the performance on growing media(such as mpeg-dash)

AND many more! All verifiable independently with the data at hand!

Forget about development; For the teams maintaining enterprise applications, coding agents are great boosters for testing and troubleshooting if steered wisely!