在 SageMaker AI 上用 WhisperX 实现带说话人标注的转写

Any team working with spoken audio hits the same wall with generic speech-to-text. Think contact-center calls, all-hands meetings, podcasts, depositions, and broadcast media. These workloads need two things that standard transcription gets wrong. First, timestamps land at the utterance level, off by several seconds. Second, there’s no reliable answer to “who said what.” Those gaps make transcripts hard to search, caption, redact, or analyze at scale. A missing speaker label breaks compliance review, and an imprecise timestamp breaks a caption or a redaction. WhisperX closes both gaps. It wraps OpenAI’s Whisper with batched inference, adds wav2vec2 forced alignment for precise per-word timestamps, and adds speaker diarization to label who spoke. These capabilities map directly to real workloads. Contact centers can measure talk time, check script adherence, and run sentiment analysis, while teams turn meetings into searchable notes. Media and e-learning teams generate accurate captions (in SubRip Subtitle (SRT) and Web Video Text Tracks (VTT) format) for large content libraries. Time-sensitive uses get text the moment someone speaks. In regulated fields like healthcare, legal, and finance, speaker-labeled transcripts support audits and legal discovery. The AWS WhisperX Deep Learning Container (DLC) packages all of this into a GPU-ready image. You deploy it to an Amazon SageMaker AI real-time or asynchronous endpoint without building a custom image. In this post, we show how to deploy both endpoint types and when to choose each. We also cover the production details that matter: the GPU AMI pin, scaling, Amazon Simple Storage Service (Amazon S3) setup, and cost controls. This post is part of a multimodal series showcasing specialized AWS DLCs. The series spans three AWS DLCs across four use cases: (1) vLLM-Omni for text-to-speech, (2) vLLM-Omni for image and video, (3) WhisperX for speech-to-text (this post), and (4) llama.cpp. What is WhisperX, and what workloads it unlocks Whisper is a popular open source automatic speech recognition (ASR) model family from OpenAI that transcribes spoken audio into text accurately across many languages. It focuses on high-quality transcription and produces timestamps at the phrase or segment level. WhisperX is an open source project that builds on Whisper and extends it for production workloads. It adds per-word timestamps, speaker labels, and faster transcription, three capabilities that together turn raw audio into structured, analyzable transcripts. Why a purpose-built Deep Learning Container The AWS WhisperX DLC is a maintained, GPU-ready image that already contains Whisper, the alignment models, and the diarization weights, with no Hugging Face token required. It follows the standard Amazon SageMaker AI serving contract, so you deploy it like any other model. Serving contract: The container serves on port 8080 and exposes POST /invocations for inference and GET /ping for health checks. Request format: The endpoint expects multipart/form-data, with the audio as the file part plus optional string fields such as language, diarize, and response_format. Amazon SageMaker AI passes the ContentType header (including the multipart boundary) through to the container unchanged. Output formats: json, verbose_json, srt, and vtt, so the same endpoint feeds analytics pipelines and video editors alike. Choosing between real-time and asynchronous endpoints Amazon SageMaker AI supports both real-time and asynchronous endpoints, so you can serve the same WhisperX DLC through either pattern. The decision usually comes down to clip length and interactivity. For long audio, use the asynchronous endpoint: it’s the recommended path when transcription, alignment, and diarization need more time to run. Reserve the real-time endpoint for short, interactive clips that finish within the Amazon SageMaker AI 60-second response cap. Dimension Real-time endpoint Asynchronous endpoint Best for Short, interactive clips Long audio, high-volume batch Latency Synchronous, must finish config -> model for each endpoint. for name, cfg, model in [ (ENDPOINT_NAME, ENDPOINT_CONFIG_NAME, MODEL_NAME), (ASYNC_ENDPOINT_NAME, ASYNC_ENDPOINT_CONFIG_NAME, ASYNC_MODEL_NAME), ]: sm.delete_endpoint(EndpointName=name) sm.delete_endpoint_config(EndpointConfigName=cfg) sm.delete_model(ModelName=model) Best practices and production considerations Pin the GPU AMI – Always set InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1 on GPU variants. The default host AMI ships drivers that fail to start this CUDA 12.8 image. Scale by instances, not concurrency – Inference is serialized to one request per container. Set MaxConcurrentInvocationsPerInstance=1 on asynchronous endpoints and add instances or containers for throughput. Autoscale asynchronous endpoints to zero – For bursty batch workloads, scale the asynchronous endpoint to zero instances when idle to cut cost, and use Amazon Simple Notification Service (Amazon SNS) completion notifications instead of tight polling. Right-size the GPU, and use instance pools for availability – Use ml.g4dn.xlarge (T4) for cost or ml.g5.2xlarge (A10G) for headroom. To avoid insufficient-capacity errors, list up to five instance types in an Amazon SageMaker AI instance pool. It provisions the highest-priority type first and falls back automatically when capacity is unavailable. Secure the S3 artifacts – Turn on S3 Block Public Access, default SSE-S3 or SSE-KMS encryption, and BucketOwnerEnforced ownership on the asynchronous bucket. Scope the execution role to specific buckets and keys. Handle personally identifiable information (PII) responsibly – Transcripts of calls and meetings might contain sensitive data, so encrypt artifacts, restrict access, and apply redaction downstream using the word-level timestamps. Observe and retry – Monitor with Amazon CloudWatch, alarm on failures, and retry around the cold-start behavior described earlier. For deeper GPU and inference visibility, turn on the Amazon SageMaker AI detailed metrics and Insights dashboard on CloudWatch. Conclusion In this post, we showed how to deploy the AWS WhisperX Deep Learning Container to Amazon SageMaker AI for word-level, speaker-labeled transcription. We used a real-time endpoint for short interactive clips and an asynchronous endpoint for long, high-volume audio. We covered the production details that matter most: the required GPU AMI pin, single-request-per-container scaling, S3 bucket naming, and cost controls. To go further, explore the WhisperX DLC deployment guide, review Amazon SageMaker AI asynchronous inference, and try the accompanying demo notebook against your own audio. You can also find the full working example in the AWS Samples GitHub repository. Acknowledgements The authors thank the AWS Deep Learning Containers and Amazon SageMaker AI teams for their technical review and contributions to the sample. About the authors Ayush Sharma Ayush is a Senior AI Specialist Solutions Architect at AWS, working with ISVs and startups on generative AI. His interests span multi-agent architectures and the design patterns that make autonomous AI systems reliable at scale. He works closely with model deployment, fine-tuning, and cost-efficient inference across the AWS AI/ML stack. He is passionate about turning emerging AI research into practical, real-world systems. Daniel Wirjo Daniel is a Solutions Architect at AWS, focused on frontier AI startups. As a former startup CTO, he enjoys collaborating with founders and engineering leaders to drive growth and innovation on AWS. Outside of work, Daniel enjoys taking walks with a coffee in hand, appreciating nature, and learning new ideas. Dmitry Soldatkin Dmitry is the Worldwide Leader for Specialist Solutions Architecture, SageMaker Inference at AWS. He helps customers design, build, and optimize generative AI and machine learning solutions, with interests in deep learning and deploying machine learning at scale. He has a passion for continuous innovation and using data to drive business outcomes. Yadan Wei Yadan is a Software Development Engineer on the AWS Deep Learning Containers team. He builds containers that package tested framework versions, dependencies, and AWS deployment configuration for Amazon SageMaker AI, Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), and Amazon Elastic Kubernetes Service (Amazon EKS), including the vLLM-Omni DLC used in this post.