LLMOps Engineer London
Managing the full LLM lifecycle from experiment to production in regulated financial environments
As Associate Director of Applied AI/ML at Fitch Ratings, I own the full LLMOps lifecycle for production AI systems serving credit ratings analytics. LLMOps in a regulated financial environment demands more than fast iteration: it requires rigorous evaluation, auditability, cost control, and reliability at scale.
LLMOps vs MLOps
LLMOps extends classical MLOps with the specific challenges of large language models: non-deterministic outputs, prompt sensitivity, context window management, hallucination risk, and the cost of inference at scale. The discipline covers everything from foundation model selection through to production monitoring.
Foundation Model Infrastructure
AWS Bedrock
Primary platform for multi-model access at Fitch Ratings. Bedrock provides enterprise-grade compliance, VPC isolation, and access to models from Anthropic, Amazon, and others without data leaving the cloud boundary.
- Model selection and benchmarking across Bedrock model catalog
- Guardrails for content filtering and PII protection
- Inference profiles for cost optimization
- Model invocation logging for auditability
Azure AI Foundry
Used for Azure-hosted model deployment, prompt flow management, and integration with Azure AI Search for RAG pipelines.
- Prompt flow for prompt versioning and evaluation
- Model deployment and endpoint management
- Integration with Azure AI Search for hybrid retrieval
OpenAI API
GPT model integration for specific use cases, with structured output, function calling, and fine-tuning where appropriate.
Experiment Tracking and Evaluation
Every prompt change, model swap, or retrieval configuration is evaluated systematically before reaching production:
- Experiment tracking for prompt variants, model versions, and RAG configurations
- A/B evaluation frameworks comparing system versions on representative query sets
- Automated quality benchmarks: answer correctness, faithfulness, citation accuracy
- Human-in-the-loop evaluation for high-stakes outputs
- Regression testing before every production deployment
Prompt Engineering and Management
- Prompt versioning with change history and rollback capability
- Structured prompt templates with typed input/output schemas
- System prompt hardening for instruction following reliability
- Few-shot example management and selection strategies
- Token budget management for context window efficiency
Production Monitoring
- Latency monitoring per model and per endpoint
- Cost tracking and token usage alerting
- Answer quality drift detection over time
- Error rate and fallback monitoring
- Structured logging for all LLM calls for auditability
MLOps Infrastructure
- Model training and fine-tuning pipelines with SageMaker
- Pipeline orchestration with Airflow
- Containerized model serving with Docker and Kubernetes
- Async inference and task queuing with Redis and Celery
- CI/CD for model deployment with GitHub Actions
- Deployment across AWS, Azure, bare metal (Hetzner, Digital Ocean)
Why LLMOps Matters in Financial Services
In credit ratings analytics, a hallucinated answer or a degraded model is not just a product bug, it is a compliance risk. LLMOps discipline is what separates a prototype from a system that analysts can trust and depend on every day.
Connect
Happy to discuss LLMOps architecture, evaluation strategy, or production AI infrastructure. Reach out at [email protected] or connect on LinkedIn.