Automated Testing for LLMOps - Syllabus

• Introduction
  • Overview of LLM-based testing and how it differs from traditional software testing
  • Implementing rules-based testing to assess LLM applications
• Introduction to Continuous Integration (CI)
  • Understanding the basics of CI in LLMOps
  • Using CI workflows for systematic LLM application evaluation
• Overview of Automated Evals
  • Building model-graded evaluations to test LLM applications
  • Implementing a robust evaluation LLM for testing
• Automating Model-Graded Evals
  • Automating model-graded and rules-based evaluations
  • Integrating with CI tools like CircleCI
• Comprehensive Testing Framework
  • Writing robust LLM evaluations to cover issues like hallucinations and data drift
  • Implementing safety checks for harmful or offensive output
• Optional: Exploring the CircleCI config file
  • Detailed exploration of the CircleCI configuration file
  • Best practices for optimizing CI workflows