The workflow should operate as a closed evaluation flywheel: One terminology adjustment: experimentation belongs mainly in the offline stage. The CI/CD stage should primarily apply repeatable evaluation suites and release gates rather than conduct uncontrolled experiments. 1. Business Goal Objective Define the business process the AI system should improve—not merely the model capability being demonstrated. […]
Part I: Key Evaluation Insights from the AI Engineer YouTube Channel The AI Engineer community has published a wide range of talks, workshops and case studies on LLM evaluation. Although the speakers use different tools and terminology, their recommendations consistently converge on one central idea: Effective evaluations should connect real business failures, reproducible test cases, […]
一、AI Engineer 频道的 Evals 核心内容地图 1. Domain-specific Evals：如何从业务错误开始 最重要的两场是： 这些视频共同强调： 不要先问“用什么指标”，而要先问“我们的系统到底会在哪些真实业务场景中失败”。 Hamel Husain 的方法尤其重要：找到真正懂业务的 Domain Expert，让其对真实输出做 Pass/Fail 判断并写 critique，再从这些人工判断中逐步构建自动化 LLM Judge。(youtube.com) Hamel 与 Shreya 后续把这个过程总结为： 2. 从 Playground 到 Production 主要视频包括： 这里的核心是把 Evals 从一次性 Notebook 变成持续运行的工程系统： Braintrust 的 workshop 明确覆盖 offline eval、online eval、production logging、用户反馈和人工审核；可扩展 Eval pipeline 则强调把数据、实验、评分和生产 traces 连接起来。(youtube.com) 3. Product Evals：PM 和业务人员如何参与 主要视频包括： 这类视频强调，Evals […]
下面是一份可直接用于公司内部分享的英文 Knowledge Share。核心设计是： 每一级 AI 项目都继承前一级的 Evals，同时增加与新复杂度对应的评估层。项目越复杂，不是替换旧指标，而是在共同的 Eval Core 上增加新的 Eval Packs。 From Simple AI Workflows to Multi-Agent Systems A Layered Evaluation Framework for Enterprise AI Projects Suggested duration: 25–30 minutesTarget audience: Product managers, AI engineers, data scientists, platform engineers, domain experts, security teams, and business owners 1. Why Do We Need Different Eval Strategies? […]
yield conn is used in Python’s context manager pattern with the @contextmanager decorator. It’s a special way to manage resources (like database connections) automatically. How it works in your code: In database.py: In main.py: Execution Flow: Why use yield? ✅ Benefits: Feature With yield Without yield Auto-close connection ✅ Yes ❌ Manual Exception safety ✅ […]
Django Overview What is Django? Django is a high-level Python web framework that enables rapid development of secure and maintainable websites. It follows the MTV (Model-Template-View) pattern (similar to MVC). Key Features: Philosophy: “The web framework for perfectionists with deadlines” Django vs Flask Feature Django Flask Type Full-stack framework Micro-framework Philosophy “Batteries included” Minimalist, flexible […]
Create virtual environment Start API app
Object-Oriented Programming is a programming paradigm that organizes code into objects, which combine data (attributes) and behavior (methods). Python supports four main OOP principles: 1. Encapsulation Bundling data and methods that operate on that data within a single unit (class), and restricting direct access to some components. 2. Inheritance Creating new classes based on existing […]