The Analytical AI Handbook

What is Analytical AI? Why does Analytical AI matter? Tasks are typically measurable. You can create a ground-truth dataset using expert annotations that can be validated against for correctness. Other generative AI outputs are not directly measurable, which is why you need to build evals (a special case of analytical AI) to measure them. Tasks are often specific and discriminative, not general and emergent. You use an LLM’s autoregressive reasoning and instruction-following capabilities to make decisions, but reduce “creativity” in favor of consistency. For this reason, the task can often be run on the smallest possible model that’s been evaluated for task accuracy, rather than reaching for the largest, maximally-intelligent model. Because analytical AI typically does not involve a transaction with a user, more latency is tolerated - so batch and other flexible workload processing models are acceptable, often saving tremendously on costs and overall processing time. This is analogous to OLTP vs. OLAP/map-reduce style data processing. Who is this guide for? Data, ML, and analytics teams using LLMs to transform unstructured datasets to structured ones AI engineers and product managers building evals and trying to improve the reliability of their AI products Operations teams looking to scale the expertise of their domain experts via reliable AI decision models Research teams building judges and other verifiable reward functions Why did we write this guide? How to Use This Handbook