Unlike the other products, LLM evaluation is less robust. Accuracy, precision, recall do not have the same value and connection with a use case that they do in a classification-based task, where they can be assigned to a context.
Something like a confusion matrix, is only as relevant as a similary matrix it represents.
- Functionality
- Cost-Benefit Analysis
- Low cost, low flexibility: Benchmarks Exact Matching Similarity
- Med cost, med flexibility: Fuctional Correctness model based Evaluation
- ROI
- Scalability and Integration
- User Experience