Evaluations

Evaluations provide insights into your Inferable application over time.

For example, if your application is responsible for fetching order details and returning the result, an evaluation can be be run against the application to gauge the percentage of times that it returns the correct result based on a set of known inputs.

This allows you to evaluate the impact of instruction updates and tool changes on the behavior your applications.

​

Inferable Promptfoo Provider

Inferable provides a Provider for the Promptfoo evaluation framework.

With the Inferable Promptfoo provider, you can:

• Automate evaluations of your Inferable applications
• Compare different instruction variations
• Assess the performance of your templates over time
• Identify and debug issues in your applications
• Mock function results for consistent evaluations

The Inferable Promptfoo provider is available from within the Inferable SDK at file://${__dirname}/../node_modules/inferable/bin/eval/promptfoo.js and can be used within your Promptfoo configuration file.

We are in the process of adding support for running evaluations directly within Inferable. In the meantime, please reach out to us at [email protected] for help getting started with evaluations.

​

Manual Feedback

The Inferable UI also provides a way to record manual feedback via the “Feedback” button.