Skip to main content

Metrics

Fireraven Metrics provide a system for evaluating your Large Language Model's (LLM) output. These metrics are defined concepts that Fireraven uses to analyze specific aspects of your LLM's performance. While metrics are created at the organizational level and can be associated with individual chatbots, it's important to note that not all metrics visible in the dashboard's Metrics tab are necessarily applied to every chatbot. To jumpstart your monitoring process, Fireraven offers a variety of pre-configured default metrics out of the box.

Create a metric​

To create a new metric, start by going to the Metrics tab in the left side menu and clicking on the New metric button.

  • Metric Name: Provide a clear and descriptive name that reflects the purpose of the metric.
  • Description: Write a complete and precise description that someone unfamiliar with your context could understand.
  • Criticality Level: Select the criticality level from the dropdown menu, indicating the importance of the metric (ranging from None to Critical).
  • Threshold: The detection threshold of your metric over which an issue is considered detected. For metrics you want to detect, place the green zone on the right (low detection means it is an issue). For metrics you want to avoid, place the green zone on the left (high detection means it is an issue).

Once you're satisfied with all the details you've entered, simply press the Create button, and your new metric will be ready to associate with a chatbot!

Edit a metric​

To edit a metric, go to the Metrics tab in your dashboard. Find the metric you want to change, then hover over it and click the pencil icon that appears on the right side of the table. This will let you modify any of the details you originally entered for that metric.

Default metrics

Default metrics cannot be modified at this time, and their criticality level is set to Medium. However, we plan to allow editing of default criticalities in the future.

Delete a metric​

To delete a metric, go to the Metrics tab in your dashboard. Find the metric you want to remove, hover over it, and click the trash can icon on the right side of the table. You will be prompted to confirm the deletion before it's permanently removed.

danger

If the metric has never been used, it will be permanently deleted. However, if the metric has been used, it won't be deleted; instead, it will be archived so you can still view its usage in the monitoring dashboard. Please note that archived metrics cannot be restored.

Use a metric​

To use a metric with a chatbot, start by going to the chatbot's configuration panel. In the General tab, look for the section called Metrics. Here, you can either search for specific metrics or scroll through the list of available ones. To enable a metric, simply select it from the list, and a checkmark will appear next to it. You can see all the selected metrics listed under the search bar. If you want to remove a metric, just click on the x icon next to it.