Skip to Main Content

Using Generative AI in Academic Study: Evaluate Your Output

A guide to best practices for using Generative AI in your university study

🔬 Evaluate Your Output: How to Assess the Tool and the Output

Being AI literate does not mean you need to understand all the ins and outs of AI. However it does require you to be actively learning about the technologies involved.

It is important to critically assess the output of AI tools, and the tools themselves, considering strengths and limitations, in order to make informed and academically responsible judgements.

The ROBOT Test: Evaluate the Tool

Before you start using any AI tool for studying, the first and most important step is to evaluate the tool itself. The ROBOT Test, developed by a team led by librarians at McGill University, helps you assess the reliability and validity of an AI technology before you trust its output.

Why start here?

If the tool itself isn’t credible or transparent, its answers won’t be either.

This evaluation ensures you’re working with a tool that meets basic standards for accuracy, bias, and ethical use.

 

Once the AI tool passes the ROBOT Test, you can move on to applying the SIFT Test to evaluate the actual content it produces. This two-step process, first assessing the tool, then the output, helps maintain academic integrity and ensures you’re using AI responsibly.

 

Reliability
  • How reliable is the information available about the AI technology?
  • If it’s not produced by the party responsible for the AI, what are the author’s credentials? Bias?
  • If it is produced by the party responsible for the AI, how much information are they making available?
  • Is information only partially available due to trade secrets?
  • How biased is they information that they produce?

 

Objective
  • What is the goal or objective of the use of AI?
  • What is the goal of sharing information about it?
    • To inform?
    • To convince?
    • To find financial support?

 

Bias
  • What could create bias in the AI technology?
  • Are there ethical issues associated with this?
  • Are bias or ethical issues acknowledged?
    • By the source of information?
    • By the party responsible for the AI?
    • By its users?

 

Owner
  • Who is the owner or developer of the AI technology?
  • Who is responsible for it?
    • Is it a private company?
    • The government?
    • A think tank or research group?
  • Who has access to it?
  • Who can use it?

 

Type
  • Which subtype of AI is it?
  • Is the technology theoretical or applied?
  • What kind of information system does it rely on?
  • Does it rely on human intervention?

 

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

To cite in APA: Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test

SIFT: Evaluate the Output

Once you’ve chosen a reliable AI tool, the next step is to evaluate the information it provides using the SIFT Test. This is essential because AI-generated content can include inaccuracies, bias, or incomplete context. Applying SIFT ensures that what you use in your study is credible, well-supported, and academically sound.

Remember to ask yourself:

Are the sources provided real and accurate?

Is it possible to verify the original source and validate the information provided?

Is the output biased?

How current and relevant is the information supplied?