Being AI Literate does not mean you need to understand all the ins and outs of AI. However, AI Literacy does require you to be actively learning about the technologies involved. It is important to critically assess the output of GenAI, considering both its strengths and limitations, in order to make informed judgements, especially when using LLM tools as a responsible university student.
Some of the questions we typically ask ourselves about sources may be more difficult to answer when analysing GenAI, because the process it takes to arrive at answers is not public.
Here are some guidelines to evaluating the information GenAI gives you.
Compare
Have a look around for other reliable and trustworthy sources to verify the AI’s claims. Try to find alternative sources that cover the same topic, or even the original context that a claim came from.
Check for hallucinations
While you can ask for a GenAI tool to cite its sources, it is known to create very convincing fake citations.
It can even create citations that have the names of real researchers who study the topic you've asked about. However, the article named in the citation might not exist or may not be from the journal it cites. These invented citations are referred to as “hallucinations.”
Run a search to confirm these articles actually exist. If you can't find it in either Library Search or Google Scholar, it is most likely an AI hallucination.
Confirm
If the source article exists, refer to the original source article and check the information the generative AI tool claims to have found in it. Remember that AI is not built to cite truthfully and accurately, so it may name an article that does not actually contain the information.
Check currency
Currency (when a document was created, edited, updated, or revised) is an important factor in evaluating any information source. If you need recent information on a world event or a new development in research, GenAI tools will not likely have that information in its dataset.
Quick Tips!
Remember to ask yourself:
- Are the sources provided real and accurate? Is it possible to verify the original source and validate the information provided?
- Is the output biased?
- How current and relevant is the information supplied? Some text-based AI generators work with information available only up to a certain date.
The ROBOT Test
The LibrAIry, a team led by librarians from McGill University, have created The ROBOT Test to help evaluate the legitimacy of a particular AI technology. This tool can help you, when reading about AI applications, to determine the legitimacy of the technology and its output.
Reliability
- How reliable is the information available about the AI technology?
- If it’s not produced by the party responsible for the AI, what are the author’s credentials? Bias?
- If it is produced by the party responsible for the AI, how much information are they making available?
- Is information only partially available due to trade secrets?
- How biased is they information that they produce?
Objective
- What is the goal or objective of the use of AI?
- What is the goal of sharing information about it?
- To inform?
- To convince?
- To find financial support?
Bias
- What could create bias in the AI technology?
- Are there ethical issues associated with this?
- Are bias or ethical issues acknowledged?
- By the source of information?
- By the party responsible for the AI?
- By its users?
Owner
- Who is the owner or developer of the AI technology?
- Who is responsible for it?
- Is it a private company?
- The government?
- A think tank or research group?
- Who has access to it?
- Who can use it?
Type
- Which subtype of AI is it?
- Is the technology theoretical or applied?
- What kind of information system does it rely on?
- Does it rely on human intervention?
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
To cite in APA: Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test