![\"A](\"https:\/\/braindevs.net\/blog\/\/wp-content\/uploads\/2023\/02\/AdobeStock_131794461-300x162.jpeg\")
<\/a>re great, and you should check them both out. (I’m on those sites several times a week.)<\/p>\nBut, we’ve heard A LOT about AI in recent weeks. Can we rely on newer websites to help us out?<\/p>\n
I spent an hour or so on Monday playing around on two sites; here’s what I found.<\/p>\n
The Inevitable ChatGPT<\/h2>\n
I won’t rehash all the Twitter agita on this subject, but gosh ChatGPT has gotten lots of attention. (Twitter tells me it’s either the savior of education, or the end of education, a deranged stalker of New York Times reporters, or a nothingburger.)<\/p>\n
Can it help us explore research into education<\/em>?<\/p>\nTo answer this question, I asked Chat GPT to describe “schema theory.” Here are the first two paragraphs of its answer:<\/p>\n
“Schema theory” is a cognitive psychology theory that describes how people understand, organize, and store information about the world. The theory suggests that people use mental frameworks or “schemas” to categorize information and simplify their understanding of the world around them.<\/p>\n
A schema can be thought of as a mental structure that represents knowledge about a particular category of information, such as objects, events, or social interactions. For example, a person might have a schema for a “restaurant” that includes information about the typical layout, menu items, and social norms associated with dining out.<\/p><\/blockquote>\n
So far, so good. (The restaurant example of schema is quite common.)<\/p>\n
I asked it to dig a bit deeper, and to identify problems or limitations with schema theory. Again, I got a solid set of answers. For instance, a limitation:<\/p>\n
Overgeneralization: Schemas can sometimes lead to overgeneralization, causing people to perceive things in a stereotypical or biased manner. For example, people with a schema of a “criminal” may unconsciously associate criminal behavior with certain groups.<\/p><\/blockquote>\n
When I asked it for research supporting schema theory, its first three answers were several decades old, so I asked it for research from the last 20 years.<\/p>\n
Of the three it listed, this one looked most promising:<\/p>\n
Lee, J. J., & Anderson, N. (2015). The role of schema in the interpretation of ambiguous scenes. Memory & Cognition, 43(8), 1186-1197.<\/p>\n
But, strangely, when I tried to find that study on Google Scholar, it didn’t seem to exist<\/em>.<\/p>\nStranger still, when I went to the website for the journal\u00a0Memory and Cognition<\/em>, it also doesn’t exist. (To be clear: the\u00a0journal<\/em> exists; the article doesn’t exist in it.)<\/p>\nYou’re not going to believe this, but it seems that ChatGPT simply MADE UP a research study supporting schema theory.<\/p>\n
The Chat Gets Awkward<\/h2>\n
I pointed out to ChatGPT that the source doesn’t exist, and it responded:<\/p>\n
I apologize, it seems I made a mistake. The study by Lee and Anderson (2015) that I mentioned in my previous answer does not seem to exist. As a language model, I strive to provide accurate and up-to-date information, but sometimes I may make errors or misunderstand the information given to me. I appreciate your feedback and will take this into account in my future responses.<\/p><\/blockquote>\n
Okay, I’m not happy about this. But it did fess up to its mistake, and promise to do better.<\/p>\n
So, the conversation continued:<\/p>\n
\n
\n
\n
Me<\/strong>: Can you identify recent studies — within the last 20 years — that study the cognitive psychology of schema theory specifically for students learning academic topics?<\/p>\nChatGPT<\/strong>: Yes, there have been recent studies that have investigated the role of schemas in students’ learning of academic topics.* Here are a few examples:<\/p>\nKail, R. (2017). Schema theory and its implications for the design of learning environments. Review of Educational Research, 87(1), 93-135.<\/p>\n
…<\/p>\n
Well, that first one looks promising. But, I don’t want to be fooled twice, so I asked the obvious question:<\/p>\n
\n
\n
\n
Me<\/strong>: Does the Kail 2017 study really exist?<\/p>\nChatGPT<\/strong>: Yes, the Kail (2017) study does exist. It was published in the Review of Educational Research, a highly respected peer-reviewed journal in the field of education.<\/p>\nReader, the Kail study\u00a0does NOT exist<\/em>! ChatGPT just MADE UP A SECOND STUDY. And, having made it up, it then stuck by its (untrue) story.<\/p>\nI won’t go through all the iterations of this journey in detail.<\/p>\n
But, yes, I asked about a third study. Yes, ChatGPT assured me it exists. No, it really doesn’t.<\/p>\n
So, at this point, I’m kinda done. Any artificial intelligence that simply conjures sources out of thin air…I myself won’t be relying on it soon.<\/p>\n
If only there were another option…<\/p>\n
Another Option!<\/h2>\n
At about this time, my friend Rob McEntarffer asked me if I’d tried Elicit.org. (Important note: bad things might happen if you type “illicit” instead of “elicit.” I haven’t checked, but: be careful out there.)<\/p>\n
Rob is wise in the ways of MBE, and so I tried it.<\/p>\n
At this point, I’m as hopeful about Elicit.org as I am discouraged about ChatCPT.<\/p>\n
Elicit asks users to frame fairly specific questions<\/em>. It then looks for study abstracts that seem relevant to those questions, and reports back.<\/p>\nSo, I asked:<\/p>\n
“Can schema theory improve classroom instruction?”<\/p><\/blockquote>\n
I got a list of seven studies. All seven sounded exactly on topic. And — here’s some surprisingly good news — at least four of the studies exist<\/em>!<\/p>\nI know because I downloaded them and printed them out. They are, in fact, in my “read this right now” pile.<\/p>\n
Now, Elicit has a somewhat easier job that ChatGPT, because it’s answering narrow<\/em> questions about research studies, not broad<\/em> questions about ANYTHING.<\/p>\nBut, I found it easy to use and — at least on my first test drive — reliable and helpful.<\/p>\n
TL;DR<\/h2>\n
If you’re looking for online sources to find and evaluate research-based claims:<\/p>\n
Add Elicit.org to scite.ai and connectedpapers.com as useful research resources.<\/p>\n
Until it stops making stuff up, avoid ChatGPT.<\/p>\n<\/div>\n<\/div>\n
\n* Notice, by the way, that ChatGPT got the possesive (“students’ learning”) correct in this answer. That’s an unusual rule — “use the possessive case before gerunds” — and a difficult apostrophe: plural possessive. So, it’s not great with sources, but it knows from grammar and punctuation!<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
If we’re going to make “research-based” claims about education — and we are! — then we better have the research goods. So, how can we at Learning and the Brain — and you as teachers and administrators — find and evaluate research? For a few years now, I’ve relied on two great sources. Scite.ai helps […]<\/p>\n","protected":false},"author":18,"featured_media":6978,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-6973","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-lb-blog"],"_links":{"self":[{"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/posts\/6973","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/comments?post=6973"}],"version-history":[{"count":9,"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/posts\/6973\/revisions"}],"predecessor-version":[{"id":6986,"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/posts\/6973\/revisions\/6986"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/media\/6978"}],"wp:attachment":[{"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/media?parent=6973"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/categories?post=6973"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.braindevs.net\/blog\/wp-json\/wp\/v2\/tags?post=6973"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}