{"id":1243,"date":"2021-10-22T09:44:12","date_gmt":"2021-10-22T09:44:12","guid":{"rendered":"https:\/\/researcher.life\/blog\/?p=1243"},"modified":"2023-02-21T06:32:44","modified_gmt":"2023-02-21T06:32:44","slug":"do-research-paper-recommendations-work","status":"publish","type":"post","link":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/","title":{"rendered":"Do Research Paper Recommendations Work? [With Research Paper Example]"},"content":{"rendered":"

\"\"<\/p>\r\n\r\n

Most content-based websites contain recommendations. Whether you are using Amazon, Netflix, Spotify, or Google Scholar, most sites provide some kind of link to other recommended content. How effective are these links? And can we compare recommender tools from one site to another? Is it valuable for a publisher to add links with research paper recommendations?<\/p>\r\n

One thing to make clear immediately is that academic research paper recommendation tools work in a very different way to other kinds of recommendation. For an e-commerce site, recommendations are typically calculated by tracking what people bought. If you buy an inkjet printer, the chances are you will want to buy printer ink next, so it makes sense to provide a recommender tool based on what users bought after buying the item searched for.<\/p>\r\n

But this kind of recommendation is of little use for academic searching. Firstly, academics aren\u2019t buying anything; we don\u2019t know if the article links they found were useful or not. You could argue that citations from other articles are the recommendations section of a research paper, and is some measure of success: if several academics cited this article, it must be important. But that isn\u2019t necessarily the criterion of relevance for academics: what a researcher frequently wants to know is if anyone has investigated (say) the effect of aspirin on diabetes patients, whether or not their work has yet been cited. The UNSILO research paper recommendations algorithm does not make any use of citations; it is completely content-based, without any usage tracking.<\/p>\r\n

A simple contrast between recommendations is to compare how the recommender tool behaves when no relevant link is found. A tool such as Netflix is designed to provide the user with something new to watch, even if you search for a title Netflix doesn\u2019t offer (Netflix never returns zero results from a search). In contrast, academics are not interested in finding hits that are not relevant; they want only research paper recommendations that work. Unfortunately, even Google Scholar returns hits if a meaningless combination of terms is added (if you\u2019re looking for research paper recommendations for \u201caspirin metallurgy volcanoes trainers\u201d on Google Scholar, you\u2019ll surely get 30+ results). Despite its title, Google Scholar is fundamentally a string-matching tool just like its bigger parent, and it isn\u2019t difficult to see its limitations.<\/p>\r\n

The introduction of AI-based tools has transformed the provision of research paper recommendations, moving from string search to concept-based search. Let\u2019s say I am interested in articles on diabetes and aspirin. However much of an expert I might be, I certainly don\u2019t know how the researchers phrased their title: is it \u201cdiabetes\u201d or \u201cdiabetic\u201d, or is diabetes referred to more elliptically, as \u201cimpaired glucose tolerance\u201d? A concept-based tool such as UNSILO, which is not reliant on a few keywords for each article, can find all the main variants of phrases and ensure a search produces comprehensive research paper recommendations.<\/p>\r\n

\"\"<\/p>\r\n

If we do succeed in providing relevant research paper recommendations, how can we measure its success or otherwise? Major surveys of how academics search for and find content (e.g.,\u00a0How Readers Discover Content<\/a>, a long-term study updated every three years) show that academics use search and discovery very extensively, despite (we may assume) being subject-matter experts in their chosen domain. In other words, checking citations or recommendations section of research papers, listening to presentations, and reading articles still leaves gaps for an academic, gaps that that they fill using search. The details of how they search may vary, but it\u2019s clear that most if not all academics are involved in searching for new content as part of their academic work.<\/p>\r\n

Perhaps the most common metric used by publishers is the click-through rate (CTR), the number of times a user on the page clicks on one of the links provided. Click-through rate here means the number of times people who open a content page such as this click on one of the links on the right. This simple measure enables different research paper recommendation tools to be compared, but also gives an idea of how widely used these tools are. Has anyone measured the use of these links? Yes, a 2014 research paper showed that ratings by humans correlated quite closely with CTR (perhaps not surprising):<\/p>\r\n

\r\n

Ratings in the user study correlated strongly with CTR. This indicates that explicit user satisfaction (ratings) is a good approximation of the acceptance rate of recommendations (CTR), and vice versa.\u00a0 (Beel and Langer, 2014<\/a>)<\/p>\r\n<\/blockquote>\r\n

In other words, researchers only click on research paper recommendation links they judge to be most relevant. Interestingly, the same paper found that providing links via what they termed \u201cinferred ground-truths\u201d (such as citations) are generally flawed and far less valuable for evaluating research paper recommendation systems.<\/p>\r\n

What kind of click-through rates are found in practice? From feedback via publishers, we hear that a typical research paper recommendation system may achieve at best a 6.5% CTR. Although this figure may seem low, anyone familiar with actual usage of websites will know that the statistics for interaction with a site beyond the first search are always very low. The click-through rate for display adverts on a webpage are typically 0.1% \u2013 and even that figure often comprises mainly bots, automated systems that click on all the links on a site on a regular basis.<\/p>\r\n

Can click-through rates be compared? An article in Forbes (Fou, 2020<\/a>) points out that every website will have a different result for CTR, and the results are not comparable \u2013 the time a user spends on a hotel booking site will be much greater than the time taken by an academic to find one scholarly article. So comparisons are only meaningful within the same site, or with other websites that have a similar function.<\/p>\r\n

Another metric of site engagement is page views per session. For instance, an average pages per session of 2 means that each user to the website visited two pages before leaving. While this metric will vary widely from one site to another, it is a good way of identifying trends over time on a single website, or of comparing two websites with a similar function (such as two academic publisher sites). Using a research paper recommendation tool typically results in a clear increase in pages per session. A typical value when using a research paper recommendation tool will be above 2.<\/p>\r\n

Given the differences between sites, it would be invidious to single out specific products, but publishers report to us widely differing results when comparing research paper recommendation tools. There is a very clear difference between recommenders \u2013 variations of 2x or even 3x the usage of one tool compared with another are common. We are happy to provide access to UNSILO research paper recommendation tools for evaluation purposes, and we strongly recommend trialling two or more products in this way, ideally via an A\/B test lasting over at least a month.<\/p>\r\n

One conclusion is very clear: research paper recommendation tools work. Any recommender is good, but concept-based tools seem to provide a good experience for researchers that they use again and again. Recommended links are a vital part of the academic workflow, and they significantly increase engagement. After all, any publisher who misses out on a significant enhancement in site usage and benefit to users is not making the most of their content.<\/p>\r\n\r\n\r\n","protected":false},"excerpt":{"rendered":"

Most content-based websites contain recommendations. Whether you are using Amazon, Netflix, Spotify, or Google Scholar, most sites provide some kind of link to other recommended<\/p>\n","protected":false},"author":21,"featured_media":1244,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_editorskit_title_hidden":false,"_editorskit_reading_time":0,"_editorskit_is_block_options_detached":false,"_editorskit_block_options_position":"{}","footnotes":""},"categories":[488,37],"tags":[120,91,48,31,122],"class_list":["post-1243","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-academic-writing","category-r-discovery","tag-literature-reading","tag-literature-search","tag-research-discovery","tag-research-paper","tag-research-recommendations"],"yoast_head":"\nDo Research Paper Recommendations Work? [With Research Paper Example] | Researcher.Life<\/title>\n<meta name=\"description\" content=\"Research paper recommendations are important in the phase of analysis. Learn about recommendation sections of a research paper with example, and recommender tools in searching relevant literature for your research.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Do Research Paper Recommendations Work? [With Research Paper Example] | Researcher.Life\" \/>\n<meta property=\"og:description\" content=\"Research paper recommendations are important in the phase of analysis. Learn about recommendation sections of a research paper with example, and recommender tools in searching relevant literature for your research.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-10-22T09:44:12+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-21T06:32:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1252\" \/>\n\t<meta property=\"og:image:height\" content=\"798\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Michael Upshall\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Michael Upshall\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/\"},\"author\":{\"name\":\"Michael Upshall\",\"@id\":\"https:\/\/researcher.life\/blog\/#\/schema\/person\/084bf4b1039d411c4e4bd32a400bb2aa\"},\"headline\":\"Do Research Paper Recommendations Work? [With Research Paper Example]\",\"datePublished\":\"2021-10-22T09:44:12+00:00\",\"dateModified\":\"2023-02-21T06:32:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/\"},\"wordCount\":1231,\"image\":{\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png\",\"keywords\":[\"Literature reading\",\"Literature search\",\"Research discovery\",\"Research Paper\",\"Research recommendations\"],\"articleSection\":[\"Academic Writing\",\"R Discovery\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/\",\"url\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/\",\"name\":\"Do Research Paper Recommendations Work? [With Research Paper Example] | Researcher.Life\",\"isPartOf\":{\"@id\":\"https:\/\/researcher.life\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png\",\"datePublished\":\"2021-10-22T09:44:12+00:00\",\"dateModified\":\"2023-02-21T06:32:44+00:00\",\"author\":{\"@id\":\"https:\/\/researcher.life\/blog\/#\/schema\/person\/084bf4b1039d411c4e4bd32a400bb2aa\"},\"description\":\"Research paper recommendations are important in the phase of analysis. Learn about recommendation sections of a research paper with example, and recommender tools in searching relevant literature for your research.\",\"breadcrumb\":{\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage\",\"url\":\"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png\",\"contentUrl\":\"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png\",\"width\":1252,\"height\":798},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/researcher.life\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Academic Writing\",\"item\":\"https:\/\/researcher.life\/blog\/category\/academic-writing\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Do Research Paper Recommendations Work? [With Research Paper Example]\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/researcher.life\/blog\/#website\",\"url\":\"https:\/\/researcher.life\/blog\/\",\"name\":\"\",\"description\":\"Educational resources and simple solutions for your research journey\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/researcher.life\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/researcher.life\/blog\/#\/schema\/person\/084bf4b1039d411c4e4bd32a400bb2aa\",\"name\":\"Michael Upshall\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/researcher.life\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/72eaa2042ca4c26fc2f14adb35ca223fe5a2f9efc8d505f28aa0e3749f025e1c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/72eaa2042ca4c26fc2f14adb35ca223fe5a2f9efc8d505f28aa0e3749f025e1c?s=96&d=mm&r=g\",\"caption\":\"Michael Upshall\"},\"sameAs\":[\"https:\/\/unsilo.ai\/author\/michael-upshall\/\"],\"url\":\"https:\/\/researcher.life\/blog\/article\/author\/michael-upshall\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Do Research Paper Recommendations Work? [With Research Paper Example] | Researcher.Life","description":"Research paper recommendations are important in the phase of analysis. Learn about recommendation sections of a research paper with example, and recommender tools in searching relevant literature for your research.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/","og_locale":"en_US","og_type":"article","og_title":"Do Research Paper Recommendations Work? [With Research Paper Example] | Researcher.Life","og_description":"Research paper recommendations are important in the phase of analysis. Learn about recommendation sections of a research paper with example, and recommender tools in searching relevant literature for your research.","og_url":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/","article_published_time":"2021-10-22T09:44:12+00:00","article_modified_time":"2023-02-21T06:32:44+00:00","og_image":[{"width":1252,"height":798,"url":"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png","type":"image\/png"}],"author":"Michael Upshall","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Michael Upshall","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#article","isPartOf":{"@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/"},"author":{"name":"Michael Upshall","@id":"https:\/\/researcher.life\/blog\/#\/schema\/person\/084bf4b1039d411c4e4bd32a400bb2aa"},"headline":"Do Research Paper Recommendations Work? [With Research Paper Example]","datePublished":"2021-10-22T09:44:12+00:00","dateModified":"2023-02-21T06:32:44+00:00","mainEntityOfPage":{"@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/"},"wordCount":1231,"image":{"@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png","keywords":["Literature reading","Literature search","Research discovery","Research Paper","Research recommendations"],"articleSection":["Academic Writing","R Discovery"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/","url":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/","name":"Do Research Paper Recommendations Work? [With Research Paper Example] | Researcher.Life","isPartOf":{"@id":"https:\/\/researcher.life\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage"},"image":{"@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png","datePublished":"2021-10-22T09:44:12+00:00","dateModified":"2023-02-21T06:32:44+00:00","author":{"@id":"https:\/\/researcher.life\/blog\/#\/schema\/person\/084bf4b1039d411c4e4bd32a400bb2aa"},"description":"Research paper recommendations are important in the phase of analysis. Learn about recommendation sections of a research paper with example, and recommender tools in searching relevant literature for your research.","breadcrumb":{"@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#primaryimage","url":"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png","contentUrl":"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png","width":1252,"height":798},{"@type":"BreadcrumbList","@id":"https:\/\/researcher.life\/blog\/article\/do-research-paper-recommendations-work\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/researcher.life\/blog\/"},{"@type":"ListItem","position":2,"name":"Academic Writing","item":"https:\/\/researcher.life\/blog\/category\/academic-writing\/"},{"@type":"ListItem","position":3,"name":"Do Research Paper Recommendations Work? [With Research Paper Example]"}]},{"@type":"WebSite","@id":"https:\/\/researcher.life\/blog\/#website","url":"https:\/\/researcher.life\/blog\/","name":"","description":"Educational resources and simple solutions for your research journey","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/researcher.life\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/researcher.life\/blog\/#\/schema\/person\/084bf4b1039d411c4e4bd32a400bb2aa","name":"Michael Upshall","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/researcher.life\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/72eaa2042ca4c26fc2f14adb35ca223fe5a2f9efc8d505f28aa0e3749f025e1c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/72eaa2042ca4c26fc2f14adb35ca223fe5a2f9efc8d505f28aa0e3749f025e1c?s=96&d=mm&r=g","caption":"Michael Upshall"},"sameAs":["https:\/\/unsilo.ai\/author\/michael-upshall\/"],"url":"https:\/\/researcher.life\/blog\/article\/author\/michael-upshall\/"}]}},"jetpack_featured_media_url":"https:\/\/blog.researcher.life\/wp-content\/uploads\/2021\/10\/Recommend-Cambridge-2.png","_links":{"self":[{"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/posts\/1243","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/comments?post=1243"}],"version-history":[{"count":6,"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/posts\/1243\/revisions"}],"predecessor-version":[{"id":2093,"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/posts\/1243\/revisions\/2093"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/media\/1244"}],"wp:attachment":[{"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/media?parent=1243"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/categories?post=1243"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/researcher.life\/blog\/wp-json\/wp\/v2\/tags?post=1243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}