Preliminary Conference Agenda

Originality Evaluation Techniques based on the Processing and Understanding of Scientific Papers with AI

With the number of scientific papers is increasing rapidly, how to identify papers with great originality in a timely manner is a significant research problem. The AI technologies (e.g., LLM) present new approaches for processing, understanding, and evaluating scientific papers. In our report, first, we briefly introduce our research foundation, which focuses on utilizing AI technologies to process and understand scientific text. Subsequently, we report our two recent studies. In our first study, we analyze the feasibility of employing LLM to conduct originality evaluation in zero-shot learning. Specifically, we design a novel prompt and build two new evaluation datasets, and test originality evaluation performance on different LLMs. We find that LLM to some extent can serve as the qualified reviewer. However, LLM typically is overly tolerant reviewer, and therefore their evaluation performance is not perfect. To improve LLM’s evaluation performance, in our second study, we build a new rating dataset, and use QLoRA algorithm to fine-tune LLM, by which we can obviously improve their evaluation performance. Specifically, our model can effectively distinguish new papers published in journal with distinct Impact Factors only based on the research content. Finally, we introduce and demonstrate our newly developed Originality Evaluation System, which can generate Originality Score, Originality Type, and Originality Description to evaluate scientific papers.

Bio

Wei Lu is the Dean of the School of Information Management, Wuhan University. His research interests include knowledge organization, data intelligence, AI governance, etc. He has led more than ten research projects such as the National 2030 "New Generation Artificial Intelligence" major project, the major project funded by the National Social Science Foundation of China, and applied for more than 20 national standards, invention patents, and software copyrights. He has also developed software such as question answering robots (RobertAI), the scientific information intelligent processing platform (ScienceAI) and so forth. He has papers published on AAAI, ACL, SIGIR, EMNLP, JASIST, IP&M and Research Policy etc. He received the Excellent Achievement Award in Humanities and Social Sciences from the Ministry of Education, China and the Excellent Achievement Award in Social Sciences from Hubei Province. He also serves as a member of the Education Guidance Committee for Management Science and Engineering of the Ministry of Education, China, Vice Chairman of the China Index Society, Vice Chairman of the China Information Systems Professional Committee (CNAIS), and Executive Director of the China Science and Technology Information Society.