Abstract

Citations within academic literature keep gaining more importance both for the work of scholars and for improving digital libraries related tools and services. We present in this article the preliminary results of an investigation on the characterisations of citations whose objective is to propose a framework for globally enriching citations with explicit information about their nature, role and characteristics. This article focuses on the set of properties we are studying to support the automatic analysis of large corpora of citations. This model is grounded on a literature review also detailed here, and has been submitted to a group of several hundreds of scholars of all disciplines in the form of a survey. The results confirm that these properties are perceived as useful.

Introduction

Citations are fundamental tools for scholars, anytime during their career. Everyone focuses on different aspects and looks for them while working. A PhD student surveying the literature for her thesis exploits citations to find relevant articles; a senior researcher deepening his research exploits citations to continuously find new material; a reviewer reads citations to understand if the citing works are up-to-date and well-connected to others; a professor writing a project proposal uses citations to spot recent works and useful links; and several other examples could be listed here.

A research line about citations that have been studied for long time, and is increasingly re-gaining importance, is about the identification of authors' motivations for citation. Several schemas of what they are often called citation functions have been proposed by the community towards this goal, very heterogeneous in terms of number of options, granularity and, above all, type of classification: some schemas focus on structural features (position in the article, section, etc.), some on semantic aspects (negative or positive citations, rhetorical aspects, etc.); some on objective properties, some others on more subjective ones; moreover characterisations also differ a lot among disciplines.

We present here a review of these schemas trying to identify common traits and suggestions for new models, and the results of a survey we conducted with more than three hundred scholars to investigate which characterisations are really useful for the final users. To the best of our knowledge, this is the first quantitative study on how users perceive qualified citations, with the aim of deriving which aspects are considered more important and why.

This work is part of a larger project on enriching citations, called SCAR - Semantic Colouring of Academic References. In our vision, each citation (both references in bibliographies and incoming citations) is treated as an individual, first-class entity, which can be classified, accessed, filtered and grouped with other references according to different criteria.

This work is in fact a first step to identify a suitable set of properties for characterising citations, that will be used as reference for automatic classification in the SCAR project. Our goal here is also to collect further feedback from the community, so as to incrementally refine our model.

The article is structured as follows. reviews the related works, from which we derived a data model for characterisations presented in detail in . The scholars survey is presented in before drawing some conclusions in .

Characterising citations: a literature review

Nowadays, the impact of authors, the comparative evaluation of scientists and other research evaluations are mostly based on quantitative metrics for citations (e.g. h-index). The validity of these approaches has been often criticised (see for e.g. ) as it does not account for different qualitative aspects of citations. Negative or self-citations should be weighted in a different way compared for example to affirmative or methodological citations. White in however tackles the penalisation of negative citations saying that it is anyway an achievement to have one’s work noticed by others. It is clear that some cases might be controversial but this is not always the case. A famous example reported by is the case of a South Korean research scientist, Hwang Woo-suk, whose faked research gained popularity with its 200 citations, most of which were negative. Negative citations in this last example have a significant weight.

Authors might show ambivalent opinions regarding a cited work (as discussed by ), dissemble their real viewpoint for politeness and stylistic reasons and change their opinion over time.

Citations functions do not differ only in authors' motivations but they depend also on the research domain where the use of citations varies from discipline to discipline (see ) both on motivational and structural level. The question of qualitative bibliometrics is gaining interest in literature and researchers are suggesting different approaches to the problem (see , ).

One of the first step in this direction is the delineation of a citation functions schema which works as a basis for an automatic citation characterisation tool. This is not an easy task considering the different features and aspects that have to be taken into account.

One of the first work on the topic is Garfield 1965 whose list of authors’ motivations for citation behaviour has been widely cited in literature. Garfield is also one of the first authors to envisage a machine able to automatically identify such motivations. In fact, most of the existing citation functions schemas are the result of theoretical studies and manual analysis more than basis for automatic tools.

Existing schemas differ in the number of citation functions they include and consequently in degree of detail. For example, propose four categories of citations (background, fundamental idea, technical basis and comparison) while more than thirty. A detailed and well formed classification model is CiTO ( ) whose 41 citations properties are hierarchically structured and divided into factual and rhetorical (further classified as negative, neutral, positive). CiTO has been already used in different tools and other projects like CiteULike and Data.open.c.uk as well as reference scheme to annotate citations directly via browser or CMS plugins .

Several studies on the citation functions (See for e.g. ) refer to the four-categories schema proposed by Moravcsik and Murugesan : conceptual (theoretical) vs. operational (methodological); evolutionary (build on cited work) vs. juxtapositional (alternative to cited work); confirmative or negational and organic (necessary to understand the current work) vs. perfunctory (not really necessary for the development of the current work). The last pair is interesting because it introduces the concept of information utility of a citation. The perfunctory function has been pursued in different citation functions schemas such as , and .

The question of the utility and importance of a citation compared to others is a recurrent topic in literature. divides his different categories in primary and secondary sources, conceive citation to be central or peripheral, figures background citation and fundamental background ideas and one of citation functions is significant i.e. the cited work is important for the current article.

The relevance of a citation is also related to the difference between constituent and parenthetical citations (See ); does a citation in brackets has the same value of a citation with a syntactic role in the sentence? This is a question it might be interesting to investigate. In the same way, single citations and groups of citations, especially when redundant (in the Moravcsik’s concept of redundant, i.e. several articles each of which makes the same point), should be accurately considered.

Besides the shared comparison axis, existing schemas have other recurrent properties both on semantic and objective level. Since two of the most contemplated semantic properties concern the use of a citation in order to affirm or negate/criticise a cited or citing work. named them confirmative and negational functions, confirmational and negational; for a cited source can be positively or negatively evaluated; put in their fine-grained citation function classification schema a negative function (when the weakness of the cited work is discussed) and a positive function (when the cited work is successful). These two authors' motivations are strictly related to the work of polarity analysis of citations (see ).

Other recurrent semantic features are the idea of supporting and we found it for example in , , (who classifies it as a sub-category of the affirmation property), and and those of extending, see for example and .

On the objective side, two main aspects arise from the majority of the schemas: citations used as background and those which are references to methods and data sources. Among the ones who adopt the background function, distinguish general background from specific background and like other colleagues define the background as historical (e.g. ). Methods and data, seen both as a single ( ) or separated ( ) properties, are the most recurrent functions from Garfield 1965 to Dong 2011 where they are named technical basis.

While some properties like methods and data seem to be highly related to the scientific domain most of the schemas belong to, the other properties are more interdisciplinary and thus can be applied to different domains (see for an overview). Our analysis is further confirmed by Tahamtan’s study of the most used citation functions in literature ( ).

A scholar browsing references, or a list of cited by works, would also be interested in knowing other aspects of the citations, for instance where each citation appears in the article. Sections bring important information about a citation as well as its location in a broader sense (beginning, central, final part). As point out «the citation location might reveal the reasons and decision rules for citing a specific document». Hernández et al. affirm and show in their 2016 survey ( )the existing relationship between the location of a citation and its polarity. Moreover, different locations often mean different relevance. For example note that citations outside the introduction are perceived to be more valuable since this section often contains general background information and the highest number of citations (compared to other sections). These aspects can also help the automatic identification of citation functions (see ) but their validity depends on disciplinary structures which might differ among domains.

Facets for characterising citations

The characterisations discussed so far constitute a valuable background for this work. We continue our past research as part of a new project called SCAR, whose goal is to enrich lists of citations with explicit information about their role, features and impact. The basic idea is to let scholars access enhanced publications in which the bibliography is not a monolithic unit, as it happens today, but a collection of entries that can be shown, filtered and aggregated according to multiple properties. The same capabilities can be added to the lists of incoming citations: these are very common in digital libraries and citation indexes that often show lists of related articles when reading one. Characterising both outgoing and incoming citations is the main goal of our research. We also plan to build interfaces and tools to exploit these characterisations.

Some questions arise naturally: which properties shall we use to characterise citations? How are these properties related to each other? Which information is most interesting for the users?

This section introduces our initial list of properties, derived from the literature review and our past studies in this area.

summarises our data model, organised in seven classes, shown in the first column of the table. Single characterisations are listed in the right column and identified by a label composed of a letter corresponding to the class and a progressive number. These references will be used throughout the article.

The first class is named citation metrics. In fact, one of the most common task citations are used for is to identify the most relevant articles in a research area: the more an article is cited the higher its impact. Thus, despite the very basic nature of this feature, we believe it is helpful to enrich bibliographies with the global citation count of each cited article (m1). This metrics count does not consider for example that an article might be cited many times by the same source, a situation that might indicate that the cited article was very influential for the citing one (See and ). This is captured as (m2).

The considerations about the impact of cited articles can be easily extended to the authors: a list of citations that also include data about each author, in fact, might be used to identify potentially interesting works written by the experts of a given domain (a2). The authors metrics class is completed by hints about self-citations (a1): their identification is often needed to picture the scientific production of an author and for evaluation purposes.

Citations can also be classified according to the publication venue of the cited work (s1) differentiating for instance between workshop, conference and journal articles. Such a characterisation is helpful when evaluating research contributions. Another interesting dimension is the actual contribution of the cited work (s2): citations could be annotated as citing research articles, surveys, in-use reports, and so on. This is helpful when exploring a research topic: the ability to easily spot surveys or application reports makes searches much easier and faster; similarly it is useful to identify full research articles when searching contributions within scientific events. The fact that an article was awarded is a further information that could be exploited by the scholars (s3).

Temporal information about cited or citing articles are also relevant here. Consider, for instance, the task of checking the freshness of references while reviewing an article: this could be done automatically providing users with filtering widgets that work on annotated references and their time-related data (t1).

Citations could also be enriched with information about their textual context. The term 'citation context' has been introduced to indicate the sentence of the paragraph in which an article is cited but different conceptions of its size window have been investigated during the past years (See and ). Showing such context has been proved to be useful for the readers and several platforms already include it (c1) (e.g. CiteSeerX and ResearchGate).

Other contextual information could be provided about the section containing the citation (c2). As previously shown, several studies confirm the significance of the location of citations as an important feature for the authors' motivations identification, see also the Teufel’s Argumentative Zoning theory ( ) and the work of Abu-Jabara ( ) where article section is used as one of the features to identify citation purpose and polarity. In the same work, Abu-Jabara considers also the difference between single and group of citations as an important aspect for citation classification (labelled as c3 in the table).

The last two classes of our list comprise the most challenging characterisations: citation functions and polarity. As seen in a previous section, several citation functions schemas have been proposed through the years which converge in different aspects.In we studied how CiTO ontology is perceived and used by users. The idea of the study came for the observation that the projects which adopt CiTO employ a sub-set of its properties. For example the Link to Link Wordpress plugin allow users to specify ten of its properties, while the Pensoft Publishers enable authors to use only six properties. In fact, the richness of CiTO ontology can be seen also as a hindrance by annotators and a less fine grained functions set seems to be more usable. In our previous experiment, two groups of ten subjects with different backgrounds and skills were asked to annotate citations whose contexts were automatically extracted from the proceedings of the Balisage Conference . The first group was provided with the full list of 41 CiTO properties while the second one performed the same task by using a subset of 10 CiTO properties (selected with a preliminary experiment, see ).

Results show that besides the most generic and neutral functions such as citesForInformation or citesAsRelated both groups of annotators mainly used some specific functions: obtainsBackroundFrom, citesAsDataSource, citesAsReccomandationReading, Credits, usesMethodIn, Critiques and the first group employed only a sub-set of the 41 properties (See for the detailed results). The most used functions of the human annotation experiment and also those selected by other projects are in line with the most common citation features we found in literature and are the ones we included in our citation functions list (f1-f5).

The last class of our model regards polarity of citations. This type of sentiment analysis, meant to classify positive (p1), or negative (p2), or neutral (p3) categories, has been proved to be related to the article sections. Furthermore it can provide a broad pattern of agreement and disagreement among authors (see ). In addition to providing a straightforward view of authors opinions to scholars, polarity can help survey generation systems ( ) and citation-based summarisation systems ( ).

In the next section we will show how our model is perceived by scholars, investigated through an ad-hoc questionnaire. This step was conceived in order to sharpen and finalise our analysis on citation characterisations.

Surveying scholars on characterisations of citations

We designed and scripted a survey in SurveyMonkey. We then invited all members of the Elsevier's CDI Researcher Innovation Community to fill in the survey by e-mailing a direct link. From the 1200+ members of the community, we received 318 completed responses. The CDI Researcher Innovation Community is an in-house, carefully curated panel composed of researchers with wide representation across career phases, disciplines and regions. Elsevier has built this community up over 3 years, it is by invitation only and utilised for both qualitative and quantitative research.

We asked the respondents to our questionnaire to figure themselves as a user of a platform which gives access to full texts articles together with bibliographic references, and additional information on the articles cited by a given one (outgoing citations) as well as on articles citing that one (incoming citations). The full text of the questionnaire is available on-line .

The first part consisted in rating the relevance (from 0 to 5) of 13 characterisations for both incoming and outgoing citations, as well as impact factors of the articles authors, with the possibility each time to suggest other information through a free text box. We did not show all properties covered in previous section merely to keep duration of the test as low as possible. The characterisations submitted are: is extended by, same research problem, supported by or criticised by, use method in, global citation count, data sources, most cited authors, type of publication venue, type of content, self-citations, citation context, in paper citation count, section in which cited. Each of these characterisations can be used for outgoing and incoming citations, and within the questionnaire, both cases were presented each in a different section and a detailed explanation was given for each characterisation.

A final part of the survey was collecting info about the background of the respondents, focusing on the research domain, role, and habits regarding the use of digital scholar libraries.

To compare the different characterisations, we calculated, for each of them, the weighted score of relevance, the score's possible value ranging from 0 (not relevant) to 5 (very high relevance). Given the percentage p(s) of response for each score s, the weighted score S is obtained by summing for each score value s the product p(s) * s. reports the weighted score for all citation characterisations for outgoing and incoming citations.

The values of the weighted score range from 2,61 (section in which cited for outgoing citations) to 3,78 (is extended by for incoming citations). Therefore all characterisations remain around the medium score, i.e 3. The standard deviation for each calculation of the weighted score range from 1 to 1,3 across all the cases. This level of standard deviation accounts for the relatively flat distribution of the ratings of all respondents.

Several remarks can be drawn from the global results:

• Due to the small amplitude and to a standard deviation not negligible, it is not straightforward to establish a significant ranking among the highest scored characterisations, nor to discard the characterisations with the smallest score. This means that all of them can be considered of relative interest for the majority of the respondents.

• Looking at the group of 7 chracterisations whose weighted score is above 3, one can observe that the 4 most highly scored characterisations (both for incoming and outgoing citations) are citation functions and that the only factual features are all related to the global citation count (global citation count and most cited author).

• The highest gap between the weighted score value of incoming and outgoing citations for a given characterisation is found for the citation context (3,21 for incoming, 2.92 for outgoing). This shows that this information is more valuable when given for the citing articles than for the cited ones.

The free text box inviting testers to suggest additional characterisations was skipped or given a "none" or equivalent response by 95% (incoming citations) and by 85% (outgoing citations) of the respondents. Although this does not demonstrate that the above list of 13 citation characterisations is complete, it gives us a pretty good hint in showing that the most wanted characterisations were covered in our listing. We were only suggested two properties to add to our list: year of publication and impact factor of the venue in which cited work was published.

Conclusions

This article presented the results of an investigation of the SCAR project aimed at providing a global framework for citation characterisations. We identified a list of properties to characterise citations and we conducted a survey among scholars in order to strengthen the foundations of our idea. Regarding the survey, the main outcome is that all the characterisations that we proposed are considered interesting. Furthermore, citation functions, which offer an information of a substantial higher level on the motivations behind a citation, clearly got more interest than more factual characterisations that are often already found in currently available tools. This confirms that scholars are willing to see these features coming to the digital libraries they often use, and that our model covers the usage we envision.

There is a lot of way to go for the SCAR project. We are currently working on algorithms to characterise the most wanted citation functions, in order to integrate them in a Semantic Web-based technological stack, so as to produce a large dataset connected to other sources of information. In the long-run we also target a higher level of user engagement in accessing bibliographies and list of citations.

References

1. Zhao, D., Cappello, A. and Johnston, L., (2017), Functions of Uni- and Multi-Citations: Implications for Weighted Citation Analysis, in Journal of Data and Information Science, vol 2(1), pp 51-69.

2. Garzone, M., (1996), Automated Classification of Citations Using Linguistic Semantic Grammars, M.Sc. Thesis, The University of Western Ontario.

3. Frost, C.O., (1979), The Use of Citations in Literary Research: A Preliminary Classification of Citation Functions, in The Library Quarterly: Information, Community, Policy, vol 49(4), pp 399-414.

4. Finney, B., (1979), The Reference Characteristics of Scientific Texts, M.Sc. Thesis, London City University, Centre for Information Science.

5. Cole, S., (1975), The Growth of Scientific Knowledge: Theories of Deviance as a Case Study, in The Idea of Social Structure: Papers in Honour of Robert K. Merton, New York, pp 175-220.

6. Magee, M., (1966), How Research Biochemists Use Information: an Analysis of Use of Information from Cited References, M.Sc. Thesis, Graduate Library School, Univ. of Chicago.

7. Oppenheim, C. and Renn, S.P., (1978), Highly Cited Old Papers and The Reasons Why They Continue to be Cited, in Journal of the American Society for Information Science, vol 29(5), pp 225.

8. Duncan, E.B., Anderson, F.D. and Mc Aleese, R., (1981), Qualified Citation Indexing: its Relevance to Education Technology, in: Proceedings of the 1st Symposium on Information Retrieval in Education Technology, Aberdeen.

9. Qazvinian, V, Radev, D.R., Mohammad, M.S. Dorr, B., Zajic, D., Whidby, M. and Moon, T.,(2013), Generating Extractive Summaries of Scientific Paradigms, in: Journal of Artificial Intelligence Research, vol 46, pp 165-201.

10. Qazvinian, V. and Radev, D.R., (2010), Identifying Non-Explicit Citing Sentences for Citation-Based Summarization, in: Proceedings of the 48th ACL Annual Meeting, Uppsala, Sweden.

11. Ciancarini, P., Di Iorio, A., Nuzzolese, A.G., Peroni, S. and Vitali, F., (2013), Characterising Citations in Scholarly Articles: an Experiment, in Proceedings of the 1st AIC 2013, CEUR Workshop, Aachen, Germany.

12. Ciancarini, P., Di Iorio, A., Nuzzolese, A.G, Peroni, S. and Vitali, F, (2014), Evaluating Citation Functions in CiTO: Cognitive Issues, in Proceedings of the 11th ESWC 2014, Lecture Notes in Computer Science, Berlin, Germany.

13. Kaplan, D., Tokunaga, T. and Teufel, S., (2016), Citation Block Determination Using Textual Coherence, in: Journal of Information Processing, vol 24(3), 540-553.

14. Ritchie, A., Robertson, S. and Teufel, S., (2008), Comparing Citation Contexts for Information Retrieval, in: Proceeding of the 17^th CIKM, Napa Valley, CA.

15. Hou, W.R., Li, M. and Niu, D.K., (2011), Counting Citations in Texts Rather than Reference Lists to Improve the Accuracy of Assessing Scientific Contribution, in: BioEssays, vol 33(10), pp 724-727.

16. Boyack, K.W., Van Eck, N.J., Colavizza, G. and Waltman, L., (2018), Characterising In-Text Citations in Scientific Articles: a Large-Scale Analysis, in: Journal of Informetrics, vol 12(1), pp 59-73.

17. Hernández-Alvarez, M. and Gomez, J.M., (2016), Survey About Citation Context Analysis: Tasks, Techniques, and Resources, in: Natural Language Engineering, vol 22(3), pp 327-349.

18. Tahamtan, I. and Bornmann, L., (2018), Core Elements in the Process of Citing Publications: Conceptual Overview of the Literature, in: Journal of Informetrics, 12(1), pp 203-216.

19. Sula, C.A. and Miller, M., (2014), Citations, Contexts, and Humanistic Discourse: Toward Automatic Extraction and Classification, in: Literary and Linguistic Computing, vol 29(3), pp 452-464.

20. Abu-Jbara, A., Ezra J. and Radev, D., (2013), Purpose and Polarity of Citation: Towards NLP-Based Bibliometrics, in: Proceedings of the 12^th Hlt-Naacl, Atlanta, GA.

21. Tuarob, S., Mitra, P. and Giles, C.L., (2013), A Classification Scheme for Algorithm Citation Function in Scholarly Works, in: Proceedings of the 13^th JCDL, Indianapolis, IN.

22. McCain, K.W. and Turner, K., (1989), Citation Context Analysis and Aging Patterns of Journal Articles in Molecular Genetics, in: Scientometrics, vol 17(1-2), pp 127-163.

23. Harwood, N., (2009), An Interview-Based Study of the Functions of Citations in Academic Writing Across Two Disciplines, in: Journal of Pragmatics, vol 41(3), pp 497-518.

24. Peritz, B.C., (1983), A Classification of Citation Roles for the Social Sciences and Related Fields, in: Scientometrics, vol 5(5), pp 303-312.

25. Hernández-Alvarez, M., Gomez-Soriano, J.M. and Martínez-Barco, P., (2017), Citation Function, Polarity and Influence Classification, in: Natural Language Engineering, vol 23(4), pp 561-588.

26. Spiegel-Rosing, I., (1977), Science Studies: Bibliometric and Content Analysis, in: Social Studies of Science, vol 7(1), pp 97-113.

27. Lipetz, B.A., (1965), Improvement of the Selectivity of Citation Indexes to Science Literature Through Inclusion of Citation Relationship Indicators, in: Journal of the Association for Information Science and Technology, vol 16(2), pp 81-90.

28. Whidby, M., Zajic, D. and Dorr, B., (2011), Citation Handling for Improved Summarization of Scientific Documents, Technical Report.

29. Li, X., He, Y., Meyers, A. and Grishman, R., (2013), Towards Fine-Grained Citation Function Classification, in: Proceedings of Ranlp, Hissar, Bulgaria.

30. Krampen, G., Becker, R., Wahner, U., and Montada, L., (2007), On the Validity of Citation Counting in Science Evaluation: Content Analyses of References and Citations in Psychological Publications, in: Scientometrics, vol 71(2), pp 191-202.

31. Peroni, S. and Shotton, D., (2012), FaBiO and CiTO: Ontologies for Describing Bibliographic Resources and Citations, in: Journal of Web Semantics, vol 17, pp 33-43.

32. Moravcsik, M.J. and Murugesan, P., (1975), Some Results on the Function and Quality of Citations, in: Social Studies of Science, vol 5(1), pp.86-92.

33. Jochim, C. and Schütze, H., (2012), Towards a Generic and Flexible Citation Classifier Based on a Faceted Classification Scheme, in: Proceedings of the 24^th COLING, Bombay, India.

34. Dong, C. and Schäfer, U., (2011), Ensemble-Style Self-training on Citation Classification, in: Proceedings of 5th IJCNLP, Chiang Mai, Thailand.

35. Teufel, S., Siddharthan, A and Tidhar, D., (2006), Automatic Classification of Citation Function, in: Proceedings of the 11^th EMNLP Conference, Sydney, Australia.

36. Garfield, E., (1965), Can Citation Indexing be Automated?, in: Statistical Association Methods for Mechanized Documentation, Symposium Proceedings, vol 269, pp 189-192.

37. Ding, Y., Zhang, G., Chambers, T., Song, M., Wang, X., Zhai, C., (2014), Content-Based Citation Analysis: the Next Generation of Citation Analysis, in: Journal of the Association for Information Science and Technology, vol 65(9), pp 1820-1833.

38. Harwood, N., (2009), An Interview-Based Study of he Functions of Citations in Academic Writing Across Two Disciplines, in: Journal of Pragmatics, vol 41(3), pp 497-518.

39. Brooks, T.A., (1986), Evidence of Complex Citer Motivations, in: Journal of the American Society for Information Science, vol 37(1), pp.34-36.

40. White, H.D., (2004), Citation Analysis and Discourse Analysis Revisited, in: Applied Linguistics, vol 25(1), pp 89-116.

41. Teufel, S., Siddharthan, A. and Batchelor, C., (2009), Towards Discipline-Independent Argumentative Zoning: Evidence from Chemistry and Computational Linguistics, in: Proceedings of the of the 2009 EMNLP, Singapore.

42. Zhu, X., Turney, P., Lemire, D. and Vellino, A.,(2015), Measuring Academic Influence: not all Citations are Equal, in: Journal of the Association for Information Science and Technology, vol 66(2), pp 408-427.