Measuring scientific performance of ISI indexed journals in economics: the impact of synchronous and diachronous impact factors

Transcription

1 Qual Quant (2016) 50: DOI /s Measuring scientific performance of ISI indexed journals in economics: the impact of synchronous and diachronous impact factors Vignes Gopal Krishna 1 Rajah Rasiah 2 Kuru Ratnavelu 3 Published online: 18 August 2015 Springer Science+Business Media Dordrecht 2015 Abstract Different diachronous citing units were used to determine if there are differences between rankings of journals measured by synchronous impact factors of first- and late-comer journals. A 5 year impact factor was chosen, with citation lags identified as a complementary method for first comers and alternative measurement for late comers. Selected diachronous indices were considered as alternatives for evaluating the short and long term performance of late comers without citation lags. The results show that the most significant index that promotes scientific fairness and recognition is the citation concentration index. As older journals gathered higher citations, this index enables late-comers to catch up with the citations. Hence, the diachronous impact factor is useful in providing a fairer assessment of the impact of the late-comer journals and could be used as a baseline for ranking ISI journals. Keywords Impact factor Backward and forward approaches First-late comer Citation concentration index Journal Energy Index & Kuru Ratnavelu kuru052001@gmail.com Vignes Gopal Krishna vignesgopal@yahoo.com Rajah Rasiah rajah@um.edu.my Department of, Faculty of and Administration, University of Malaya, Kuala Lumpur, Malaysia Department of Development Studies, Faculty of and Administration, University of Malaya, Kuala Lumpur, Malaysia Institute of Mathematical Sciences, Faculty of Science, University of Malaya, Kuala Lumpur, Malaysia

2 2186 V. G. Krishna et al. 1 Introduction The increasing complexity of scientific knowledge has attracted the deployment of several impact evaluation methods of journals (Merton 1968; Prathap 2012). Scientific journals are classified as meso (intermediary asset) (Glänzel and Moed 2002) that are used by researchers to disseminate their research ideas to others. Arguably, the most authoritative database for classifying and quantifying the importance of journals is the Institute for Scientific Information (ISI), albeit the Web of Knowledge (WOK) database accounts for only % of peer reviewed journals (Monastersky 2005). Pendlebury and Adams (2012) have argued that the term impact does not reflect the scientific quality or excellence that lies within the subjective judgements of humans. Furthermore, as Hemmingson (2002) and Smith (1997) have pointed out, the existing methodology deployed to calculate the current impact factor has led to its manipulation thereby resulting in the distortion of the rankings. Besides, the current impact factors of journals are heavily influenced by the research field. Impact factors tend to be higher for journals that have a broader interest in basic research, but lower for more specific research or clinical research fields (Seglen 1997). Vanclay (2012) had identified major biasness of the classical impact factor through the analysis on the asymmetric nature of its computations. Rousseau (2012) echoed Vanclay s argument and proceeded to present improvements. Moed et al. (2012) also identified and outlined several measures that can serve as complementary to the existing journal impact factor employed by Thomson Reuters. The 2 and 5 years impact factors that are derived now by WOK are only able to capture short term visibility of the journals, and hence, the rankings from these results could easily be manipulated. However, the argument does not end there. It can also be argued that short-term assessments may be important for determining certain configurations of rankings. For example, Della Sala and Grafman (2009) and Swartz (2009) have argued that the limited coverage of the 2 year citation window would favour older journals and penalize the newer ones (e.g. the 2-year impact factor of Estudios de Economia is zero). The provision of a short timeline for new or late comer journals would reduce their potential to build their citation base. First comer journals started gaining recognition through the setting of agenda and standards (Danell et al. 1997) on the publications of the selected subject category from 1980 onwards. Late comer journals refer to ISI indexed journals that have publications from 1990 onwards. Glänzel et al. (2004) used a pragmatic approach to indicate the flexibility of the synchronous impact factor as it does not require large observations in the citations window, but argue that it cannot serve as a substitute to the diachronous approach. Whereas the synchronous impact factor is a backward approach, the diachronous impact factor is a forward approach. The former uses citations of publications of journals in a specified lagged year (Ingwersen et al. 2001; Ingwersen 2012). It is also known as a retrospective approach (Glänzel et al. 2004; Bouabid and Larivière 2013). The Diachronous impact factor (DIF) focuses on the dynamic years of citations window (dynamic movement of citations over time), and takes account of citations in the current and future years. This refers to the prospective approach (Glänzel et al. 2004; Bouabid and Larivière 2013). Glänzel et al. (2004) argues that the diachronous approach is the most appropriate to highlight the characteristics of the citation processes in the real context. Another important variable in studying the impact of publications is life expectancy. The term life expectancy refers to total years in existence of the publication, which is estimated by computing the distance between starting year and threshold year of publications (i.e. 2013). It is for these

3 Measuring scientific performance of ISI indexed journals 2187 reasons we introduce the concepts of first-late comer advantages to explain the rationale behind the synchronous and diachronous approaches. Hence, this paper seeks to identify different levels of impact, and to determine changes in journal rankings that result from synchronous and diachronous impact factors. To achieve these objectives we first verified whether the approaches could be used as an alternative or as a complementary indicator to the ISI impact factor of first and late comer journals. We then examine the strength of association between the impact factors. Finally, we attempt to test the validity of the first-late comer advantages through the DIF. The results are expected to be useful for librarians, universities, researchers and scientific policy makers in identifying an improved ranking of ISI indexed journals. The remainder of the paper are structured into four different sections. Section 2 sets up the methodological framework of the paper. Section 3 analyses the findings. Section 4 presents the conclusions and directions for future research. 2 Methodology and data 2.1 Sources of data The data on journal titles, citations, and impact factors for 2 and 5 years of 302 ISI indexed journals in the subject category of have been extracted from the Journal Citation Reports (JCR) 2013, and the Social Science Citation Index (SSCI) (Web of Science TM Core Collection) databases. The period from 1980 to 2013 was selected for the purpose of computing the life expectancy of the journals and validating the first-late comer advantage. Cluster analysis was used to sort out the selected ISI journals into two main groups based on the birth of publications and the concepts of first-late comer advantages (See Table 1). 2.2 Quantitative techniques This study used the deductive approach to examine changes in journal rankings by comparing the ranking scores obtained from the ISI s impact factor and the DIF. Following Garfield (1955) and Ingwersen et al. (2001): (a) The synchronous impact factor (SIF) ISI impact factor is measured as: SIF t;m ðyþ ¼ Xt i¼1 TCðY; Y iþ, X t i¼1 TPðY iþ; where SIF is the synchronous impact factor, TC(Y, Y - i) the total citations for publications in the lagged fixed years, TP(Y - i) the total publications in the lagged fixed years, t the selected lagged years of publication, m the ISI indexed journals in the subject category of. The SIF s that are used in this study are 2- and 5-year impact factors of the targeted ISI indexed journals. It is used as a benchmark for the purpose of the comparative analysis deployed in the paper to capture the short term impact of citations among scientific journals by considering the citation lags. In this study, Ingwersen et al. (2001) diachronous approach was adapted to measure the impact factor based on non-self citations (NSC):

4 2188 V. G. Krishna et al. Table 1 List of ISI indexed journals based on the two main groups (first and late comers) Group 1 ( ) (94 journals)-first comer journals Acta Oeconomic, American Economic Review, American Journal of Agricultural, American Journal of and Sociology, Annals of Regional Science, Applied, Australian Economic History Review, Australian Economic Papers, Brookings Papers on Economic Activity, Cambridge Journal of, Canadian Journal of, Developing Economies, Eastern European, Economic Development and Cultural Change, Economic Geography, Economic History Review, Economic Inquiry, Economic Journal, Letters, Economic Record, Economy and Society, Economic and Social Review, Econometrica, Economica, Ekonomicky Casopis, European Economic Review, Explorations in Economic History, Food Policy, Futures, History of Political Economy, Hitotsubashi Journal of, International Economic Review, International Labour Review, Journal of Agricultural, Journal of Banking and Finance, Journal of Consumer Affairs, Journal of Development, Journal of Development Studies, Journal of Economic Behavior and Organization, Journal of Economic Dynamics and Control, Journal of Economic Education, Journal of Economic History, Journal of Economic Issues, Journal of Economic Literature, Journal of Economic Theory, Journal of Econometrics, Journal of Environmental and Management, Journal of Finance, Journal of Financial, Journal of Financial and Quantitative analysis, Journal of Human Resources, Journal of Industrial, Journal of International, Journal of Law and, Journal of Macroeconomics, Journal of Mathematical, Journal of Monetary, Journal of Money Credit and Banking, Journal of Policy Modeling, Journal of Political Economy, Journal of Public, Journal of Regional Science, Journal of Risk and Insurance, Journal of Transport and Policy, Journal of Urban, Journal of World Trade, Jahrbucher Fur Nationalokonomie Und Statistik, Kyklos, Land, Mathematical Social Sciences, National Tax Journal, Oxford Bulletin of and Statistics, Oxford Economic Papers-New Series, Politicka Ekonomie, Public Choice, Quarterly Journal of, Regional Science and Urban, Regional Studies, Review of and Statistics, Review of Economic Studies, Revue D Etudes Comparatives Est-Quest, Review of Radical Political, South African Journal of, Scandinavian Journal of, Scottish Journal of Political Economy, Southern Economic Journal, Theory and Decision, Tijdschrift Voor Economische En Sociale Geografie, Transportation Research Part A-Policy and Practice, Transportation Research Part B-Methodological, Trimestre Economico, World Development, and World Economy

5 Measuring scientific performance of ISI indexed journals 2189 Table 1 continued Group 2 ( ) (208 journals)-late comer journals Agricultural, Agricultural -Zemedelska Ekonomika, Agribusiness, American Economic Journal-Applied, American Economic Journal-Economic Policy, American Economic Journal-Macroeconomics, American Economic Journal- Microeconomics, American Law and Review, Amfiteatru Economic, Annals of and Finance, Annual Review of, Annual Review of Financial, Annual Review of Financial, Annual Review of Resource, Applied Letters, Applied Economic Perspectives and Policy, Argumenta Oeconomica, Asia Pacific Journal of Accounting and, Asian Economic Journal, Asian Economic Papers, Asian Economic Policy Review, Asian Journal of Technology Innovation, Asian Pacific Economic Literature, Astin Bulletin, Australian Economic Review, Australian Journal of Agricultural and Resource, Bulletin of Economic Research, Baltic Journal of, BE Journal of Economic Analysis and Policy, BE Journal of Economic Analysis and Policy, BE Journal of Macroeconomics, BE Journal of Theoretical, Cambridge Journal of Regions, Economy and Society, Cepal Review, Cesifo Economic Studies, China Agricultural Economic Review, China Economic Review, China and World Economy, Cliometrica, Computational, Contemporary Economic Policy, Custos e Agronegocio On Line, Defence and Peace, E & M Ekonomie a Management, Ecological, Economic Computation and Economic Cybernetics Studies and Research, Economic Development Quarterly, of Education Review, of Governance, and Human Biology, Econ. Journal Watch, Economic and Labour Relations Review, Economia Mexicana-Nueva Epoca, Economic Policy, Economia Politica, and Politics, Economic Systems, Economic Systems Research, Economic Theory, of Transition, Econometrics Journal, Econometric Reviews, -The Open Access Open Assessment E-Journal, Ekonomska Istrazivanja- Economic Research, Emerging Markets Finance and Trade, Emerging Markets Review, Empirical, Empirica, Environmental and Resource, Estudio de Economia, European Journal of Health, European Journal of the History of Economic Thought, European Journal of Law and, European Journal of Political Economy, European Review of Agricultural, European Review of Economic History, Europe-Asia Studies, Experimental, Federal Reserve Bank of St Louis Review, Feminist, FinanzArchiv, Fiscal Studies, Games and Economic Behavior, Geneva Risk and Insurance Review, German Economic Review, Global Economic Review, Hacienda Publica Espanola-Review of Public, Health, History of Economic Ideas, Iktisat Isletme ve Finans, IMF Economic Review, Industrial and corporate change, Industry and Innovation, Independent Review, Information and Policy, International Environmental Agreements Politics, Law and, International Finance, International Journal of Economic Theory, International Journal of Health Care, Finance, and, International Journal of Transport, International Review of and Finance, International Tax and Public Finance, Investigacion Economica, Inzinerine Ekonomika- Engineering, Journal of African Economies, Journal of Agricultural and Resource, Journal of Agrarian Change, Journal of Applied, Journal of the Asia Pacific Economy, Journal of Australian Political Economy, Journal of Behavioral Finance, Journal of Business and Management, Journal of Competition Law and,

6 2190 V. G. Krishna et al. Table 1 continued Journal of Cultural, Journal of, Journal of Economic Geography, Journal of Economic Growth, Journal of Economic Inequality, Journal of Economic Interaction and Coordination, Journal of and Management Strategy, Journal of Economic Policy Reform, Journal of Economic Surveys, Journal of Empirical Finance, Journal of the European Economic Association, Journal of Evolutionary, Journal of Financial Econometrics, Journal of Financial Stability, Journal of Forest, Journal of Housing, Journal of Human Capital, Journal of Institutional, Journal of International Financial Markets Institutions and Money, Journal of International Trade and Economic Development, Journal of the Japanese and International Economies, Journal of Korea Trade, Journal of Media, Journal of Pension and Finance, Journal of Population, Journal of Productivity Analysis, Journal of Public Economic Theory, Journal of Real Estate Finance and, Journal of Real Estate Research, Journal of Regulatory, Journal of Risk and Uncertainty, Journal of Sports, Journal of Transport Geography, Journal of Common Market Studies, Japanese Economic Review, Japan and the World Economy, Korean Economic Review, Kyklos, Labour, Land, Macroeconomic Dynamics, Manchester School, Marine Resource, Mathematical Finance, Metroeconomica, North American Journal of and Finance, New Political Economy, Open Economies Review, Oxford Review of Economic Policy, Pacific Economic Review, Panoeconomicus, Papers in Regional Science, Pharmacoeconomics, Portuguese Economic Journal, Post-Communist Economies, Post-Soviet Affairs, Prague Economic Papers, Quantitative Marketing and, Quantitative, Quantitative Finance, Real Estate, Louvain Economic Review, Resource and Energy, Review of Derivatives Research, Review of Development, Revista de Economia Aplicada, Review of Economic Design, Review of Economic Dynamics, Review of of the Household, Revue D Economie Politique, Review of Environmental and Policy, Review of Finance, Revista de Historia Economica, Revista de Historia Industrial, Review of Industrial Organization, Review of International, Review of International Organizations, Review of International Political Economy, Review of Network, Review of World, Romanian Journal of Economic Forecasting, South African Journal of Economic and Management Sciences, Series-Journal of the Spanish Economic Association, Small Business, Socio-Economic Review, Spatial Economic Analysis, Studies in Nonlinear Dynamics and Econometrics, Technological and Economic Development of Economy, Theoretical, Tourism, Transformations in Business and, Transport Policy, Transportation Research Part E-Logistics and Transportation Review, Value in Health, Work Employment and Society, World Bank Research Observer, World Trade Review, Zeitschrift fur Wirtschaftsgeographie, and Proceedings of Rijeka, Faculty of

7 Measuring scientific performance of ISI indexed journals 2191 (b) Diachronous impact factor based on non-self citations (DIFNSC) is measured as follows: pþt 1 DIFNSC t;m ðyþ ¼ X NSCðY þ i; YÞ=TPðYÞ; i¼p where DIFNSC is the DIF based on non-self-citations, NSC(Y? i, Y) the non selfcitations for publications in the present and subsequent years, TP(Y) the total publications in the year Y, t the life expectancy of publications within a particular journal, m the ISI indexed journals in the subject category of. Following Lu et al. (2014) journal concentration index, we propose the diachronous impact factor based on citation concentration index (CCI) based on citing articles (instead of journals) as the former was considered as a social capital of journals: (c) Diachronous impact factor based on citation concentration index (DIFCCI) is measured as follows: " pþt 1 DIFCCI t;m ðyþ ¼1 X # CAðY þ i; YÞ=TCðY þ i; YÞ i¼p where DIFCCI is the DIF based on citation concentration index for a particular journal, CA(Y? i, Y) the citing articles for publications in the present and subsequent years, TC(Y? i, Y) the total citations for publications in the present and subsequent years, t the life expectancy of publications within a particular journal, m the ISI indexed journals in the subject category of. Following Prathap (2011) and Glanzel(2008) s Energy index based on the theory of performance and energy analogy [vis viva on the field of kinetics or dynamics (Terall 2004)] were utilized as a baseline to develop JEI based on non self citations: (d) Diachronous impact factor based on journal Energy Index (DIFJEI) is measured as follows: q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 DIFJEI t;m ðyþ ¼ DIFNSC t;m ðyþ : TPðYÞ; where DIFNSC is the DIF based on non self citations, TP(Y) the total publications in the year Y, t the life expectancy of publications within a particular journal, m the ISI indexed journals in the subject category of. The diachronous impact factors used to evaluate the performance of citations among scientific journals based on the forward approach (present and future years). Since it focuses on dynamic years, it can be classified as a dynamic version of impact factor. Nonself citations and citing articles were taken into consideration for its capability of capturing the recognition of the contents in the current journals among authors of unique journals. Unique journals refer to journals in different subject categories. CCI is taken into consideration to measure concentration level of citations received from citing articles. If CCI is close to zero, then, it shows that the current journals deal with zero or lower concentration of citations gained from citing articles and vice versa. According to Terall (2004), the notion of energy is based on the interactional effects of mass and velocity. In the context of this study, Mass represents the quantity of publications where as velocity refers to the quality of publications. The overall formulation of DIFJEI

8 2192 V. G. Krishna et al. Table 2 Characteristics of first and late comer ISI indexed journals, First comer journals a Late comer journals b a First comer journals consists of ISI indexed journals that have life expectancy of 34 years ( ) b Late comer journals consists of ISI indexed journals that have life expectancy of 24 years ( ) or less than 24 years Average impact factor (2 year) Total publications 235, ,869 Median (total citations) 15, Median (self citations) Median (non-self citations) 13, Minimum (publications) Maximum (publications) 11,412 23,735 Median (publications) Median (CCI) Maximum (CCI) Minimum (CCI) Median (citing articles) 10, Maximum (citing articles) 165,736 34,611 Minimum (citing articles) Median (non-self citing articles) 10, Maximum (non-self citing articles) 161,985 32,042 Minimum (non-self citing articles) has the capacity of explaining the metaphysical nature of performance among both first and late comer journals. It shows the movements of citations among the publications over the selected time through the interaction effects of quantity and quality. It was normalized through the squaring method due to the variations of issues and publications within ISI indexed journals. Each of the indices was normalized to serve the variations of citing issues, number of publications and affiliated subject categories. It is necessary for the purpose of comparison based on the rankings of journals. 3 Results and discussion Table 2 summarizes the quantitative descriptions of first and late comer journals in. Among first comers, American Journal of Agricultural has the maximum and Hitotsubashi Journal of has the minimum number of publications. American Economic Review deals with the highest number of citing articles with non self citations. Among first comer journals, Journal of Finance has the highest CCI, Australian Economic Papers has the lowest CCI. Among late comers, Value in Health has the maximum publications and its citations exceeded the first comer journal. Hacienda Publica Espanola-Review of Public has lower publications compared to Hitotsubashi Journal of. Ecological has the highest number of citing articles with highest number of non-self citations. It is very much lower compared to first comer journal. Technological and Economic Development of Economy leads the concentration on citing articles whereas Revista de Historia Industrial, Korean Economic Review, Economia Mexicana-Nueva Epoca, and Custos e Agronegocio On Line have no concentration on citing articles. This shows that most late comer journals concentrate on citations received from few citing articles compared to first comers.

9 Measuring scientific performance of ISI indexed journals 2193 Only five out of the 20 most cited Journals in are sold commercially such as Journal of Financial, Journal of Econometrics, Journal of Economy Theory, Journal of Public, and Journal of Monetary (Bergstrom 2001). Tables 3 and 4 indicate the list of ISI journals in based on affiliated and nonaffiliated subject categories. Journals with affiliated subject categories include in disciplines, such as finance, business, environmental studies, sociology and law whereas journals with non-affiliated subject categories focus only on. Only 42 and 101 first and late comer journals respectively, are linked to affiliated subject categories such as Journal of Finance, Journal of Financial, Small Business, Ecological, and etc. whereas 52 and 107 first and late comer journals respectively are related to such as Journal of International, Journal of Post Keynesian, American Economic Review, Technological and Economic Development of Economy, Economic Systems Research and etc. Also, Bergstrom (2001) made the point that the six most cited ISI journals in, i.e. American Economic Association Journals (American Economic Review, Journal of Economic Perspectives, and Journal of Economic Literature), Econometrica, Journal of Political Economy (JPE), Quarterly Journal of, Journal of Finance, and Journal of Consumer Research are non-profit journals while their average subscriptions fees are about US180 per year. Whereas JPE publishes articles that are solicited by the journal, the remaining five journals subscribe to blind refereeing. While this framework offers authors what seems on paper an objective review process, it can be abused. Given that papers refereed can be easily found through search engines as mimeographs, working papers or proceedings prior to submission, the double blind refereeing procedure in reality often becomes a single blind refereeing process. Therefore, referees under such systems often eventually enjoy an asymmetric advantage over the authors. This problem is particularly acute in social science disciplines, which are heavily dominated by ideological differences. The non-blind review system may be superior so that it allows authors to remove referees who could bias the review process. The selected indices for both groups of journals were not normally distributed due to the pattern of heterogeneity among highly skewed citations (Refer to Table 5 and Appendices). Hence, we deployed the non-parametric approach of correlation test (Spearman rank correlation) (Refer to Table 6; Figs. 1, 2), Mann Whitney U-Wilcoxon W test and Friedman test to test the differences of impact indices between and within the journals. High correlation between 2- and 5-year impact factors was observed for both first and late comer journals. It is due to the fact that they are consistent with their technical definitions (Table 6). Also, % of the correlations were observed between the 2-year impact factor and the selected diachronous indices except for citation concentration index for both groups, though, their technical interpretations are not the same (Table 6). This is because the diachronous impact factors have captured partial citations and publications over 2 years (2011 and 2012) of selected journals. Although there was a highest correlation between the 2- and 5-year impact factors, small changes in journal rankings was still observed among 66 scientific journals in which the significant changes were identified in 53 late comer journals (Tables 7, 8). Impact improvements were observed among 23 late comers compared to 11 first comers, while impact deteriorations were observed among 30 late comer journals and only 2 first comers (Table 7). Impact neutrality (zero change in journal ranking) was observed among the remaining 236 journals in which majority were first comer journals. Among the late comer journals that showed significant positive and negative changes in impact were: the Journal of Economic Surveys, Journal of Risk and Uncertainty, Cambridge Journal of Regions

10 2194 V. G. Krishna et al. Table 3 List of first comer journals based on their subject categories ISI indexed journal Journal of Finance Journal of Financial Transportation Research Part B-Methodological Journal of Risk And Insurance Journal of Banking & Finance Journal of Economic Education Journal of Urban Journal of Environmental and Management Regional Studies Journal of International Journal of Post Keynesian American Economic Review Journal of Monetary Econometrica Journal of Econometrics Journal of Economic Theory Politicka Ekonomie Journal of Economic Issues World Development National Tax Journal Public Choice Journal of Economic History Economic History Review Journal of Public Transportation Research Part A-Policy and Practice History of Political Economy Journal of World Trade Journal of Transport and Policy Land Quarterly Journal of Journal of Political Economy Journal of Money Credit and Banking Acta Oeconomica Journal of Development Journal of Regional Science Ekonomicky Casopis South African Journal of Regional Science and Urban Futures Journal of Financial and Quantitative Analysis Journal of Human Resources Subject category Transportation, Education & Educational Research, Urban Studies, Business, Environmental Studies, Environmental Studies,, Geography Social Sciences, Mathematical Methods, Social Sciences, Mathematical Methods, Political Science, Planning & Development, Political Science, History of Social Sciences, History of Social Sciences, Transportation, History of Social Sciences, International Relations, Law, Transportation, Environmental Studies, Environmental Studies, Planning & Development, Environmental Studies, Urban Studies, Planning & Development, Industrial Relations & Labor,

11 Measuring scientific performance of ISI indexed journals 2195 Table 3 continued ISI indexed journal Revue D Etudes Comparatives Est-Ouest Explorations In Economic History Journal of Mathematical Food Policy Economy and Society Applied Journal of Economic Behavior & Organization Journal of Consumer Affairs Economic Geography Cambridge Journal of Review of Economic Studies Journal of Economic Dynamics & Control Journal of Agricultural World Economy Review of and Statistics Australian Economic History Review Journal of Industrial Oxford Bulletin of and Statistics Mathematical Social Sciences Economic Journal European Economic Review Journal of Law & Journal of Policy Modeling Journal of Development Studies Economic Development and Cultural Change Jahrbucher Fur Nationalokonomie Und Statistik Annals of Regional Science Review of Radical Political Letters Theory and Decision Economic Record Tijdschrift Voor Economische En Sociale Geografie International Economic Review Trimestre Economico Canadian Journal of -Revue Canadienne D Economique International Labour Review Economic and Social Review Subject category History of Social Sciences, Social Sciences, Mathematical Methods, Sociology, Business, Geography, Business, Finance, International Relations, Social Sciences, Mathematical Methods, History of Social Sciences, Social Sciences, Mathematical Methods, Social Sciences, Mathematical Methods, Law, Planning & Development, Area Studies, Planning & Development, Social Sciences, Mathematical Methods, Environmental Studies, Geography, Social Sciences, Mathematical Methods, Geography, Industrial Relations & Labor, Sociology,

12 2196 V. G. Krishna et al. Table 3 continued ISI indexed journal Journal of Macroeconomics Kyklos Eastern European Journal of Economic Literature Southern Economic Journal Developing Economies Oxford Economic Papers-New Series Economic Inquiry Scandinavian Journal of Scottish Journal of Political Economy Economica American Journal of and Sociology Hitotsubashi Journal of American Journal of Agricultural Brookings Papers on Economic Activity Australian Economic Papers Subject category Planning & Development, Political Science, Sociology, Economy and Society, Industrial and Corporate Change, Oxford Review of Economic Policy, and European Journal of Health. From here, it can be seen that a 5 year impact factor is a complementary method for first comers whilst it serves as an alternative indicator for late comer journals to measure their short term performance. Significant differences in rankings was observed among 143 first and late comer journals (94 % of total journals) when comparing the 2-year impact factor and the non self citations (see Tables 8, 9), which shows correlation effects of 74 % and 81 % between the 2 indices for both groups. It has shown that DIFNSC favours both first and late comer journals but, more priority was given to late comers (Table 9). Impact improvements was observed among 44 late comer journals compared to 34 first comers, while impact deterioration was observed among 56 late and 9 first comer journals (Table 9). Among the late comer journals that experienced significant changes in rankings were Journal of Risk and Uncertainty, Mathematical Finance, Industrial and Corporate Change and Journal of Economic Surveys. For DIFJEI, the generation of positive energy in terms of impact improvement was observed among 34 late comers compared to 105 first comers whereas negative energy through impact deterioration was seen among 81 late comers compared to 86 first comers (Tables 8, 10). The changes of rankings through DIFJEI are almost consistent with the remaining indices except DIFCCI due to the high correlations between them. It favours first comers rather than late comers as the former has the capacity of reflecting the dual momentum of positive and negative energies of performance Among the significant first comers are Letters, American Journal of Agricultural, Journal of Economic Theory, European Economic Review, Journal of Banking & Finance, Applied. Almost all the diachronous indices have highlighted the significant biasness among the late comer journals compared to first comers. Among all, the most significant indicator that can be used to evaluate both short and long term performance for both journals is DIFCCI

13 Measuring scientific performance of ISI indexed journals 2197 Table 4 List of late comer journals based on their subject categories ISI indexed journal Technological and Economic Development of Economy Inzinerine Ekonomika-Engineering Transformations in Business & Small Business Ecological Economic Systems Research Defence and Peace Amfiteatru Economic Journal of Business and Management Economia Politica Journal of Real Estate Finance and Real Estate Agricultural -Zemedelska Ekonomika Asian Economic Policy Review Journal of Economic Geography Games and Economic Behavior Health Journal of Productivity Analysis Emerging Markets Finance and Trade Post-Soviet Affairs Journal of Risk and Uncertainty Journal of Transport Geography of Education Review Panoeconomicus Tourism Europe-Asia Studies Marine Resource Transportation Research Part E-Logistics and Transportation Review Journal of Korea Trade Emerging Markets Review Environmental & Resource Journal of Sports Econ Journal Watch JCMS-Journal of Common Market Studies China Economic Review Journal of Agrarian Change Subject categories Business, Business, Management, Environmental Studies, Business, Business, Finance, Urban Studies, Business, Finance, Urban Studies, Geography, Health Policy & Services, Business, Social Sciences, Mathematical Methods, Business, International Relations, Area Studies, Political Science, Geography, Transportation, Education & Educational Research, Hospitality, Leisure, Sport & Tourism, Area Studies, Political Science, Environmental Studies, Transportation, Environmental Studies, Hospitality, Leisure, Sport & Tourism, International Relations, Political Science, Planning & Development,

14 2198 V. G. Krishna et al. Table 4 continued ISI indexed journal & Human Biology Zeitschrift Fur Wirtschaftsgeographie Journal of Media Romanian Journal of Economic Forecasting Journal of Regulatory Cambridge Journal of Regions Economy and Society Transport Policy Journal of Competition Law & Journal of Real Estate Research Journal of Housing Journal of Pension & Finance Mathematical Finance European Journal of Political Economy Economic Development Quarterly Spatial Economic Analysis Experimental Journal of Population Agricultural Feminist Journal of Financial Stability International Review of & Finance Journal of Evolutionary Economic Computation and Economic Cybernetics Studies and Research International Tax and Public Finance Journal of International Financial Markets Institutions & Money Industrial and Corporate Change Journal of Economic Growth Papers in Regional Science E & M Ekonomie A Management Information and Policy Asian Journal of Technology Innovation European Review of Agricultural North American Journal of and Finance Iktisat Isletme Ve Finans Review of International Organizations International Environmental Agreements-Politics Law and Journal of Cultural Subject categories Public, Environmental & Occupational Health, Geography, Communication, Geography, Transportation, Law, Urban Studies, Business, Finance, Social Sciences, Mathematical Methods, Political Science, Planning & Development, Urban Studies, Demography, Women s Studies, Business, Management, Environmental Studies, Geography, Management, Business, International Relations, Political Science, Environmental Studies, Law, Political Science,

15 Measuring scientific performance of ISI indexed journals 2199 Table 4 continued ISI indexed journal Economic Theory Journal of Institutional Review of Environmental And Policy Review of International Political Economy of Transition Astin Bulletin Quantitative Finance Cliometrica Journal of Agricultural and Resource Review of World World Trade Review Labour Metroeconomica Journal of Economic Inequality Ekonomska Istrazivanja-Economic Research Economic and Labour Relations Review International Journal of Economic Theory Review of Economic Dynamics Pharmacoeconomics New Political Economy Resource and Energy International Review of Law and Industry and Innovation Journal of Forest Value in Health China & World Economy IMF Economic Review European Journal of the History of Economic Thought Applied Letters China Agricultural Economic Review Socio-Economic Review Review of Industrial Organization Investigacion Economica Australian Journal of Agricultural and Resource Journal of & Management Strategy Geneva Risk and Insurance Review Post-Communist Economies Review of of The Household Journal of African Economies Review of International Subject categories Environmental Studies, International Relations, Political Science, Social Sciences, Mathematical Methods, Business, Finance, Social Sciences, Mathematical Methods, History, History of Social Sciences, International Relations, International Relations, Law, Industrial Relations & Labor, Health Policy & Services, International Relations, Political Science, Environmental Studies, Law, Management, Health Policy & Services, History of Social Sciences, Political Science, Sociology, Management, Management,

16 2200 V. G. Krishna et al. Table 4 continued ISI indexed journal Econometric Reviews Journal of Economic Policy Reform Journal of The Japanese and International Economies Journal of The European Economic Association Oxford Review of Economic Policy Macroeconomic Dynamics Journal of Australian Political Economy Agribusiness Review of Network Federal Reserve Bank of St Louis Review QME-Quantitative Marketing and Australian Economic Review American Economic Journal-Applied Zbornik Radova Ekonomskog Fakulteta U Rijeci- Proceedings Of Rijeka Faculty of Journal of Financial Econometrics European Review of Economic History International Journal of Transport Series-Journal of the Spanish Economic Association Journal of the Asia Pacific Economy Japanese Economic Review Applied Economic Perspectives and Policy Argumenta Oeconomica Review of Development Pacific Economic Review American Economic Journal-Economic Policy Journal of Behavioral Finance Asia Pacific Journal of Accounting & Studies in Nonlinear Dynamics and Econometrics Open Economies Review Economic Policy European Journal of Law and American Economic Journal-Macroeconomics Prague Economic Papers Manchester School -The Open Access Open-Assessment E-Journal Annals of and Finance International Finance Journal of Empirical Finance American Law and Review Estudios De Economia Subject categories Social Sciences, Mathematical Methods, Planning & Development, International Relations, Political Science, Business, Social Sciences, Mathematical Methods, Business, History of Social Sciences, Transportation, Planning & Development, Social Sciences, Mathematical Methods, Law, Law,

17 Measuring scientific performance of ISI indexed journals 2201 Table 4 continued ISI indexed journal Japan and the World Economy Journal of Economic Surveys Work Employment and Society Review of Finance World Bank Research Observer Annual Review of Resource Revista De Historia Economica Computational Finanzarchiv European Journal of Health Fiscal Studies German Economic Review Cepal Review Contemporary Economic Policy History of Economic Ideas Cesifo Economic Studies Theoretical Asian Economic Papers Empirical Econometrics Journal Journal of Human Capital B E Journal of Economic Analysis & Policy Journal of International Trade & Economic Development Independent Review Journal of Economic Interaction And Coordination American Economic Journal-Microeconomics Economic Systems Global Economic Review of Governance Journal of Public Economic Theory Recherches Economiques De Louvain Louvain Economic Review Review of Derivatives Research South African Journal of Economic and Management Sciences B E Journal of Theoretical International Journal of Health Care Finance & Annual Review of Financial Asian-Pacific Economic Literature Subject categories Industrial Relations & Labor, Sociology, Planning & Development, Environmental Studies, History, History of Social Sciences, Management, Health Policy & Services, Public Administration, History, History of Social Sciences, Social Sciences, Mathematical Methods, Social Sciences, Mathematical Methods, Political Science, Management, Business, Finance, Health Policy & Services,

18 2202 V. G. Krishna et al. Table 4 continued ISI indexed journal Annual Review of Revista De Economia Aplicada B E Journal of Macroeconomics Revue D Economie Politique Review of Economic Design & Politics Portuguese Economic Journal Asian Economic Journal Empirica Quantitative Bulletin of Economic Research Journal of Applied Baltic Journal of Custos E Agronegocio on Line Economia Mexicana-Nueva Epoca Hacienda Publica Espanola-Review of Public Korean Economic Review Revista De Historia Industrial Subject categories Political Science, Political Science, Business, History, History of Social Sciences, (Tables 8, 11). It has supported 60 % and 57 % of late and first comer journals in revealing their true scientific performance and fairness based on a forward approach. Through the evaluation of the performance of journals using the index, it can be seen that both older and late comer journals perform better in expanding citations. While older gain higher degree of popularity and recognition, they also provide space for newer or later comer journals to build up their reputation by using the former as a benchmark. In other words, the first-late comer approach offers a win win situation for both older and newer journals. Tables 10, 11 and 12 has indicated some biasness that can be seen between first and late comer journals in the context of subject categories. The synchronous version of impact factor only favours the first comer journals with affiliated and non affiliated subject categories such as Journal of Finance, Journal of Financial and etc. due to the advantage of early scientific recognition. The clear mode of biasness between the first comer and later comer journals can be evaluated through the significant differences of diachronous impact factors and highest momentum of reflection goes to first comers rather than late comers such as Value in Health, Technological and Economic Development of Economy and etc. DIFCCI has the capacity to eliminate the general biasness of the synchronous impact factor that deals with short-term windows as it deals with only citations of full length articles and with lagged years. The reliability of the latter can be seen by tracking the upward and downward movement of journal rankings using Garfield (1955). As can be seen the existing impact factor distorts the scientific fairness (Ingwersen et al. 2001) among the older and newer journals. Older journals may have accumulated higher citations on

19 Measuring scientific performance of ISI indexed journals 2203 Table 5 Quantitative forms of normality tests (Kolmogorov Smirnov and Shapiro Wilk) for first and late comer journals a,b 2-year and 5-year synchronous impact factors c,d,e Diachronous impact factors based on non self citations, citation concentration index and journal energy index ** Significant at 0.05 (5 %) Variable Kolmogorov Smirnov Shapiro Wilk P value P value First comer journals SIF_2 a 0.00** 0.00** SIF_5 b 0.00** 0.00** DIFNSC c 0.00** 0.00** DIFCCI d ** DIFJEI e 0.00** 0.00** Late comer journals SIF_2 a 0.00** 0.00** SIF_5 b 0.00** 0.00** DIFNSC c 0.00** 0.00** DIFCCI d 0.00** 0.00** DIFJEI e 0.00** 0.00** Table 6 Spearman correlation coefficients between synchronous and diachronous versions of impact factors for first and late comer journals SIF_2 a SIF_5 b DIFNSC c DIFCCI f DIFJEI g First comer journals SIF_2 a SIF_5 b 0.97** DIFNSC c 0.81** 0.86** DIFCCI d 0.35** 0.31** 0.31** DIFJEI e 0.78** 0.78** 0.81** 0.34** Late comer journals SIF_2 a SIF_5 b 0.95** DIFNSC c 0.74** 0.81** DIFCCI d 0.42** 0.34** 0.34** DIFJEI e 0.60** 0.65** 0.85** 0.48** a,b 2-year and 5-year synchronous impact factors c,d,e Diachronous impact factors based on non self citations, citation concentration index and journal energy index ** Significant at 0.05 (5 %) their scientific contents because of the higher degree of recognition based on concepts or issues. Newer journals may not gained enough citations due to the lack of recognition. Under this proposed framework they would have more time to catch up with older journals by building their own scientific citation base and international recognition. The latter strategy can be done by considering the life expectancy of publications that are based on diachronous or dynamic timing of citations. In addition, the bias introduced by the ISI impact factor can be seen in the concentration of the scientific documents. The current impact factor only takes up citations from full

20 2204 V. G. Krishna et al. Fig. 1 Matrix scatter plot of impact indices for first comer journals length articles, while ignoring those other scientific documents, such as, conference proceedings, article reviews, book reviews, and etc. DIFCCI has taken up citations of all the stated scientific documents for evaluating the scientific concentration among the first and late comers. DIFCCI is normalized by adjusting for the variations of issues, number of indexed publications and affiliated subject categories within the selected ISI Journals in. The existing impact factor does not have serious concerns for this. This research is different from Lu et al. (2014) and Ajiferuke and Wolfram (2010) in terms of the sampling period of investigation, citation-citer analysis, life expectancy, concentration of subject category, and the concepts of first-late comer advantages. The contribution of this paper is from the perspective that it has analyzed the real short and long terms performance of ISI Indexed Journals in by taking care of the life expectancy of publications, concept of first-late comer advantages and development of DIFCCI and DIFJEI. Overall, this exercise shows that the assessment of performance among scientific journals should not be based on just one dimension. It should be based on multi-dimensions (Vanclay 2009). It can then be validated through evaluations using the journal diffusion factors (Frandsen 2004; Rowlands 2002), Citer analysis (Ajiferuke and Wolfram 2010; Lu

21 Measuring scientific performance of ISI indexed journals 2205 Fig. 2 Matrix scatter plot of impact indices for late comer journals et al. 2014), and other related indices. The main limitations of this exercise are: firstly, it deals with only citations through comparisons using the synchronous impact factor and the diachronous impact factor; secondly, it focuses on the evaluation of the characteristics among ISI Indexed Journals in a single subject category. 4 Conclusions Although ISI s JIF is widely used in evaluating the performance of scientific journals, it has come under heavy criticisms recently (Lu et al. 2014; Raj and Zainab 2012). These shortcomings have resulted in proposals for new indicators among many scholars. Thus, this paper sought alternative methods that can complement the JIF so that late-comer journals are not unfairly disadvantaged against first-comer journals. While the most significant index that promotes scientific fairness and recognition is the citation concentration

22 2206 V. G. Krishna et al. Table 7 Changes in the ranks of journals between 2 and 5 year impact factors (synchronous impact factors) for first and late comer journals Changes in rank Number of journals First comer journals? Late comer journals? ? Table 8 Results of Mann Whitney-U-Wilcoxon-W and Friedman test between the journals and within impact indices of first and late comer journals Mean ± SD P value a SIF_2 FC 1.36 ± ** LC 0.92 ± 0.77 SIF_5 FC 1.98 ± ** LC 1.14 ± 0.99 DIFNSC FC ± ** LC 4.02 ± 4.93 DIFCCI FC 0.31 ± ** LC 0.21 ± 0.14 DIFJEI FC 1,725, ± 3,166, ** LC 101, ± 355, SIF_2, SIF_5, DIFNSC, DIFCCI, DIFJEI FC 0.00 b ** LC 0.00 b ** a Obtained from Mann Whitney U-Wilcoxon-W test b Obtained from Friedman test ** Significant at 0.05 (5 %) index, we found the synchronous approach to offer a complementary instrument for evaluating short term mode of performance and permanent impact of publications among first comers. In addition, the diachronous impact factor offers a good alternative evaluation method for journals as it promotes fairness among first and late comer journals. The evidence shows that these approaches would provide an opportunity for late comers to improve their short and long scientific performance as the forward approach ignores citation lags. Table 13 illustrates the features of synchronous and diachronous approaches of impact factors via various aspects of evaluations such as years of investigations, performance evaluation, scientific documents, lagged structure of years, usefulness, advantages and disadvantages. Synchronous approach is more applicable to the scientific investigations on the short term performance of citations or citing agents that are tied up with fixed timing and it favours most of the first comer journals that have created their citation system from

23 Measuring scientific performance of ISI indexed journals 2207 Table 9 Changes in the ranks of journals between 2 year and DIFNSC for first and late comer journals Changes in rank Number of journals First comer journals? ?2 2 Late comer journals? ? Table 10 Changes in the ranks of journals between 2 year and DIFJEI for first and late comer journals Changes in rank Number of journals First comer journals? ? ? Late comer journals? ? ? the early stage. It is often called as a classical approach as it evaluates the citations from present to past via backward dimension. Diachronous approach is more suitable and applicable to the scientific investigations on the short and long term performance of citations or citing agents that are consistent with dynamic timing and it favours both first and late comers, but more priorities are given to late comers who have just started to create their citation system. Future research should examine the segregated effects of the unique heterogeneous agents (authors, institutions, funding agencies, research areas, subject categories, and research grants) on changes in journal rankings among ISI Indexed Journals in the different subject categories. This way, the interactional effects between diachronous mode of impact

24 2208 V. G. Krishna et al. Table 11 Changes in the ranks of journals between 2 year and DIFCCI for first and late comer journals Changes in rank Number of journals First comer journals? ? Late comer journals? ? Table 12 Results of synchronous and diachronous approaches based on subject categories First comer journals Late comer journals P value a P value b IF_2 DIFTC N IF_2 DIFTC N Subject affiliated , ** 0.12 Non-subject affiliated ,600 DIF-IF (DIF-101) 0.00** 0.02** (IF-100) a Obtained from Mann Whitney test based on the values of DIFTC b Obtained from Mann Whitney U-test based on the values of IF_2.DIFTC is diachronous impact factor that refers to the total citations in all the years of publications within selected journals. IF_2 refers to 2 year impact factor based on synchronous approach ** Significant at 0.05 (5 %) Table 13 Features of synchronous and diachronous approaches of impact factors Synchronous approach Diachronous approach Years of investigation Performance evaluation Scientific document Lagged structure of years Fixed years Short-term performance Only full length articles Yes Flexible years Short and Long term performance All the scientific documents, including conference proceedings and reviews Usefulness Only favours first comer journals Priority will be given to late comer journals but first comer journals will also be considered for impact evaluation No

25 Measuring scientific performance of ISI indexed journals 2209 Table 13 continued Advantages Disadvantages Synchronous approach (a) Can be used mostly by first comer journals to evaluate the past performance of their publications (b) Can capture the permanent impact of journals (c) Evaluate the short term performance from present to past via backward approach (a) No dynamic changes of citations as the focus is mainly on backward approach (b) Focus is only on full length articles (c) The basic mode of technical aspects don t deal with issues or titles and keywords Diachronous approach (a) Can be used both by first and late comer journals to evaluate the past, current and future performance of their publications (b) Can capture the actual impact of journals (c) Evaluate the short and long term performance from present to present and future via forward approach (a) The basic mode of technical aspects have not explored the usage of issues, titles and keywords within the computations although there is a way to deal with it factors, and theories of micro-meso-macro and meso trajectory phases of innovation could be further explored. Acknowledgement We would like to acknowledge Dr. Ram Gopal Raj and Mr. Ephrance Abu Ujum for their comments as well as UM High Impact Research Grant (HIR) C/625/1/HIR/MOHE/SC/13 for supporting this research work. Appendix See Figs. 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12. Fig. 3 Normal Q Q plot of 2 year impact factor for first comer journals

26 2210 V. G. Krishna et al. Fig. 4 Normal Q Q plot of 5 year impact factor for first comer journals Fig. 5 Normal Q Q plot of diachronous impact factor based on non-self citations for first comer journals

27 Measuring scientific performance of ISI indexed journals 2211 Fig. 6 Normal Q Q plot of diachronous impact factor based on citation concentration index for first comer journals Fig. 7 Normal Q Q plot of diachronous impact factor based on journal energy index for first comer journals

28 2212 V. G. Krishna et al. Fig. 8 Normal Q Q plot of 2 year impact factor for late comer journals Fig. 9 Normal Q Q plot of 5 year impact factor for late comer journals

29 Measuring scientific performance of ISI indexed journals 2213 Fig. 10 Normal Q Q plot of diachronous impact factor based on non-self citations for late comer journals Fig. 11 Normal Q Q plot of diachronous impact factor based on citation concentration index for late comer journals

30 2214 V. G. Krishna et al. Fig. 12 Normal Q Q plot of diachronous impact factor based on journal energy index for late comer journals References Ajiferuke, I., Wolfram, D.: Citer analysis as a measure of research impact: library and information science as a case study. Scientometrics 83(3), (2010) Bergstrom, T.C.: Free labor for costly journals? J. Econ. Perspect. 15, (2001) Bouabid, H., Larivière, V.: The lengthening of papers life expectancy: a diachronous analysis. Scientometrics 97(3), (2013) Danell, R., Engwall, L., Persson, O.: The first mover and the challenger: the relationship between two journals in organization research. Scientometrics 40(3), (1997) Della Sala, S., Grafman, J.: Five-year impact factor. Cortex 45(8), 911 (2009) Frandsen, T.F.: Journal diffusion factors a measure of diffusion? Aslib Proc. 56(1), 5 11 (2004) Garfield, E.: Citation indexes for science: a new dimension in documentation through association of ideas. Science 122, (1955) Glänzel, W., Moed, H.F.: Journal impact measures in bibliometric research. Scientometrics 53(2), (2002) Glänzel, W., Thijs, B., Schlemmer, B.: A bibliometric approach to the role of author self-citations in scientific communication. Scientometrics 59, (2004) Glanzel, W.: On some new bibliometric applications of statistics related to the h-index. Scientometrics 77(1), (2008) Hemmingson, A.: Manipulation of impact factors by editors of scientific journals. Am. J. Roentgenol. 178(3), 767 (2002) Ingwersen, P.: The pragmatics of a diachronic journal impact factor. Scientometrics 92, (2012) Ingwersen, P., Larsen, B., Rousseau, R., Russell, J.: The publication-citation matrix and its derived quantities. Chin. Sci. Bull. 46(6), (2001) Lu, K., Ajiferuke, I., Wolfram, D.: Extending citer analysis to journal impact evaluation. Scientometrics 100(1), (2014) Merton, R.K.: The Matthew effect in science. Science 159(3810), (1968) Moed, H.F., Colledge, L., Reedijk, J., Moya-Anegon, F., Guerrero-Bote, V., Plume, A., Amin, M.: Citationbased metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way. Scientometrics 92(2), (2012) Monastersky, R.: The number that s devouring science. Chron. High. Educ. 52(8), 14 (2005) Pendlebury, D.A., Adams, J.: Comments on a critique of the Thompson Reuters journal impact factor. Scientometrics 92, (2012) Prathap, G.: Quasity, when quantity has a quality all of its own-toward a theory of performance. Scientometrics 88(2), (2011)