A Generalized Measure Of Diversity: Application To Longitudinal Data On .

Utpal Kumar De et al. / Journal of the Indian Society of Agricultural Statistics 71(3) 2017 253–263Pandya 1998). The EI is defined as EI – iN 1 Pi Log Pi .Its value varies from 0 (perfect concentration) to Log N(perfect diversity). The upper value of EI can exceedone or be less than one when N is greater or less than thebase of logarithm. Thus it does not correspond to anystandard scale of measuring degree of diversification.In MEI number of crops, N is considered as the baseNof logarithm. Symbolically, MEI i 1 ( Pi . Log N Pi ) .The lowest and the highest value of MEI is zero (whenonly one crop is cultivated in the whole area) and one(perfect diversification) respectively. Though it isbetter than those other measures (due to its uniformscale) it does not consider the change in value dueto changes in number of crop activities (Shiyani andPandya 1998).Hannah and Kay (1977) stated that most ofthe indices are special cases of the general indexφ 1 1 φI φ ( in where Pi is the share of ith item and ϕ1 Pi )is a parameter, such that ϕ 0 and ϕ 1. For ϕ 2 theindex becomes 1 in 1 Pi2 , which is the inverse of theHerfindahl Index of measuring disparity. In extremecase of ϕ 1 the index becomes the Entropy Index.This general index measures both the number of itemsand the evenness of item shares, with the parameter ϕdetermines the weighting of emphasis on numberof items versus evenness. When, ϕ is zero it simplycounts the number of items (Patil and Taillie 1982).Using this general index, Tauer and Seleka (1994)did not find significant relationship between the cashreceipt variability with the reduction in diversificationof agricultural production across 38 US states during1960 to 1989. Since they used cash receipts to measurediversity across commodities, it does not capturediversity by production practice.Hoffmann (2007) however tried to provide adistribution based measure of diversity which is moreflexible and represents a useful complement to modelsof generalized feature based diversity, such as Nehringand Puppe’s (2002) “Theory of Diversity”. There aretwo fundamentally different generalised models ofdistribution based diversity indices. The so-called‘Rényi diversity’ is an additive and non-concave oneparameter generalisation of the Shannon Entropy(Rényi 1961). It is used to measure biodiversity(Ricotta 2003), statistical sampling (Mayoral 1998),255economic diversity modelling (Beran 1999) etc. Theother generalisation of Shannon entropy, which isconcave but not additive was derived by informationtheorists Havrda and Charvát (1967), statisticians Patiland Taillie (1982), physicist Tsallis (1988) and alsoused for biodiversity measurement (Keylock 2005).Also, researchers used a number of inequalityindices to examine concentration of the whole in a fewitems as opposed to diversification (Clarke 1993, Gini1912, Roll 1992, Tabner 2006). Various concentrationand diversification indices are used by Meilak (2008)to examine the export concentration and its link withthe size of the economy. Most of these concentrationindices exhibit the general form: CI PiWi ; i 1, 2,., n, where Wi the weight attached to the export shareof a particular export category, Pi the share of exportcategory i and n the number of export categories.Most of these indices are based on the sharesof each category and used arbitrary weights to therespective items and applicable to cross section data.These measures ignore relative size variations incommodity groups and these can equally describe acountry exporting one product and a country exportingx product groups with similar shares.Though, these measures satisfy the generalcharacteristics of a concentration or diversificationmeasure, it calls for improvement upon such indicesin order to incorporate the interdependencies amongthe variables. Also, while analysing temporal changes,there is general tendency of changes in variables withtime. It may be important to adjust with such trendin order to understand the changing diversity due tohuman effort on consideration of other factors. In thispaper, we tried to develop an alternative measure ofdiversification by taking into account the bi-variatecorrelation coefficients (of time series of the variables)and excluding the trend effects. Also, we compare theresults to examine the improvement in the resultsobtained by using such measure.Please note that as in the present case correlationsamong crops (here crop groups) over time is consideredexcluding the time trend the emphasis of this measureis on the time series or panel data. For cross sectioncomparison of diversity say across zone no question oftime effect or correlation at one point would occur andhence simple HHI can be applied.

256Utpal Kumar De et al. / Journal of the Indian Society of Agricultural Statistics 71(3) 2017 253–263Review of Some Relevant MeasuresDiversification and the Proposed MeasureofKay (1977) proposed a set of axioms based on entry ofnew firms, merger of firms etc.For a meaningful measure of diversity it is firstnecessary to convert all the variables into shares.This is to make the measures independent of unitof measurement. Suppose pi is the share of ith item.Assume without loss of generality that p1 p2 pn, where n is the number of items in the group.Naturally, pi 1.There are many other concentration indicesproposed in the literature, for example, Hall andTideman Index (Hall and Tideman 1967, Hause1977, Anbarci and Katzman 2005), Entropy measureof concentration (Hart 1971), ComprehensiveConcentration Index (CCI), by Horvarth (1970);Hannah and Kay (1977) measure of concentration;Erlat and Akyuz Index of Concentration (2001); Uindex by Davies (1979) and Linda Index (Linda 1976).One can also define the corresponding diversificationindex to each of the concentration indices. Since weshall mainly concentrate on Herfindahl DiversityIndex (HDI), we refrain ourselves of narrating allthese indices.If P is a measure of concentration then a measureof diversity may either be 1/P (reciprocal of P), or 1–P(complementary of P). The value of 1–P lies between0 and 1 as P lies in between 0 and 1. That is why thismeasure (1–P) is preferred over 1/P as a measure ofdiversification.Among other indices, Absolute ConcentrationRatio is the oldest and a very popular measure whichis defined asACR ,where, K is the top K firms in the industry. This isalso known as K-firm concentration ratio. Within anindustry, K is often taken as a number between 3 and5, because it is often the case that 3-5 largest firmshave about 70%-80% share of the industry. K is thusknown as the focal point of concentration. However, Kshould be fixed if we want to compare among differentindustries. If firms of all the industries are taken thenthe value of K may be very large. The correspondingdiversity index is Absolute Diversification Index(ADI).ADI 1 –We can define a more general form of mostpopular Herfindahl-Hirschman Index (HHI) by takingthe following weighted sum of Pi:GHI ,where wi is the weight attached to Pi , so that 1. HHI is then viewed as the weighted sumof Pi , where the weight of Pi is nothing but Pi itself.Thus smaller firms contribute less to the value ofthe index. Hall and Tideman (1967) proposed somedesirable properties which should be satisfied by agood measure of concentration. HHI satisfies all theaxioms proposed by Hall and Tideman. Apart fromsome of the Hall and Tideman axioms, Hannah andFollowing HDI, Douglas Rae (1967, 1968)proposed an index of fractionalization in the party, wheresystem, which is defined asDDR the index of fractionalization in the party systemand Pi is the proportion of party i of votes given (Seealso Rae et al. 1970, Vayrynen 1970). Greenberg’smeasure of linguistic diversity (Greenberg 1956) isalso same as the above index.A similar index of fragmentation, defined by Raeand Taylor (1970), is, whichis the probability that a randomly selected pair ofindividuals in a society will belong to different groups.ni is the number of members of the ith group.2. THE PROPOSED MEASUREWhile measuring the oligopolistic power in themarket one should consider the existing cooperationbetween firms. When there is more cooperationbetween the firms, there is more power of the firmsin the market. Correlations between agriculturalvariables (here proportion of area under crops) shouldreduce the diversity index. Almost all the productionsof the commodities move in the same direction astime. Time here acts as an intervening variable. So theeffect of time from each variable should be removedbefore finding the correlation structure.Let us assume that the correlation between areaunder ith and jth crop over time is ρij after eliminatingthe effect of time. This is in fact the correspondingpartial correlation coefficient. The proposed measure

Utpal Kumar De et al. / Journal of the Indian Society of Agricultural Statistics 71(3) 2017 253–263is Modified Diversity Index (MDI), defined asnnMDI 1 – i 1 j 1 pi pij pj 1 – P’RP, where R is thematrix of ρij values and P is the column vector of sharesof area under crops for the current period. Observethat P’RP can be taken as a measure of concentrationindex. There are some salient features of this index.(a) 0 MDI ADI, (assuming that any twovariables are non-negatively correlated). It is0 when ρij 1 for all i and j, i.e., when any twovariable move in the same-direction withoutany error. In this casebecomes (ΣPi)2 1. It is HDI when the variables are completelyindependent of each other, ρij 0 for all i and j.257(due to irrigation, fertiliser and technologicalprogress) proportion of land allocation towards fruits& vegetables and oilseeds increase over time.This is done for each year starting from 1951‑52to 2010-11. Thus we have data for 60 years. Thefollowing line diagrams give us a clear idea about thetrend of area for each of the major groups.(b) 0 MDI 1, assuming that the variables canmove freely, i.e., when ρij can take any valuein the range – 1 to 1. [P’RP P’1P 1, where is the matrix consisting of 1’sonly. Since R is a symmetric non-negativedefinite matrix, we have P’RP 0.]3. D ESCRIPTION OF DATA ANDCALCULATIONSThe present analysis is based on the secondarydata on area under various crops since 1951 to 2011collected from various issues of Statistical Hand Bookof Assam, Economic Survey of Assam and Reportsfrom Directorate of Economics and Statistics andDirectorate of Agriculture, Government of Assam.Though several studies used earnings from theproduction of crops in order to compute the diversityindex, here we use area under crops for the purpose.This is because the farmers try to maximise theirreturns from limited land under them through suitableallocation of land among the cultivable crops. Thusthe land size allocation to different crops reflects theintention of the farmers which may not be realizedthrough production. Moreover, the area of cropcultivation is more robust than the actual production,which is subject to technology available at the timeof production and to the climatic behaviour of nature.Here instead of considering all individual crops,six major crop groups namely, cereals, pulses, oilseeds,fibres, plantations and fruits and vegetables have beenconsidered. If all the 29 crops grown in Assam areconsidered, then it will be too laborious. Moreover,despite the fact that some food crops now-a-daysbecame cash crops, with the progress of agricultureFig. 1. Trends in the Areas under Major Crop Groups in AssamIt is clear from Fig. 1 that all but one major grouphave increasing trend. The one which has recorded adecreasing trend is ‘Fibres’. Fibres do not come underfood crops. The patterns of growth of the groups otherthan Cereals are not easily discernible because ofCereals itself, which captures over 70 per cent of thetotal area under cultivation. The trends of other fivegroups can be clearly visible from Fig. 2 drawn forthese five groups only.Fig. 2: Trends in the Areas under Major Crops except CerealsIf the patterns of trends are observed, one cannotice that there is an increasing movement in all butFibre up to the year 1987-88. Also the increasing trendis somewhat slow up to 1975-76. After 1987-88, it

258Utpal Kumar De et al. / Journal of the Indian Society of Agricultural Statistics 71(3) 2017 253–263either decreases or increases at a slower rate than thepast. The similarity of the trends would imply highpositive correlations among the groups except possiblywith the areas under Fibres. Since from the figures it isevident that the trends are similar for each group, thebivariate correlation coefficients among these groupshave been computed to check whether it is really so.The correlation matrix is as follows.Table 1. Bivariate Correlations of the Areas under Major CropGroupsCereals PulsesCerealsPulsesOil SeedsFibresPlantationCropsMisc. 77**-.613**1Plantation ***-.834-.860**1.989**1** Correlation is significant at the 0.01 level (2-tailed).Area under Fibres is negatively correlated with allother area groups. This was expected from the trenddiagram. But, the degrees of correlations, regardlesswhether negative or positive are too high in absolutevalue for the areas under crop to be of further effectiveuse without trend corrections. Time seems to be anintervening variable. Thus all the variables for timehave been corrected and then diversity index foreach year have been computed by using the proposedformula. To compare these indices the diversity indiceshave been computed using existing method. This leadsto the following six types of indices:1. Herfindahl-Hirschman Diversification Indexwithout correcting the variables for time.4. Entropy Index after correcting the variablesfor timeCEI iN 1 Pijφ Log Piφ log Nwhereis the share after correcting thevariables for time.5. Modified Diversity Index (MDI)nMDI 1 – i 1 nj 1 pi pij pj 1 – P’RP,where ρij is the correlation between ith and jthareas, P is the vector of shares and R is thecorrelation matrix of areas.6. Modified Diversity Index after correcting thevariables for time (Corrected MDI)nCMDI 1 – i 1 pi pij pj 1 – PφRφPφwhereis the share of area of ith crop groupafter correcting the variables for time, P* is thecorresponding vector and R* is the correlationmatrix of areas after correcting the variablesfor time.Here it should be noted that correction for timeshould be mean preserving. Otherwise the indices aremeaningless. More precisely, first we run regression ofeach variable on time, find the residuals and add eachresidual with the mean value of the variable.4. OBSERVATIONS AND DISCUSSIONFrom the data, first of all the bivariate correlationsρij, and the shares Pi have been computed. Thereafter,HDI and MDI are computed. The variables are thenHDI 1 – in 1 Pi2 .2. Herfindahl-HirschmanDiversificationIndex after correcting the variables for time(Corrected HDI)CHDI 1 – in 1 Pi*2 , whereis the shareafter correcting the variables for time.3. Entropy Index (EI) iN 1 Pi Log Pi log N ,where Pi is the proportion of area under ith cropgroup. Since the Entropy or corrected EntropyIndex takes the maximum value Log N, wedivided the values of indices obtained byLog N to make it comparable with HDI orMDI.Fig. 3: Trends in the Diversity Indices of the Areas under Major CropGroups

Utpal Kumar De et al. / Journal of the Indian Society of Agricultural Statistics 71(3) 2017 253–263259Table 2. Year wise Diversity Indices of the Areas under Major Crop Groups in 00.5690.5430.7150.605corrected for time and translated so that they becomemean preserving. The correlations and the sharesare computed from corrected mean preserving areas.These may be termed as corrected correlations andcorrected shares respectively. In Table 2 we presentthe six diversity indices.From Fig. 3 it is clear that all the indices declinedtill 1975-76 and then gradually increased up to1987-88. Thereafter, all the indices became more orless stagnant. In case of HDI and Entropy the actualand corrected indices are found to be very close andthe corrected values lie below the actual one before1975‑76 and gradually crossed above the actual figuresafter 1975-76. In case of MDI also CMDI lies belowthe MDI till 1975-76 and gone above it afterwardsTable 3. Bivariate Correlations of the Diversity Indices of theAreas under Major Crop Groups in CEI1**. Correlation is significant at the 0.01 level (2-tailed).and the gap increased continuously. Here, as all thevariables in absolute form increased with time, thecorrected (against time trend) indices lies above theactual figures in their rising phase (after 1975-76) andlies below the actual figures in their declining phase.

260Utpal Kumar De et al. / Journal of the Indian Society of Agricultural Statistics 71(3) 2017 253–263Another interesting point to be noted is that the MDIis always higher than the HDI and Entropy (EI) inboth corrected and uncorrected forms. But all the sixindices have similar pattern of movement, first declineto give more emphasis on the food crops for foodsecurity concern (up to 1975-76) and then increasedthough slowly up to 1987-88 and then stabilized withvery slow changes in diversity. This is reflected byhigh correlations among them, as shown in Table 3.The proposed Modified Diversity Index (MDI)and its corrected version (CMDI) are thus differentfrom and lie above the other four indices. If wecompare CMDI and MDI we see prominent contrastbetween the two indices. The difference between thetwo values is initially higher than the correspondingother to pairs (EI vs. CEI and HDI vs. CHDI) and thisdifference gradually decreases at a point in a similarmanner. When they increase, the difference betweenthe two is again sharper than the other two pairs. ThusCMDI reflects better diversity pattern.In conclusion, it can said that after Independencethe diversity of crop areas more or less has a decreasingtrend up to the period of 1976-77 and then increased upto 1987-88. After 1987-88 it is more or less stagnant.The reason for this is not clear and should be tracedin migration patterns, floods or other economic andnatural phenomena.It was already noted that the area under crops foreach of the major groups were found by aggregatingthe areas under individual crops as follows:CerealsRice (Paddy), Maize, Wheat and Other CerealPulsesGram, Arhar and Other Rabi PulsesOil SeedsRape and Mustard, Sesamum, Linseed andCastorFibresJute, Mesta and CottonPlantation CropsTea, Sugar Cane, Areca Nut and TobaccoFruits andVegetablesPotato, Sweet Potato, Chilli, Turmeric, Tapioca,Banana, Onion and Pine AppleA closer look in the data reveals that data forsome of the individual crops were not available up to1975-76 and thus the aggregated values were foundneglecting the unavailable observations. This furtherconfirms our proposition that the indices shouldbe calculated separately for all the sub-periods. Toexamine how the correlations changes we may lookinto the bivariate correlations separately for the threeperiods as presented in Table 4.Table 4. Bivariate Correlations of the Areas under Major CropsSeparately for the Three PeriodsCereals PulsesOilSeedsFibresPlantationCropsMiscCropsPeriod 1 (1951-52 to 1975-76)Cereals1.000Pulses0.888** 0.863**1.0000.

1960 to 1989. Since they used cash receipts to measure diversity across commodities, it does not capture diversity by production practice. Hoffmann (2007) however tried to provide a distribution based measure of diversity which is more flexible and represents a useful complement to models of generalized feature based diversity, such as Nehring