Chapter 2. 1.what are the types of Data Sets

Size: px

Start display at page:

Download "Chapter 2. 1.what are the types of Data Sets"

Ellen Horton
5 years ago
Views:

1 Record Relatoal records Data matr, :umercal matr Documet data: tet documets: term-frequecy vector Trasacto data Graph ad etwork World Wde Web Socal or formato etworks Molecular Structures Chapter.what are the types of Data Sets >>>. what are Importat Characterstcs of Structured Data Dmesoalty : Curse of dmesoalty Sparsty : Oly presece couts Resoluto: Patters deped o the scale Dstrbuto:Cetralty ad dsperso Ordered Vdeo data: sequece of mages Temporal data: tme-seres Sequetal Data: trasacto sequeces Geetc sequece data Spatal, mage ad multmeda: Spatal data: maps Image data: Vdeo data: 3 Data Obects Data sets are made up of data obects. A data obect represets a etty. Eamples: sales database: customers, store tems, sales medcal database: patets, treatmets uversty database: studets, professors, courses >>> Also called samples, eamples, staces, data pots, obects, tuples. >>>Data obects are descrbed by attrbutes. >>>Database rows data obects; colums attrbutes. >>> Attrbute (or dmesos, features, varables): a data feld, represetg a characterstc or feature of a data obect. E.g., customer _ID, ame, address 4. Dscrete vs. Cotuous Attrbutes Dscrete Attrbute Has oly a fte or coutably fte set of values E.g., zp codes, professo, or the set of words a collecto of documets Sometmes, represeted as teger varables Note: Bary attrbutes are a specal case of dscrete attrbutes Cotuous Attrbute Has real umbers as attrbute values E.g., temperature, heght, or weght Practcally, real values ca oly be measured ad represeted usg a fte umber of dgts Cotuous attrbutes are typcally represeted as floatg-pot varables

2 >>>5.Attrbute Types Nomal: categores, states, or ames of thgs Har_color = { black, blod, brow, grey, red, whte} martal status, occupato, ID umbers, zp codes Bary Nomal attrbute wth oly states (0 ad ) Symmetrc bary: both outcomes equally mportat e.g., geder Asymmetrc bary: outcomes ot equally mportat. e.g., medcal test (postve vs. egatve) Coveto: assg to most mportat outcome (e.g., HIV postve) Ordal Values have a meagful order (rakg) but magtude betwee successve values s ot kow. Sze = {small, medum, large}, grades, army rakgs Numerc Quatty (teger or real-valued) Rato Iheret zero-pot e.g., temperature Kelv, legth, couts, moetary quattes Iterval No zero-pot. Scale of equal-szed uts. Values have order E.g., temperature C or F, caledar dates 6. Basc Statstcal Descrptos of Data Motvato :To better uderstad the data: cetral tedecy, varato ad spread Data dsperso characterstcs : meda, ma, m, quatles, outlers, varace, etc. Numercal dmesos correspod to sorted tervals Dsperso aalyss o computed measures correspod to trasformed cube >>>6. Graphc Dsplays of Basc Statstcal Descrptos Boplot: graphc dsplay of fve-umber summary Hstogram: -as are values, y-as repres. frequeces Quatle plot: each value s pared wth f dcatg that appromately 00 f % of data are Quatle-quatle (q-q) plot: graphs the quatles of oe uvarat dstrbuto agast the correspodg quatles of aother Scatter plot: each par of values s a par of coordates ad plotted as pots the plae

Hstogram Aalyss Hstogram: Graph dsplay of tabulated frequeces, show as bars It shows what proporto of cases fall to each of several categores >>>Dffers from a bar chart that t s the area of the bar

3 7. Boplot Aalyss Fve-umber summary of a dstrbuto: Mmum, Q, Meda, Q3, Mamum Data s represeted wth a bo The eds of the bo are at the frst ad thrd quartles,.e., the heght of the bo s IQR The meda s marked by a le wth the bo Whskers: two les outsde the bo eteded to Mmum ad Mamum Outlers: pots beyod a specfed outler threshold, plotted dvdually 8. Hstogram Aalyss Hstogram: Graph dsplay of tabulated frequeces, show as bars It shows what proporto of cases fall to each of several categores >>>Dffers from a bar chart that t s the area of the bar that deotes the value, ot the heght as bar charts, a crucal dstcto whe the categores are ot of uform wdth The categores are usually specfed as o-overlappg tervals of some varable. The categores (bars) must be adacet why Hstograms Ofte Tell More tha Boplots The two hstograms may have the same boplot represetato The same values for: m, Q, meda, Q3, ma But they have rather dfferet data dstrbutos

4 0. Quatle Plot Dsplays all of the data (allowg the user to assess both the overall behavor ad uusual occurreces) For a data data sorted creasg order, f dcates that appromately 00 f% of the data are below or equal to the value. Quatle-Quatle (Q-Q) Plot Graphs the quatles of oe uvarate dstrbuto agast the correspodg quatles of aother Vew: Is there s a shft gog from oe dstrbuto to aother? yes We eed to label the dark plotted pots as Q, Meda, Q3 that would help uderstadg ths graph.

5 . Scatter plot Provdes a frst look at bvarate data to see clusters of pots, outlers, etc Each par of values s treated as a par of coordates ad plotted as pots Determe Postvely ad Negatvely Correlated Data Postve egatve left half fragmet s postvely correlated rght half s egatve correlated Ucorrelated Data 3. Propertes of Normal Dstrbuto Curve(μ: mea, σ: stadard devato) From μ σ to μ+σ: cotas about 68% of measuremets From μ σ to μ+σ: cotas about 95% of t From μ 3σ to μ+3σ: cotas about 99.7% of t

mddle two values Estmated by terpolato (for grouped data): meda L w w / ( ( freq meda freq) l ) wdth Mode Value that

6 4. Measurg the Cetral Tedecy Mea (algebrac measure) (sample vs. populato): Note: s sample sze ad N s populato sze. N Weghted arthmetc mea: Trmmed mea: choppg etreme values Meda: Mddle value f odd umber of values, or average of the mddle two values Estmated by terpolato (for grouped data): meda L w w / ( ( freq meda freq) l ) wdth Mode Value that occurs most frequetly the data Umodal, bmodal, trmodal Emprcal formula: mea mode 3 ( mea meda) Symmetrc vs. Skewed Data

7 5. Measurg the Dsperso of Data Quartles, outlers ad boplots Quartles: Q (5 th percetle), Q3 (75 th percetle) Iter-quartle rage: IQR = Q3 Q Fve umber summary: m, Q, meda, Q3, ma Boplot: eds of the bo are the quartles; meda s marked; add whskers, ad plot outlers dvdually Outler: usually, a value hgher/lower tha.5 IQR Varace ad stadard devato (sample: s, populato: σ) Varace: (algebrac, scalable computato) s ( ) N N ( ) [ ( ) ] Stadard devato s (or σ) s the square root of varace s ( or σ ) 6. Promty refers to a smlarty or dssmlarty Smlarty Numercal measure of how alke two data obects are Value s hgher whe obects are more alke Ofte falls the rage [0,] Dssmlarty (e.g., dstace) Numercal measure of how dfferet two data obects are Lower whe obects are more alke Mmum dssmlarty s ofte 0 Upper lmt vares 7. Data Matr ad Dssmlarty Matr Data matr data pots wth p dmesos Two modes f f f p p p Dssmlarty matr data pots, but regsters oly the dstace A tragular matr Sgle mode 0 d(,) d(3,) : d(,) 0 d(3,) : d(,) 0 : 0 اللي جاي قوانين ومسائل صحصح لو سمحت

8 Promty Measure for Nomal Attrbutes Ca take or more states, e.g., red, blue, gree (geeralzato of a bary attrbute) Method : Smple matchg : m: # of matches, p: total # of varables d(, ) p p m Method : Use a large umber of bary attrbutes creatg a ew bary attrbute for each of the M omal states Promty Measure for Bary Attrbutes A cotgecy table for bary data Dstace measure for symmetrc bary varables: Dstace measure for asymmetrc bary varables: Jaccard coeffcet (smlarty measure for asymmetrc bary varables): Note: Jaccard coeffcet s the same as coherece : >>>8.determe Dssmlarty betwee Bary Varables Name Geder Fever Cough Test- Test- Test-3 Test-4 Jack M Y N P N N N Mary F Y N P N P N Jm M Y P N N N N Geder s a symmetrc attrbute The remag attrbutes are asymmetrc bary

9 Let the values Y ad P be, ad the value N 0 Stadardzg Numerc Data Z-score: z 0 d( ack, mary) d( ack, m) 0.67 d( m, mary) 0.75 X: raw score to be stadardzed, μ: mea of the populato, σ: stadard devato egatve whe the raw score s below the mea, + whe above A alteratve way: Calculate the mea absolute devato s f ( m m m m f f f f f f ( Where ). f f f >> Usg mea absolute devato s more robust tha usg stadard devato f ) Dstace o Numerc Data: Mkowsk Dstace Mkowsk dstace: A popular dstace measure where = (,,, p) ad = (,,, p) are two p-dmesoal data obects, ad h s the order (the dstace so defed s also called L-h orm) Propertes d(, ) > 0 f, ad d(, ) = 0 (Postve defteess) d(, ) = d(, ) (Symmetry) d(, ) d(, k) + d(k, ) (Tragle Iequalty) A dstace that satsfes these propertes s a metrc Specal Cases of Mkowsk Dstace h = : Mahatta (cty block, L orm) dstace E.g., the Hammg dstace: the umber of bts that are dfferet betwee two bary vectors d(, ) p p

10 h = : (L orm) Eucldea dstace ) ( ), ( p p d h. supremum (Lma orm, L orm) dstace. Ths s the mamum dfferece betwee ay compoet (attrbute) of the vectors 9. Calculate Data Matr ad Dssmlarty Matr ( Mkowsk Dstaces)

11 Ordal Varables Order s mportat, e.g., rak. A ordal varable ca be dscrete or cotuous Ca be treated lke terval-scaled replace f by ther rak r,, M } f { f map the rage of each varable oto [0, ] by replacg -th obect the f-th varable by z f rf M f compute the dssmlarty usg methods for terval-scaled varables Attrbutes of Med Type A database may cota all attrbute types Nomal, symmetrc bary, asymmetrc bary, umerc, ordal Oe may use a weghted formula to combe ther effects d d p ( f ) ( f ) ( f, ) p ( f ) f f s bary or omal: d (f) = 0 f f = f, or d (f) = otherwse

12 f s umerc: use the ormalzed dstace f s ordal Compute raks rf ad Treat zf as terval-scaled z f Cose Smlarty A documet ca be represeted by thousads of attrbutes, each recordg the frequecy of a partcular word (such as keywords) or phrase the documet. rf M f Other vector obects: gee features mcro-arrays, Applcatos: formato retreval, bologc taoomy, gee feature mappg, Cose measure: If d ad d are two vectors (e.g., term-frequecy vectors), the cos(d, d ) = (d d ) / d d, where dcates vector dot product, d : the legth of vector d 0 Fd the smlarty betwee documets ad. d = (5, 0, 3, 0,, 0, 0,, 0, 0) d = (3, 0,, 0,,, 0,, 0, ) dd = 5*3+0*0+3*+0*0+*+0*+0*+*+0*0+0* = 5 d = (5*5+0*0+3*3+0*0+*+0*0+0*0+*+0*0+0*0) 0.5 =(4) 0.5 = 6.48 d = (3*3+0*0+*+0*0+*+*+0*0+*+0*0+*) 0.5 =(7) 0.5 = 4. cos(d, d ) = 0.94

is the score of the 1 st student, x

is the score of the 1 st student, x 8 Chapter Collectg, Dsplayg, ad Aalyzg your Data. Descrptve Statstcs Sectos explaed how to choose a sample, how to collect ad orgaze data from the sample, ad how to dsplay your data. I ths secto, you wll