Least Squares Fitting of Data

Transcription

1 Least Squares Fttng of Data Davd Eberly Geoetrc Tools, LLC Copyrght c All Rghts Reserved. Created: July 15, 1999 Last Modfed: February 9, 2008 Contents 1 Lnear Fttng of 2D Ponts of For x, fx)) 2 2 Lnear Fttng of nd Ponts Usng Orthogonal Regresson 2 3 Planar Fttng of 3D Ponts of For x, y, fx, y)) 3 4 Hyperplanar Fttng of nd Ponts Usng Orthogonal Regresson 4 5 Fttng a Crcle to 2D Ponts 4 6 Fttng a Sphere to 3D Ponts 6 7 Fttng an Ellpse to 2D Ponts Dstance Fro Pont to Ellpse Mnzaton of the Energy Functon Fttng an Ellpsod to 3D Ponts Dstance Fro Pont to Ellpsod Mnzaton of the Energy Functon Fttng a Parabolod to 3D Ponts of the For x, y, fx, y)) 9 1

2 Ths docuent descrbes soe algorths for fttng 2D or 3D pont sets by lnear or quadratc structures usng least squares nzaton. 1 Lnear Fttng of 2D Ponts of For x, fx)) Ths s the usual ntroducton to least squares ft by a lne when the data represents easureents where the y-coponent s assued to be functonally dependent on the x-coponent. Gven a set of saples {x, y )}, deterne A and B so that the lne y = Ax + B best fts the saples n the sense that the su of the squared errors between the y and the lne values Ax + B s nzed. Note that the error s easured only n the y-drecton. Defne EA, B) = [Ax + B) y ] 2. Ths functon s nonnegatve and ts graph s a parabolod whose vertex occurs when the gradent satstfes E = 0, 0). Ths leads to a syste of two lnear equatons n A and B whch can be easly solved. Precsely, and so 0, 0) = E = 2 x2 x [Ax + B) y ]x, 1) x A = x y 1. B y The soluton provdes the least squares soluton y = Ax + B. 2 Lnear Fttng of nd Ponts Usng Orthogonal Regresson It s also possble to ft a lne usng least squares where the errors are easured orthogonally to the proposed lne rather than easured vertcally. The followng arguent holds for saple ponts and lnes n n densons. Let the lne be Lt) = td + A where D s unt length. Defne X to be the saple ponts; then X = A + d D + p D where d = D X A) and D s soe unt length vector perpendcular to D wth approprate coeffcent p. Defne Y = X A. The vector fro X to ts projecton onto the lne s Y d D = p D. The squared length of ths vector s p 2 = Y d D) 2. The energy functon for the least squares nzaton s EA, D) = p2. Two alternate fors for ths functon are and EA, D) = EA, D) = D T Y T [I DD T] Y ) [ ] ) Y Y )I Y Y T D = D T MA)D. 2

3 Usng the frst for of E n the prevous equaton, take the dervatve wth respect to A to get [ A = 2 I DD T] Y. Ths partal dervatve s zero whenever Y = 0 n whch case A = 1/) X the average of the saple ponts). Gven A, the atrx MA) s deterned n the second for of the energy functon. The quantty D T MA)D s a quadratc for whose nu s the sallest egenvalue of MA). Ths can be found by standard egensyste solvers. A correspondng unt length egenvector D copletes our constructon of the least squares lne. For n = 2, f A = a, b), then atrx MA) s gven by ) n MA) = x a) 2 + y b) x a) 2 x a)y b) x. a)y b) y b) 2 For n = 3, f A = a, b, c), then atrx MA) s gven by x a) 2 x a)y b) x a)z c) MA) = δ x a)y b) y b) 2 y b)z c) x a)z c) y b)z c) z c) 2 where δ = x a) 2 + y b) 2 + z c) 2. 3 Planar Fttng of 3D Ponts of For x, y, fx, y)) The assupton s that the z-coponent of the data s functonally dependent on the x- and y-coponents. Gven a set of saples {x, y, z )}, deterne A, B, and C so that the plane z = Ax + By + C best fts the saples n the sense that the su of the squared errors between the z and the plane values Ax +By +C s nzed. Note that the error s easured only n the z-drecton. Defne EA, B, C) = [Ax + By + C) z ] 2. Ths functon s nonnegatve and ts graph s a hyperparabolod whose vertex occurs when the gradent satstfes E = 0, 0, 0). Ths leads to a syste of three lnear equatons n A, B, and C whch can be easly solved. Precsely, 0, 0, 0) = E = 2 [Ax + By + C) z ]x, y, 1) and so x2 x y x x y x A x z y2 y B y = y z 1. C z The soluton provdes the least squares soluton z = Ax + By + C. 3

4 4 Hyperplanar Fttng of nd Ponts Usng Orthogonal Regresson It s also possble to ft a plane usng least squares where the errors are easured orthogonally to the proposed plane rather than easured vertcally. The followng arguent holds for saple ponts and hyperplanes n n densons. Let the hyperplane be N X A) = 0 where N s a unt length noral to the hyperplane and A s a pont on the hyperplane. Defne X to be the saple ponts; then X = A + λ N + p N where λ = N X A) and N s soe unt length vector perpendcular to N wth approprate coeffcent p. Defne Y = X A. The vector fro X to ts projecton onto the hyperplane s λ N. The squared length of ths vector s λ 2 = N Y ) 2. The energy functon for the least squares nzaton s EA, N) = λ2. Two alternate fors for ths functon are EA, N) = Y T [NN T] ) Y and ) EA, N) = N T Y Y T N = N T MA)N. Usng the frst for of E n the prevous equaton, take the dervatve wth respect to A to get [ A = 2 NN T] Y. Ths partal dervatve s zero whenever Y = 0 n whch case A = 1/) X the average of the saple ponts). Gven A, the atrx MA) s deterned n the second for of the energy functon. The quantty N T MA)N s a quadratc for whose nu s the sallest egenvalue of MA). Ths can be found by standard egensyste solvers. A correspondng unt length egenvector N copletes our constructon of the least squares hyperplane. For n = 3, f A = a, b, c), then atrx MA) s gven by x a) 2 x a)y b) x a)z c) MA) = x a)y b) y b) 2 y b)z c) x a)z c) y. b)z c) z c) 2 5 Fttng a Crcle to 2D Ponts Gven a set of ponts {x, y )}, 3, ft the wth a crcle x a)2 + y b) 2 = r 2 where a, b) s the crcle center and r s the crcle radus. An assupton of ths algorth s that not all the ponts are collnear. The energy functon to be nzed s Ea, b, r) = L r) 2 4

5 where L = x a) 2 + y b) 2. Take the partal dervatve wth respect to r to obtan Settng equal to zero yelds r = 2 L r). r = 1 L. Take the partal dervatve wth respect to a to obtan a = 2 L r) L a = 2 x a) + r L ) a and take the partal dervatve wth respect to b to obtan Settng these two dervatves equal to zero yelds and b = 2 L r) L b = 2 y b) + r L ). b a = 1 b = 1 x + r 1 y + r 1 Replacng r by ts equvalent fro / r = 0 and usng L / a = a x )/L and L / b = b y )/L, we get two nonlnear equatons n a and b: where a = x + L L a =: F a, b) b = ȳ + L L b =: Ga, b) x = 1 x ȳ = 1 y L = 1 L L a = 1 L b = 1 Fxed pont teraton can be appled to solvng these equatons: a 0 = x, b 0 = ȳ, and a +1 = F a, b ) and b +1 = Ga, b ) for 0. Warnng. I have not analyzed the convergence propertes of ths algorth. In a few experents t sees to converge just fne. a x L b y L L a L b. 5

6 6 Fttng a Sphere to 3D Ponts Gven a set of ponts {x, y, z )}, 4, ft the wth a sphere x a)2 + y b) 2 + z c) 2 = r 2 where a, b, c) s the sphere center and r s the sphere radus. An assupton of ths algorth s that not all the ponts are coplanar. The energy functon to be nzed s Ea, b, c, r) = L r) 2 where L = x a) 2 + y b) 2 + z c). Take the partal dervatve wth respect to r to obtan Settng equal to zero yelds r = 2 L r). r = 1 L. Take the partal dervatve wth respect to a to obtan a = 2 L r) L a = 2 x a) + r L ), a take the partal dervatve wth respect to b to obtan b = 2 L r) L b = 2 y b) + r L ), b and take the partal dervatve wth respect to c to obtan c = 2 L r) L c = 2 z c) + r L ). c Settng these three dervatves equal to zero yelds a = 1 x + r 1 and and b = 1 c = 1 y + r 1 z + r 1 Replacng r by ts equvalent fro / r = 0 and usng L / a = a x )/L, L / b = b y )/L, and L / c = c z )/L, we get three nonlnear equatons n a, b, and c: L a L b. L c. a = x + L L a =: F a, b, c) b = ȳ + L L b =: Ga, b, c) c = z + L L c =: Ha, b, c) 6

7 where x = 1 x ȳ = 1 y z = 1 z L = 1 L L a = 1 L b = 1 L c = 1 Fxed pont teraton can be appled to solvng these equatons: a 0 = x, b 0 = ȳ, c 0 = z, and a +1 = F a, b, c ), b +1 = Ga, b, c ), and c +1 = Ha, b, c ) for 0. Warnng. I have not analyzed the convergence propertes of ths algorth. In a few experents t sees to converge just fne. a x L b y L c z L 7 Fttng an Ellpse to 2D Ponts Gven a set of ponts {X }, 3, ft the wth an ellpse X U)T R T DRX U) = 1 where U s the ellpse center, R s an orthonoral atrx representng the ellpse orentaton, and D s a dagonal atrx whose dagonal entres represent the recprocal of the squares of the half-lengths lengths of the axes of the ellpse. An axs-algned ellpse wth center at the orgn has equaton x/a) 2 + y/b) 2 = 1. In ths settng, U = 0, 0), R = I the dentty atrx), and D = dag1/a 2, 1/b 2 ). The energy functon to be nzed s EU, R, D) = L r) 2 where L s the dstance fro X to the ellpse wth the gven paraeters. Ths proble s ore dffcult than that of fttng crcles. The dstance L requres fndng roots to a quartc polynoal. Whle there are closed for forulas for the roots of a quartc, these forulas are not easly anpulated algebracally or dfferentated to produce an algorth such as the one for a crcle. The approach nstead s to use an teratve nzer to copute the nu of E. 7.1 Dstance Fro Pont to Ellpse It s suffcent to solve ths proble when the ellpse s axs-algned. For other ellpses, they can be rotated and translated to an axs-algned ellpse centered at the orgn and the dstance can be easured n that syste. The basc dea can be found n Graphcs Ges IV an artcle by John Hart on coputng dstance between pont and ellpsod). Let u, v) be the pont n queston. Let the ellpse be x/a) 2 + y/b) 2 = 1. The closest pont x, y) on the ellpse to u, v) ust occur so that x u, y v) s noral to the ellpse. Snce an ellpse noral s x/a) 2 +y/b) 2 ) = x/a 2, y/b 2 ), the orthogonalty condton ples that u x = t x/a 2 and v y = t y/b 2 7

8 for soe t. Solvng yelds x = a 2 u/t + a 2 ) and y = b 2 v/t + b 2 ). Replacng n the ellpse equaton yelds ) 2 au + bv ) 2 = 1. t + a 2 t + b 2 Multplyng through by the denonators yelds the quartc polynoal F t) = t + a 2 ) 2 t + b 2 ) 2 a 2 u 2 t + b 2 ) 2 b 2 v 2 t + a 2 ) 2 = 0. The largest root t of the polynoal corresponds to the closest pont on the ellpse. The largest root can be found by a Newton s teraton schee. If u, v) s nsde the ellpse, then t 0 = 0 s a good ntal guess for the teraton. If u, v) s outsde the ellpse, then t 0 = ax{a, b} u 2 + v 2 s a good ntal guess. The teraton tself s t +1 = t F t )/F t ), 0. Soe nuercal ssues need to be addressed. For u, v) near the coordnate axes, the algorth s llcondtoned. You need to handle those cases separately. Also, f a and b are large, then F t ) can be qute large. In these cases you ght consder unforly scalng the data to O1) as floatng pont nubers frst, copute dstance, then rescale to get the dstance n the orgnal coordnates. 7.2 Mnzaton of the Energy Functon TO BE WRITTEN LATER. The code at the web ste uses a varaton on Powell s drecton set ethod.) 8 Fttng an Ellpsod to 3D Ponts Gven a set of ponts {X }, 3, ft the wth an ellpsod X U)T R T DRX U) = 1 where U s the ellpsod center and R s an orthonoral atrx representng the ellpsod orentaton. The atrx D s a dagonal atrx whose dagonal entres represent the recprocal of the squares of the half-lengths of the axes of the ellpsod. An axs-algned ellpsod wth center at the orgn has equaton x/a) 2 + y/b) 2 + z/c) 2 = 1. In ths settng, U = 0, 0, 0), R = I the dentty atrx), and D = dag1/a 2, 1/b 2, 1/c 2 ). The energy functon to be nzed s EU, R, D) = L r) 2 where L s the dstance fro X to the ellpse wth the gven paraeters. Ths proble s ore dffcult than that of fttng spheres. The dstance L requres fndng roots to a sxth degree polynoal. There are no closed forulas for the roots of such polynoals. The approach nstead s to use an teratve nzer to copute the nu of E. 8.1 Dstance Fro Pont to Ellpsod It s suffcent to solve ths proble when the ellpsod s axs-algned. For other ellpsods, they can be rotated and translated to an axs-algned ellpsod centered at the orgn and the dstance can be easured 8

9 n that syste. The basc dea can be found n Graphcs Ges IV an artcle by John Hart on coputng dstance between pont and ellpsod). Let u, v, w) be the pont n queston. Let the ellpse be x/a) 2 + y/b) 2 + z/c) 2 = 1. The closest pont x, y, z) on the ellpsod to u, v) ust occur so that x u, y v, z w) s noral to the ellpsod. Snce an ellpsod noral s x/a) 2 + y/b) 2 + z/c) 2 ) = x/a 2, y/b 2, z/c 2 ), the orthogonalty condton ples that u x = t x/a 2, v y = t y/b 2, and w z = t z/c 2 for soe t. Solvng yelds x = a 2 u/t + a 2 ), y = b 2 v/t + b 2 ), and z = c 2 w/t + c 2 ). Replacng n the ellpsod equaton yelds ) 2 ) 2 ) 2 au bv cw t + a 2 + t + b 2 + t + c 2 = 1. Multplyng through by the denonators yelds the sxth degree polynoal F t) = t + a 2 ) 2 t + b 2 ) 2 t + c 2 ) 2 a 2 u 2 t + b 2 ) 2 t + c 2 ) 2 b 2 v 2 t + a 2 ) 2 t + c 2 ) 2 c 2 w 2 t + a 2 ) 2 t + b 2 ) 2 = 0. The largest root t of the polynoal corresponds to the closest pont on the ellpse. The largest root can be found by a Newton s teraton schee. If u, v, w) s nsde the ellpse, then t 0 = 0 s a good ntal guess for the teraton. If u, v, w) s outsde the ellpse, then t 0 = ax{a, b, c} u 2 + v 2 + w 2 s a good ntal guess. The teraton tself s t +1 = t F t )/F t ), 0. Soe nuercal ssues need to be addressed. For u, v, w) near the coordnate planes, the algorth s llcondtoned. You need to handle those cases separately. Also, f a, b, and c are large, then F t ) can be qute large. In these cases you ght consder unforly scalng the data to O1) as floatng pont nubers frst, copute dstance, then rescale to get the dstance n the orgnal coordnates. 8.2 Mnzaton of the Energy Functon TO BE WRITTEN LATER. The code at the web ste uses a varaton on Powell s drecton set ethod.) 9 Fttng a Parabolod to 3D Ponts of the For x, y, fx, y)) Gven a set of saples {x, y, z )} and assung that the true values le on a parabolod z = fx, y) = p 1 x 2 + p 2 xy + p 3 y 2 + p 4 x + p 5 y + p 6 = P Qx, y) where P = p 1, p 2, p 3, p 4, p 5, p 6 ) and Qx, y) = x 2, xy, y 2, x, y, 1), select P to nze the su of squared errors EP) = P Q z ) 2 where Q = Qx, y ). The nu occurs when the gradent of E s the zero vector, E = 2 P Q z )Q = 0. 9

10 Soe algebra converts ths to a syste of 6 equatons n 6 unknowns: ) Q Q T P = z Q. The product Q Q T s a product of the 6 1 atrx Q wth the 1 6 atrx Q T, the result beng a 6 6 atrx. Defne the 6 6 syetrc atrx A = Q Q T and the 6 1 vector B = z Q. The choce for P s the soluton to the lnear syste of equatons AP = B. The entres of A and B ndcate suatons over the approprate product of varables. For exaple, sx 3 y) = x3 y : sx 4 ) sx 3 y) sx 2 y 2 ) sx 3 ) sx 2 y) sx 2 ) p 1 szx 2 ) sx 3 y) sx 2 y 2 ) sxy 3 ) sx 2 y) sxy 2 ) sxy) p 2 szxy) sx 2 y 2 ) sxy 3 ) sy 4 ) sxy 2 ) sy 3 ) sy 2 ) p 3 szy sx 3 ) sx 2 y) sxy 2 ) sx 2 = 2 ) ) sxy) sx) p 4 szx) sx 2 y) sxy 2 ) sy 3 ) sxy) sy 2 ) sy) p 5 szy) sx 2 ) sxy) sy 2 ) sx) sy) s1) sz) p 6 10