Least Squares Fttng of Data Davd Eberly Geoetrc Tools, LLC http://www.geoetrctools.co/ Copyrght c 1998-2014. All Rghts Reserved. Created: July 15, 1999 Last Modfed: February 9, 2008 Contents 1 Lnear Fttng of 2D Ponts of For x, fx)) 2 2 Lnear Fttng of nd Ponts Usng Orthogonal Regresson 2 3 Planar Fttng of 3D Ponts of For x, y, fx, y)) 3 4 Hyperplanar Fttng of nd Ponts Usng Orthogonal Regresson 4 5 Fttng a Crcle to 2D Ponts 4 6 Fttng a Sphere to 3D Ponts 6 7 Fttng an Ellpse to 2D Ponts 7 7.1 Dstance Fro Pont to Ellpse................................... 7 7.2 Mnzaton of the Energy Functon................................ 8 8 Fttng an Ellpsod to 3D Ponts 8 8.1 Dstance Fro Pont to Ellpsod.................................. 8 8.2 Mnzaton of the Energy Functon................................ 9 9 Fttng a Parabolod to 3D Ponts of the For x, y, fx, y)) 9 1
Ths docuent descrbes soe algorths for fttng 2D or 3D pont sets by lnear or quadratc structures usng least squares nzaton. 1 Lnear Fttng of 2D Ponts of For x, fx)) Ths s the usual ntroducton to least squares ft by a lne when the data represents easureents where the y-coponent s assued to be functonally dependent on the x-coponent. Gven a set of saples {x, y )}, deterne A and B so that the lne y = Ax + B best fts the saples n the sense that the su of the squared errors between the y and the lne values Ax + B s nzed. Note that the error s easured only n the y-drecton. Defne EA, B) = [Ax + B) y ] 2. Ths functon s nonnegatve and ts graph s a parabolod whose vertex occurs when the gradent satstfes E = 0, 0). Ths leads to a syste of two lnear equatons n A and B whch can be easly solved. Precsely, and so 0, 0) = E = 2 x2 x [Ax + B) y ]x, 1) x A = x y 1. B y The soluton provdes the least squares soluton y = Ax + B. 2 Lnear Fttng of nd Ponts Usng Orthogonal Regresson It s also possble to ft a lne usng least squares where the errors are easured orthogonally to the proposed lne rather than easured vertcally. The followng arguent holds for saple ponts and lnes n n densons. Let the lne be Lt) = td + A where D s unt length. Defne X to be the saple ponts; then X = A + d D + p D where d = D X A) and D s soe unt length vector perpendcular to D wth approprate coeffcent p. Defne Y = X A. The vector fro X to ts projecton onto the lne s Y d D = p D. The squared length of ths vector s p 2 = Y d D) 2. The energy functon for the least squares nzaton s EA, D) = p2. Two alternate fors for ths functon are and EA, D) = EA, D) = D T Y T [I DD T] Y ) [ ] ) Y Y )I Y Y T D = D T MA)D. 2
Usng the frst for of E n the prevous equaton, take the dervatve wth respect to A to get [ A = 2 I DD T] Y. Ths partal dervatve s zero whenever Y = 0 n whch case A = 1/) X the average of the saple ponts). Gven A, the atrx MA) s deterned n the second for of the energy functon. The quantty D T MA)D s a quadratc for whose nu s the sallest egenvalue of MA). Ths can be found by standard egensyste solvers. A correspondng unt length egenvector D copletes our constructon of the least squares lne. For n = 2, f A = a, b), then atrx MA) s gven by ) n MA) = x a) 2 + y b) 2 1 0 0 1 x a) 2 x a)y b) x. a)y b) y b) 2 For n = 3, f A = a, b, c), then atrx MA) s gven by 1 0 0 x a) 2 x a)y b) x a)z c) MA) = δ 0 1 0 x a)y b) y b) 2 y b)z c) 0 0 1 x a)z c) y b)z c) z c) 2 where δ = x a) 2 + y b) 2 + z c) 2. 3 Planar Fttng of 3D Ponts of For x, y, fx, y)) The assupton s that the z-coponent of the data s functonally dependent on the x- and y-coponents. Gven a set of saples {x, y, z )}, deterne A, B, and C so that the plane z = Ax + By + C best fts the saples n the sense that the su of the squared errors between the z and the plane values Ax +By +C s nzed. Note that the error s easured only n the z-drecton. Defne EA, B, C) = [Ax + By + C) z ] 2. Ths functon s nonnegatve and ts graph s a hyperparabolod whose vertex occurs when the gradent satstfes E = 0, 0, 0). Ths leads to a syste of three lnear equatons n A, B, and C whch can be easly solved. Precsely, 0, 0, 0) = E = 2 [Ax + By + C) z ]x, y, 1) and so x2 x y x x y x A x z y2 y B y = y z 1. C z The soluton provdes the least squares soluton z = Ax + By + C. 3
4 Hyperplanar Fttng of nd Ponts Usng Orthogonal Regresson It s also possble to ft a plane usng least squares where the errors are easured orthogonally to the proposed plane rather than easured vertcally. The followng arguent holds for saple ponts and hyperplanes n n densons. Let the hyperplane be N X A) = 0 where N s a unt length noral to the hyperplane and A s a pont on the hyperplane. Defne X to be the saple ponts; then X = A + λ N + p N where λ = N X A) and N s soe unt length vector perpendcular to N wth approprate coeffcent p. Defne Y = X A. The vector fro X to ts projecton onto the hyperplane s λ N. The squared length of ths vector s λ 2 = N Y ) 2. The energy functon for the least squares nzaton s EA, N) = λ2. Two alternate fors for ths functon are EA, N) = Y T [NN T] ) Y and ) EA, N) = N T Y Y T N = N T MA)N. Usng the frst for of E n the prevous equaton, take the dervatve wth respect to A to get [ A = 2 NN T] Y. Ths partal dervatve s zero whenever Y = 0 n whch case A = 1/) X the average of the saple ponts). Gven A, the atrx MA) s deterned n the second for of the energy functon. The quantty N T MA)N s a quadratc for whose nu s the sallest egenvalue of MA). Ths can be found by standard egensyste solvers. A correspondng unt length egenvector N copletes our constructon of the least squares hyperplane. For n = 3, f A = a, b, c), then atrx MA) s gven by x a) 2 x a)y b) x a)z c) MA) = x a)y b) y b) 2 y b)z c) x a)z c) y. b)z c) z c) 2 5 Fttng a Crcle to 2D Ponts Gven a set of ponts {x, y )}, 3, ft the wth a crcle x a)2 + y b) 2 = r 2 where a, b) s the crcle center and r s the crcle radus. An assupton of ths algorth s that not all the ponts are collnear. The energy functon to be nzed s Ea, b, r) = L r) 2 4
where L = x a) 2 + y b) 2. Take the partal dervatve wth respect to r to obtan Settng equal to zero yelds r = 2 L r). r = 1 L. Take the partal dervatve wth respect to a to obtan a = 2 L r) L a = 2 x a) + r L ) a and take the partal dervatve wth respect to b to obtan Settng these two dervatves equal to zero yelds and b = 2 L r) L b = 2 y b) + r L ). b a = 1 b = 1 x + r 1 y + r 1 Replacng r by ts equvalent fro / r = 0 and usng L / a = a x )/L and L / b = b y )/L, we get two nonlnear equatons n a and b: where a = x + L L a =: F a, b) b = ȳ + L L b =: Ga, b) x = 1 x ȳ = 1 y L = 1 L L a = 1 L b = 1 Fxed pont teraton can be appled to solvng these equatons: a 0 = x, b 0 = ȳ, and a +1 = F a, b ) and b +1 = Ga, b ) for 0. Warnng. I have not analyzed the convergence propertes of ths algorth. In a few experents t sees to converge just fne. a x L b y L L a L b. 5
6 Fttng a Sphere to 3D Ponts Gven a set of ponts {x, y, z )}, 4, ft the wth a sphere x a)2 + y b) 2 + z c) 2 = r 2 where a, b, c) s the sphere center and r s the sphere radus. An assupton of ths algorth s that not all the ponts are coplanar. The energy functon to be nzed s Ea, b, c, r) = L r) 2 where L = x a) 2 + y b) 2 + z c). Take the partal dervatve wth respect to r to obtan Settng equal to zero yelds r = 2 L r). r = 1 L. Take the partal dervatve wth respect to a to obtan a = 2 L r) L a = 2 x a) + r L ), a take the partal dervatve wth respect to b to obtan b = 2 L r) L b = 2 y b) + r L ), b and take the partal dervatve wth respect to c to obtan c = 2 L r) L c = 2 z c) + r L ). c Settng these three dervatves equal to zero yelds a = 1 x + r 1 and and b = 1 c = 1 y + r 1 z + r 1 Replacng r by ts equvalent fro / r = 0 and usng L / a = a x )/L, L / b = b y )/L, and L / c = c z )/L, we get three nonlnear equatons n a, b, and c: L a L b. L c. a = x + L L a =: F a, b, c) b = ȳ + L L b =: Ga, b, c) c = z + L L c =: Ha, b, c) 6
where x = 1 x ȳ = 1 y z = 1 z L = 1 L L a = 1 L b = 1 L c = 1 Fxed pont teraton can be appled to solvng these equatons: a 0 = x, b 0 = ȳ, c 0 = z, and a +1 = F a, b, c ), b +1 = Ga, b, c ), and c +1 = Ha, b, c ) for 0. Warnng. I have not analyzed the convergence propertes of ths algorth. In a few experents t sees to converge just fne. a x L b y L c z L 7 Fttng an Ellpse to 2D Ponts Gven a set of ponts {X }, 3, ft the wth an ellpse X U)T R T DRX U) = 1 where U s the ellpse center, R s an orthonoral atrx representng the ellpse orentaton, and D s a dagonal atrx whose dagonal entres represent the recprocal of the squares of the half-lengths lengths of the axes of the ellpse. An axs-algned ellpse wth center at the orgn has equaton x/a) 2 + y/b) 2 = 1. In ths settng, U = 0, 0), R = I the dentty atrx), and D = dag1/a 2, 1/b 2 ). The energy functon to be nzed s EU, R, D) = L r) 2 where L s the dstance fro X to the ellpse wth the gven paraeters. Ths proble s ore dffcult than that of fttng crcles. The dstance L requres fndng roots to a quartc polynoal. Whle there are closed for forulas for the roots of a quartc, these forulas are not easly anpulated algebracally or dfferentated to produce an algorth such as the one for a crcle. The approach nstead s to use an teratve nzer to copute the nu of E. 7.1 Dstance Fro Pont to Ellpse It s suffcent to solve ths proble when the ellpse s axs-algned. For other ellpses, they can be rotated and translated to an axs-algned ellpse centered at the orgn and the dstance can be easured n that syste. The basc dea can be found n Graphcs Ges IV an artcle by John Hart on coputng dstance between pont and ellpsod). Let u, v) be the pont n queston. Let the ellpse be x/a) 2 + y/b) 2 = 1. The closest pont x, y) on the ellpse to u, v) ust occur so that x u, y v) s noral to the ellpse. Snce an ellpse noral s x/a) 2 +y/b) 2 ) = x/a 2, y/b 2 ), the orthogonalty condton ples that u x = t x/a 2 and v y = t y/b 2 7
for soe t. Solvng yelds x = a 2 u/t + a 2 ) and y = b 2 v/t + b 2 ). Replacng n the ellpse equaton yelds ) 2 au + bv ) 2 = 1. t + a 2 t + b 2 Multplyng through by the denonators yelds the quartc polynoal F t) = t + a 2 ) 2 t + b 2 ) 2 a 2 u 2 t + b 2 ) 2 b 2 v 2 t + a 2 ) 2 = 0. The largest root t of the polynoal corresponds to the closest pont on the ellpse. The largest root can be found by a Newton s teraton schee. If u, v) s nsde the ellpse, then t 0 = 0 s a good ntal guess for the teraton. If u, v) s outsde the ellpse, then t 0 = ax{a, b} u 2 + v 2 s a good ntal guess. The teraton tself s t +1 = t F t )/F t ), 0. Soe nuercal ssues need to be addressed. For u, v) near the coordnate axes, the algorth s llcondtoned. You need to handle those cases separately. Also, f a and b are large, then F t ) can be qute large. In these cases you ght consder unforly scalng the data to O1) as floatng pont nubers frst, copute dstance, then rescale to get the dstance n the orgnal coordnates. 7.2 Mnzaton of the Energy Functon TO BE WRITTEN LATER. The code at the web ste uses a varaton on Powell s drecton set ethod.) 8 Fttng an Ellpsod to 3D Ponts Gven a set of ponts {X }, 3, ft the wth an ellpsod X U)T R T DRX U) = 1 where U s the ellpsod center and R s an orthonoral atrx representng the ellpsod orentaton. The atrx D s a dagonal atrx whose dagonal entres represent the recprocal of the squares of the half-lengths of the axes of the ellpsod. An axs-algned ellpsod wth center at the orgn has equaton x/a) 2 + y/b) 2 + z/c) 2 = 1. In ths settng, U = 0, 0, 0), R = I the dentty atrx), and D = dag1/a 2, 1/b 2, 1/c 2 ). The energy functon to be nzed s EU, R, D) = L r) 2 where L s the dstance fro X to the ellpse wth the gven paraeters. Ths proble s ore dffcult than that of fttng spheres. The dstance L requres fndng roots to a sxth degree polynoal. There are no closed forulas for the roots of such polynoals. The approach nstead s to use an teratve nzer to copute the nu of E. 8.1 Dstance Fro Pont to Ellpsod It s suffcent to solve ths proble when the ellpsod s axs-algned. For other ellpsods, they can be rotated and translated to an axs-algned ellpsod centered at the orgn and the dstance can be easured 8
n that syste. The basc dea can be found n Graphcs Ges IV an artcle by John Hart on coputng dstance between pont and ellpsod). Let u, v, w) be the pont n queston. Let the ellpse be x/a) 2 + y/b) 2 + z/c) 2 = 1. The closest pont x, y, z) on the ellpsod to u, v) ust occur so that x u, y v, z w) s noral to the ellpsod. Snce an ellpsod noral s x/a) 2 + y/b) 2 + z/c) 2 ) = x/a 2, y/b 2, z/c 2 ), the orthogonalty condton ples that u x = t x/a 2, v y = t y/b 2, and w z = t z/c 2 for soe t. Solvng yelds x = a 2 u/t + a 2 ), y = b 2 v/t + b 2 ), and z = c 2 w/t + c 2 ). Replacng n the ellpsod equaton yelds ) 2 ) 2 ) 2 au bv cw t + a 2 + t + b 2 + t + c 2 = 1. Multplyng through by the denonators yelds the sxth degree polynoal F t) = t + a 2 ) 2 t + b 2 ) 2 t + c 2 ) 2 a 2 u 2 t + b 2 ) 2 t + c 2 ) 2 b 2 v 2 t + a 2 ) 2 t + c 2 ) 2 c 2 w 2 t + a 2 ) 2 t + b 2 ) 2 = 0. The largest root t of the polynoal corresponds to the closest pont on the ellpse. The largest root can be found by a Newton s teraton schee. If u, v, w) s nsde the ellpse, then t 0 = 0 s a good ntal guess for the teraton. If u, v, w) s outsde the ellpse, then t 0 = ax{a, b, c} u 2 + v 2 + w 2 s a good ntal guess. The teraton tself s t +1 = t F t )/F t ), 0. Soe nuercal ssues need to be addressed. For u, v, w) near the coordnate planes, the algorth s llcondtoned. You need to handle those cases separately. Also, f a, b, and c are large, then F t ) can be qute large. In these cases you ght consder unforly scalng the data to O1) as floatng pont nubers frst, copute dstance, then rescale to get the dstance n the orgnal coordnates. 8.2 Mnzaton of the Energy Functon TO BE WRITTEN LATER. The code at the web ste uses a varaton on Powell s drecton set ethod.) 9 Fttng a Parabolod to 3D Ponts of the For x, y, fx, y)) Gven a set of saples {x, y, z )} and assung that the true values le on a parabolod z = fx, y) = p 1 x 2 + p 2 xy + p 3 y 2 + p 4 x + p 5 y + p 6 = P Qx, y) where P = p 1, p 2, p 3, p 4, p 5, p 6 ) and Qx, y) = x 2, xy, y 2, x, y, 1), select P to nze the su of squared errors EP) = P Q z ) 2 where Q = Qx, y ). The nu occurs when the gradent of E s the zero vector, E = 2 P Q z )Q = 0. 9
Soe algebra converts ths to a syste of 6 equatons n 6 unknowns: ) Q Q T P = z Q. The product Q Q T s a product of the 6 1 atrx Q wth the 1 6 atrx Q T, the result beng a 6 6 atrx. Defne the 6 6 syetrc atrx A = Q Q T and the 6 1 vector B = z Q. The choce for P s the soluton to the lnear syste of equatons AP = B. The entres of A and B ndcate suatons over the approprate product of varables. For exaple, sx 3 y) = x3 y : sx 4 ) sx 3 y) sx 2 y 2 ) sx 3 ) sx 2 y) sx 2 ) p 1 szx 2 ) sx 3 y) sx 2 y 2 ) sxy 3 ) sx 2 y) sxy 2 ) sxy) p 2 szxy) sx 2 y 2 ) sxy 3 ) sy 4 ) sxy 2 ) sy 3 ) sy 2 ) p 3 szy sx 3 ) sx 2 y) sxy 2 ) sx 2 = 2 ) ) sxy) sx) p 4 szx) sx 2 y) sxy 2 ) sy 3 ) sxy) sy 2 ) sy) p 5 szy) sx 2 ) sxy) sy 2 ) sx) sy) s1) sz) p 6 10