Least Squares Fttng of Data Davd Eberly Geoetrc Tools, LLC http://www.geoetrctools.co/ Copyrght c 1998-2015. All Rghts Reserved. Created: July 15, 1999 Last Modfed: January 5, 2015 Contents 1 Lnear Fttng of 2D Ponts of For x, fx)) 2 2 Lnear Fttng of nd Ponts Usng Orthogonal Regresson 2 3 Planar Fttng of 3D Ponts of For x, y, fx, y)) 3 4 Hyperplanar Fttng of nd Ponts Usng Orthogonal Regresson 4 5 Fttng a Crcle to 2D Ponts 5 6 Fttng a Sphere to 3D Ponts 6 7 Fttng an Ellpse to 2D Ponts 7 8 Fttng an Ellpsod to 3D Ponts 8 9 Fttng a Parabolod to 3D Ponts of the For x, y, fx, y)) 8 1
Ths docuent descrbes soe algorths for fttng 2D or 3D pont sets by lnear or quadratc structures usng least squares nzaton. 1 Lnear Fttng of 2D Ponts of For x, fx)) Ths s the usual ntroducton to least squares ft by a lne when the data represents easureents where the y-coponent s assued to be functonally dependent on the x-coponent. Gven a set of saples {x, y )}, deterne A and B so that the lne y = Ax + B best fts the saples n the sense that the su of the squared errors between the y and the lne values Ax + B s nzed. Note that the error s easured only n the y-drecton. Defne EA, B) = [Ax + B) y ] 2. Ths functon s nonnegatve and ts graph s a parabolod whose vertex occurs when the gradent satstfes E = 0, 0). Ths leads to a syste of two lnear equatons n A and B whch can be easly solved. Precsely, 0, 0) = E = 2 [Ax + B) y ]x, 1) and so x2 x x 1 A B = x y y. The soluton provdes the least squares soluton y = Ax + B. If pleented drectly, ths forulaton can lead to an ll-condtoned lnear syste. To avod ths, you should frst copute the averages x = x and ȳ = y and subtract the fro the data, x x x and y y ȳ. The ftted lne s of the for y ȳ = Ax x). 2 Lnear Fttng of nd Ponts Usng Orthogonal Regresson It s also possble to ft a lne usng least squares where the errors are easured orthogonally to the proposed lne rather than easured vertcally. The followng arguent holds for saple ponts and lnes n n densons. Let the lne be Lt) = td + A where D s unt length. Defne X to be the saple ponts; then X = A + d D + p D where d = D X A) and D s soe unt length vector perpendcular to D wth approprate coeffcent p. Defne Y = X A. The vector fro X to ts projecton onto the lne s Y d D = p D. The squared length of ths vector s p 2 = Y d D) 2. The energy functon for the least squares nzaton s EA, D) = p2. Two alternate fors for ths functon are EA, D) = Y T 2 [I DD T] Y )
and EA, D) = D T [ ] ) Y Y )I Y Y T D = D T MA)D. Usng the frst for of E n the prevous equaton, take the dervatve wth respect to A to get [ A = 2 I DD T] Y. Ths partal dervatve s zero whenever Y = 0 n whch case A = 1/) X the average of the saple ponts). Gven A, the atrx MA) s deterned n the second for of the energy functon. The quantty D T MA)D s a quadratc for whose nu s the sallest egenvalue of MA). Ths can be found by standard egensyste solvers. A correspondng unt length egenvector D copletes our constructon of the least squares lne. For n = 2, f A = a, b), then atrx MA) s gven by ) n MA) = x a) 2 + y b) 2 1 0 0 1 x a) 2 x a)y b) x. a)y b) y b) 2 For n = 3, f A = a, b, c), then atrx MA) s gven by 1 0 0 x a) 2 x a)y b) x a)z c) MA) = δ 0 1 0 x a)y b) y b) 2 y b)z c) 0 0 1 x a)z c) y b)z c) z c) 2 where δ = x a) 2 + y b) 2 + z c) 2. 3 Planar Fttng of 3D Ponts of For x, y, fx, y)) The assupton s that the z-coponent of the data s functonally dependent on the x- and y-coponents. Gven a set of saples {x, y, z )}, deterne A, B, and C so that the plane z = Ax + By + C best fts the saples n the sense that the su of the squared errors between the z and the plane values Ax +By +C s nzed. Note that the error s easured only n the z-drecton. Defne EA, B, C) = [Ax + By + C) z ] 2. Ths functon s nonnegatve and ts graph s a hyperparabolod whose vertex occurs when the gradent satstfes E = 0, 0, 0). Ths leads to a syste of three lnear equatons n A, B, and C whch can be easly solved. Precsely, 0, 0, 0) = E = 2 [Ax + By + C) z ]x, y, 1) 3
and so x2 x y x x y y2 y x y 1 A B C = x z y z z. The soluton provdes the least squares soluton z = Ax + By + C. If pleented drectly, ths forulaton can lead to an ll-condtoned lnear syste. To avod ths, you should frst copute the averages x = x, ȳ = y, and z = z, and then subtract the fro the data, x x x, y y ȳ, and z z z. The ftted plane s of the for z z = Ax x)+by ȳ). 4 Hyperplanar Fttng of nd Ponts Usng Orthogonal Regresson It s also possble to ft a plane usng least squares where the errors are easured orthogonally to the proposed plane rather than easured vertcally. The followng arguent holds for saple ponts and hyperplanes n n densons. Let the hyperplane be N X A) = 0 where N s a unt length noral to the hyperplane and A s a pont on the hyperplane. Defne X to be the saple ponts; then X = A + λ N + p N where λ = N X A) and N s soe unt length vector perpendcular to N wth approprate coeffcent p. Defne Y = X A. The vector fro X to ts projecton onto the hyperplane s λ N. The squared length of ths vector s λ 2 = N Y ) 2. The energy functon for the least squares nzaton s EA, N) = λ2. Two alternate fors for ths functon are EA, N) = Y T [NN T] Y ) and ) EA, N) = N T Y Y T N = N T MA)N. Usng the frst for of E n the prevous equaton, take the dervatve wth respect to A to get [ A = 2 NN T] Y. Ths partal dervatve s zero whenever Y = 0 n whch case A = 1/) X the average of the saple ponts). Gven A, the atrx MA) s deterned n the second for of the energy functon. The quantty N T MA)N s a quadratc for whose nu s the sallest egenvalue of MA). Ths can be found by standard egensyste solvers. A correspondng unt length egenvector N copletes our constructon of the least squares hyperplane. 4
For n = 3, f A = a, b, c), then atrx MA) s gven by x a) 2 x a)y b) x a)z c) MA) = x a)y b) y b) 2 y b)z c) x a)z c) y b)z c) z c) 2. 5 Fttng a Crcle to 2D Ponts Gven a set of ponts {x, y )}, 3, ft the wth a crcle x a)2 + y b) 2 = r 2 where a, b) s the crcle center and r s the crcle radus. An assupton of ths algorth s that not all the ponts are collnear. The energy functon to be nzed s Ea, b, r) = L r) 2 where L = x a) 2 + y b) 2. Take the partal dervatve wth respect to r to obtan Settng equal to zero yelds r = 2 L r). r = 1 L. Take the partal dervatve wth respect to a to obtan a = 2 L r) L a = 2 x a) + r L ) a and take the partal dervatve wth respect to b to obtan b = 2 L r) L b = 2 y b) + r L ). b Settng these two dervatves equal to zero yelds a = 1 x + r 1 and b = 1 y + r 1 Replacng r by ts equvalent fro / r = 0 and usng L / a = a x )/L and L / b = b y )/L, we get two nonlnear equatons n a and b: a = x + L L a =: F a, b) b = ȳ + L L b =: Ga, b) L a L b. 5
where x = 1 x ȳ = 1 y L = 1 L L a = 1 L b = 1 Fxed pont teraton can be appled to solvng these equatons: a 0 = x, b 0 = ȳ, and a +1 = F a, b ) and b +1 = Ga, b ) for 0. Warnng. I have not analyzed the convergence propertes of ths algorth. In a few experents t sees to converge just fne. a x L b y L 6 Fttng a Sphere to 3D Ponts Gven a set of ponts {x, y, z )}, 4, ft the wth a sphere x a)2 + y b) 2 + z c) 2 = r 2 where a, b, c) s the sphere center and r s the sphere radus. An assupton of ths algorth s that not all the ponts are coplanar. The energy functon to be nzed s Ea, b, c, r) = L r) 2 where L = x a) 2 + y b) 2 + z c). Take the partal dervatve wth respect to r to obtan Settng equal to zero yelds r = 2 L r). r = 1 L. Take the partal dervatve wth respect to a to obtan a = 2 L r) L a = 2 x a) + r L ), a take the partal dervatve wth respect to b to obtan b = 2 L r) L b = 2 y b) + r L ), b and take the partal dervatve wth respect to c to obtan c = 2 L r) L c = 2 z c) + r L ). c 6
Settng these three dervatves equal to zero yelds and and a = 1 b = 1 c = 1 x + r 1 y + r 1 z + r 1 Replacng r by ts equvalent fro / r = 0 and usng L / a = a x )/L, L / b = b y )/L, and L / c = c z )/L, we get three nonlnear equatons n a, b, and c: where L a L b. L c. a = x + L L a =: F a, b, c) b = ȳ + L L b =: Ga, b, c) c = z + L L c =: Ha, b, c) x = 1 x ȳ = 1 y z = 1 z L = 1 L L a = 1 L b = 1 L c = 1 Fxed pont teraton can be appled to solvng these equatons: a 0 = x, b 0 = ȳ, c 0 = z, and a +1 = F a, b, c ), b +1 = Ga, b, c ), and c +1 = Ha, b, c ) for 0. Warnng. I have not analyzed the convergence propertes of ths algorth. In a few experents t sees to converge just fne. a x L b y L c z L 7 Fttng an Ellpse to 2D Ponts Gven a set of ponts {X }, 3, ft the wth an ellpse X U)T R T DRX U) = 1 where U s the ellpse center, R s an orthonoral atrx representng the ellpse orentaton, and D s a dagonal atrx whose dagonal entres represent the recprocal of the squares of the half-lengths lengths of the axes of the ellpse. An axs-algned ellpse wth center at the orgn has equaton x/a) 2 + y/b) 2 = 1. In ths settng, U = 0, 0), R = I the dentty atrx), and D = dag1/a 2, 1/b 2 ). The energy functon to be nzed s EU, R, D) = 7 L r) 2
where L s the dstance fro X to the ellpse wth the gven paraeters. Ths proble s ore dffcult than that of fttng crcles. The dstance L s coputed accordng to the algorth descrbed n Dstance fro a Pont to an Ellpse, an Ellpsod, or a Hyperellpsod. The functon E s nzed teratvely usng Powell s drecton-set ethod to search for a nu. An pleentaton s GteApprEllpse2.h. 8 Fttng an Ellpsod to 3D Ponts Gven a set of ponts {X }, 3, ft the wth an ellpsod X U)T R T DRX U) = 1 where U s the ellpsod center and R s an orthonoral atrx representng the ellpsod orentaton. The atrx D s a dagonal atrx whose dagonal entres represent the recprocal of the squares of the half-lengths of the axes of the ellpsod. An axs-algned ellpsod wth center at the orgn has equaton x/a) 2 + y/b) 2 + z/c) 2 = 1. In ths settng, U = 0, 0, 0), R = I the dentty atrx), and D = dag1/a 2, 1/b 2, 1/c 2 ). The energy functon to be nzed s EU, R, D) = L r) 2 where L s the dstance fro X to the ellpse wth the gven paraeters. Ths proble s ore dffcult than that of fttng spheres. The dstance L s coputed accordng to the algorth descrbed n Dstance fro a Pont to an Ellpse, an Ellpsod, or a Hyperellpsod. The functon E s nzed teratvely usng Powell s drecton-set ethod to search for a nu. An pleentaton s GteApprEllpsod3.h. 9 Fttng a Parabolod to 3D Ponts of the For x, y, fx, y)) Gven a set of saples {x, y, z )} and assung that the true values le on a parabolod z = fx, y) = p 1 x 2 + p 2 xy + p 3 y 2 + p 4 x + p 5 y + p 6 = P Qx, y) where P = p 1, p 2, p 3, p 4, p 5, p 6 ) and Qx, y) = x 2, xy, y 2, x, y, 1), select P to nze the su of squared errors EP) = P Q z ) 2 where Q = Qx, y ). The nu occurs when the gradent of E s the zero vector, E = 2 P Q z )Q = 0. Soe algebra converts ths to a syste of 6 equatons n 6 unknowns: ) Q Q T P = z Q. 8
The product Q Q T s a product of the 6 1 atrx Q wth the 1 6 atrx Q T, the result beng a 6 6 atrx. Defne the 6 6 syetrc atrx A = Q Q T and the 6 1 vector B = z Q. The choce for P s the soluton to the lnear syste of equatons AP = B. The entres of A and B ndcate suatons over the approprate product of varables. For exaple, sx 3 y) = x3 y : sx 4 ) sx 3 y) sx 2 y 2 ) sx 3 ) sx 2 y) sx 2 ) p 1 szx 2 ) sx 3 y) sx 2 y 2 ) sxy 3 ) sx 2 y) sxy 2 ) sxy) p 2 szxy) sx 2 y 2 ) sxy 3 ) sy 4 ) sxy 2 ) sy 3 ) sy 2 ) p 3 szy sx 3 ) sx 2 y) sxy 2 ) sx 2 = 2 ) ) sxy) sx) p 4 szx) sx 2 y) sxy 2 ) sy 3 ) sxy) sy 2 ) sy) p 5 szy) sx 2 ) sxy) sy 2 ) sx) sy) s1) sz) p 6 9