Advanced 7 Bio-Inspired System

Transcription

1 Advanced 7 Bio-Inspired System Dept. of Mechanical Science and Engineering Nagoya University

2 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. New method for Slippage Degree Estimation 4. Experimental Results 5. Conclusion

3 Background Grasp of object by robot hand Unknown Physical quantities Weight Friction coefficient Tactile information Grip force Slippage degree Collapsing Unknown Object Object Acceptable range Calculation Perfect sticking Partial slippage Macroscopic slippage Grip force Humans : Large Acceptable force Small Control grip force by tactile information of slippage. Can estimate from slippage degree

4 Purpose Conventional method estimates the slippage degree in simply cases. Obinata, G., Ashis, D., Watanabe, N., and Moriyama, N. (2007). Vision Based Tactile Sensor Using Transparent Elastic Fingertip for Dexterous Handling. In Kolski, S. (ed.) Mobile Robots: Perception & Navigation, pp when Extend Estimation of slippage degree in following cases Case Case2 Case3 sticking occurred on deformed surface moment is applied normal force is changed Noises on images used for estimation Reduction of accuracy Reduction of effect of noises

5 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. New method for Slippage Degree Estimation 4. Experimental Results 5. Conclusion

6 Structure of Vision-Based Tactile Sensor LED lights CCD camera Transparent acrylic plate Elastic body Object Calculation of dot positions, 2 contact Columnsarea 2 Rows Acquisition of tactile information Contact force (normal, tangential, moment) Transparent silicon rubber Slippage degree

7 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. New method for Slippage Degree Estimation 4. Experimental Results 5. Conclusion

8 Estimation of Slippage Degree Partial slippage by non linear pressure distribution Sensor Object Normal force N Tangential force F Contact region S c Stick region S s F<N S Incipient slippage region F>N Stick ratio indicates slippage degree. Stick ratio S s c = = =0. 0 =0 0 Perfect sticking Partial slippage Macroscopic slippage Keeping > 0 Preventing object from slipping

9 Conventional Method to Estimate Stick Ratio Discrimination of stick/slippage region d Contact region Initial image Displacement d k : Each dot d 0 : Central dot Object d 0 d 0 d k d k Current image In stick region In slippage region

10 Estimation of with Distorted Stick Region (Case) The stick region is distorted. After the macroscopic slippage Change of moving direction of object Increase of the normal force Reference image : STATIC UPDATED d d Incorrect Correct Initial image (static) Current image Updated Reference image Reference image is selected from repository image I R (i) Current image is newly stored as the I R (i+) when d i U ( i 2,, 3 ) max min,... d max (i) : Maximum displacement of dot between I R (i) and current image U min : Threshold R

11 Estimation of with Distorted Stick Region (Case) First point : Required size of displacement Prob bable ra ange of d 0 d k k Dots in slippage region are hardly detected. U max d 0 d 0 Displacement of Object d k d k Reference image is selected such that In stick region In slippage region data variation of dots d max : Maximum displacement of dot in stick region d max U max d max > U max (threshold)

12 Estimation of with Distorted Stick Region (Case) Second point : Change of moving direction y Current image I R () I R (2) d max (i) Movement trajectory of object N i : Largest number of I R y I R (N i ) I R (i C ) I R (N i ) d max (i) I I R (N i -) R (i) I R R( (i) Stick region is reconstructed (a) Rectilinear movement x d N d N d N 2 d N d N d i d i max i max i max i max i (b) Moving direction changes max i max C max Reference image is selected such that moving direction of object doesn t change significantly during interval of two images. Selection of reference image I(i ref ) i ref i i sup : d U, i max max i C N i x C

13 Estimation of with Applied Moment (Case2) Displacements in stick region are not equable. d k : large d k : small :Reference image :Current image :Central dot d 0 Cancel of Rotation In stick region d 0 and d k are greatly different. Condition in stick region d Not satisfied d 0 k 2 Estimation of Rotation angle Rotation of all dots in current image Condition in stick region d d Satisfied d 0 k

14 Flow Chart of New Method to Estimate Capture Measure Current image Contact region, Dot positions Canceling Rotation No Yes Calculate Calculation Displacements of dots of dots between between reference and current images Second point If d max >U Calculation d 0 d k Ss N s S c N c First point Reference image Current image

15 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. New method for Slippage Degree Estimation 4. Experimental Results 5. Conclusion

16 Experimental Description LED lights CCD camera Transparent acrylic plate Normal force Transparent elastic body Moment Tangential force Object Apply Normal force Estimation of Tangential force stick ratio Moment

17 Estimation of with Distorted Stick Region (Case) Increase of stick ratio Good in the opposite direction central dot Not good in the same direction continuously Slippage direction macroscopic slippage Elastic body was severely deformed. Dots can hardly move to that direction.

18 Experimental Movie with Distorted Stick Region

19 Estimation of with Applied Moment (Case2) Can estimate Good in the opposite direction central dot Slippage direction Not good in the same direction continuously macroscopic slippage

20 Experimental Movie with Applied Moment Indicates rotation angle of object

21 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. New method for Slippage Degree Estimation 4. Experimental Results 5. Conclusion

22 Conclusion This paper proposed Estimation method of stick ratio in some major situations Denoising method to decrease data variation of dots Our sensor was developed to a practical level. Sense Slippage degree in many cases Normal force Tangential force Moment Future work Implementation of developed sensor to robot hand to verify proposed method

23 Thank you for your kind attention.

24 Another Approach for Estimation of Slippage Degree Not good in the same direction continuously Slippage direction Another approach When macroscopic slippage F 0 : Tangential force F0 0 N F F0 0 : Normal force N0 N N 0 Current F : Tangential force F F F 0 N : Normal force N N N0 Prediction of macroscopic slippage Perfect contact or partial slippage Macroscopic slippage

25 Estimation of with Normal Force Changed (Case3) Change of normal force Contact surface Change of distance CCD camera In perspective transformation spread toward outside. Dots gather toward inside. Compensation of perspective transformation ti p post k p c c pre k : Scale parameter between two images c : Central position of image p pre : dot position before compensation p post : dot position after compensation Increase of normal force

26 Calculation of Scale Parameter Sum of all distances of adjacent two dots in contact region Comparing pref p ( x, y) (,),(0,),(,),(,0) k, l ( x, y ) (,),(0,),(,),(,0),,, ) k, l p k, l curk, l p ref kx, l y curkx, l y Reference image Current image Influence by deformation of contact region Line segment Average direction of between two dots displacement Parallel : High-influence Vertical : Low-influence Unit vector of displacement direction q x, y p pref p pref curk, l k, l curkx, l y kx, l y pcur pref p pref k, l k, l curkx, l y kx, l y Elimination of parallel component Decrease of influence ( x, y) (,),(0,),(,),(,0) k, l p ref k, l ref kx, l y p p curk, l curk x, l y ( x, y) (,),(0,),(,),(,0) k, l p q q x, y x, y

27 Estimation of with Normal Force Changed (Case3) Increase of stick region with contact region central dot

28 Experimental Movie with a Normal Force Changed

29 Reference dot d 0 d k Displacement of central dot Displacement of reference dot Reference dot:the nearest dot from the center of the stick region The central dot is not always in the stick region. Since the reference dot moves along the stick region movement, we easily obtain the stick ratio even when the contact surface is moving.

30 Detail of d U max d max U In order to discriminate stick and slippage regions in stable sensitivity, d max should be considered to define. d d 0 k 0 d U max Probab ble rang ge of d 0 d k 0 U d 0 d k d k 0 Difference between reference and current images d 0 In stick region In slippage region d max U d max

31 Method to decrease of data variation of dots Variations in the dot positions from noises weighted average of target dot and the adjacent 8 dots 2,,2,, l and k p w p k l k k m l l m n pre m n m l k post ,,,,,, S S S S S S S S S w w w w w w w w w l k l k l k l k l k l k,2,2,,, l or k p w p l pre k k m l m n m l k post 3 2 3,,, S S S w w w l k l k l k Weight target dot Experimental Results target dot Standard deviation of relative dot position in reference to central dot Maximum 0.4 pixel 0.06 pixel p p

32 Selection of 0 0 is small. The sensitivity of estimation is good. 0 > dot variations To minimize the effect of the dot variations Regardless ess reference e e dot standard < 0.06 pixel deviation max = 0.06 pixel 95.44% Relative displacement < 0.24 pixel (±2 max ) 0 = 4 max = 0.24 pixel

33 Selection of U Two measures for U S n s i i The earlier the decrease of is, the smaller S is. Fast response of f estimation U should be small. D ns i2 i i 0 n s : the step number (58) ( ) i ( Regarded as the level e of estimation error Desirable U, S and D are small. i i i i ) Nearly constant D S U = 2.0 pixel

34 Estimation of Friction Coefficient k ratio atio ated stick stic ra mated estima s estim Polypropylene (=.0) Polyethylene (=.5) Stick ratio to ratio between two forces tangential force / normal force tangential force / normal force Estimation? Leather (=.33) Friction coefficient

35 Example of Application of Reference Image Second point the image captured several steps before Reference image stick ra atio Initial image Initial 初期画像 image step before 4 ステップ前 steps before stick ra atio 初期画像 0.2 ステップ前 step step o stick rati Initial 初期画像 image 6ステップ前 0.8 初期画像 6 steps before Initial image 0 ステップ前 0.6 stick ratio 0 steps before step 0.4 Not appropriate step

36 Forecasted Question Size of sensor Curvature factor :3mm Height :3mm Elastic body Equation of Rotation p postk cos0 sin sin0 cos 0 0 p p pre p 0 pre0 pre k p k : Each dot position p 0 : Central dot position pre: Before rotation post: After rotation Processing speed The processing speed depends on spec of PC. Currently, the processing speed about 4Hz.

37 Forecasted Question Selection of material Selection points Hard (for measure of contact force in wide range) Transparent (for capture of image from reverse side) Flexible (for increase of moving range of dots) Silicon rubber

38 Experimental Movie with Forces and a Moment

39 Measurement of Contact Force Normal force Tangential force Moment Contact radius (Contact t region is circle.) Relation between normal force and contact t radius Refrence Xydas, Kao:Modeling of contact mechanics and friction limit surface for soft fingers in robotics with experimental results,international te a Journal of Robotics Research,Vol.8,,No. 8,pp (999).

40 Measurement of Contact Force Normal force Tangential force Moment nom malized displ acement (m mm) tangential force (N) displacements Relation between tangential force and ddisplacements of fdots

41 Measurement of Contact Force Normal force Tangential force Moment 2 rotation angle nom malized an ngle (deg.) torque (Nmm) Relation between moment and rotation ti angle of contact t surface

42 Forecasted Question 省略した内容も説明して 連続した同じ方向の巨視滑りに対する解決策は? 閾値はどうやって決めたの? 閾値は可変にしなくていいの? 物性に依存しないの? 対象物との摩擦係数で φ の検出可能範囲が変わるよね? 力と変位の関係は弾性体の物性に依存するよね? 固着率の真値は? 摩擦係数は取得できるの? 摩擦係数じゃ器用な把持は出来ないの? 接触方向は垂直のみに限られるの? 転がり動作は無理? 他研究のセンサの滑り検出も 限られた場合しか出来ないの? 並進と回転が同時に起こっても測定できるの? なぜ滑り予知の推定が必要なの? 回転の中心はどうやって求めるの? 何故中心ドットとの相対変位なの? 物体の表面は平面じゃないと駄目なの? 動画にmacroscopic slippage が起こらないのでは?

43 Forecasted Question 力と変位の関係は弾性体の物性に依存するよね? The relation between tangential force/moment and the displacement of dot depends on the elastic coefficient of the elastic body. 対象物との摩擦係数で φ の検出可能範囲が変わるよね? The smaller the friction coefficient is, the bigger the minimum value of the estimated stick ratio is. It is one of the future works. However, the previous research shows that the range of the stick ratio is not so decreased if the friction coefficient is large in some measure (more than ). 摩擦係数じゃ器用な把持は出来ないの? I think that robot can grasp the unknown object by using the friction coefficient in some measure. However, we cannot use the coulomb friction law for the elastic body. And it is difficult to use the law when a moment is applied. Moreover, it is well known that the friction coefficient is sensitive to several factors such as conditions of contact surface.

44 Forecasted Question 閾値は可変にしなくていいの? 物性に依存しないの? (It is desirable that 0 is small because the sensitivity of the slippage becomes higher.) It is enough if 0 is a size that the effect of the dot variations is hardly occurred. Therefore, 0 depends on the image resolution and number of pixels of the image, but doesn t depend on the property of the object. The results of two measures for selection of U depend on the grasped object, but we obtain similar results from other objects. The estimated stick ratio is not so different even though U is changed. Therefore, these thresholds are not variable. However, It is considered that the thresholds become variable to change of the sensitivity for any purpose. 固着率の真値は? For example, in nanometer scale, all region of the contact surface should slip even if macroscopic slippage doesn t occur. So as to use the slippage degree effectively for the grip force control of hands, we have to define the border between sticking and slipping. Therefore, the true value is not exist and we should consider the optimal definition of the slippage degree for dexterous handling.

45 Forecasted Question 他研究のセンサの滑り検出も 限られた場合しか出来ないの? There are not many researches which consider the slippage degree. And as far as I can see, only our sensor can estimate the slippage degree in many cases introduced in this presentation. In addition, only our sensor can estimate the contact force and slippage degree simultaneously. なぜ滑り予知の推定が必要なの? When the sensor cannot estimate the slippage degree, the robots may drop the grasped object due to delay of the control because the sensor cannot respond until macroscopic slippage occurs. In addition, after the macroscopic slippage, the bigger grip force is required to stop the slippage since the friction coefficient decreases in the dynamic situation. It is desirable to respond before the macroscopic slippage occurs.

46 Forecasted Question 回転の中心はどうやって求めるの? In the stick region, we obtain the same rotation angle regardless of used dots because dots in the stick region move and rotate uniformly. And it is considered that the central dot remains in the stick region to the last moment until the stick ratio becomes to 0. Therefore, we obtain the rotation angle in reference to central dot 何故中心ドットとの相対変位なの? It is considered that the central dot remains in the stick region to the last moment until the stick ratio becomes to 0. Therefore, we estimate t the slippage degree from the displacements of dots to the central dot. (However, This is conventional concept. In this paper, we propose the reference dot instead of the central dot.) 物体の表面は平面じゃないと駄目なのな? For the curved surface whose curvature factor is small, we can estimate. For the surface whose curvature factor is large, we will try as the future work.

47 Forecasted Question 動画に macroscopic slippage が起こらないのでは? Yes. There are two reasons why the stick ratio doesn t become 0. First reason is that N s doesn t become 0 because N s includes the central dot. Second reason is the accuracy of the estimation. The dots close to central dot hardly move in reference to central dot. However, the important thing is that we should control the grip force before the stick ratio becomes too small. Ss N S c N s c

48 Estimation of with Distorted Stick Region (Case) Second point : Estimation timing Estimate t only when d max >U d max : Maximum displacement of dot in stick region U : threshold value d max : grows bigger when contact surface moves faster tangentially d 0 d k Hardly satisfied, when d max is small

49 Estimation of with Applied Moment (Case2) A long stick picking task Sensor body and the stick rotate together Most dots are regarded in slippage region ( d d ) 0 k After canceling the rotation component obtain SD in the same way Rotation angle 0 4 Rotation equation p post k cos0 sin0 sin 0 cos 0 p pre k p pre p 0 post0 3 :Reference image :Current image :Rotation center 4 2 p k : Each dot position p 0 : Central dot position pre: Before rotation post: After rotation

50 Another Approach for Estimation of Slippage Degree Not good in the same direction continuously Slippage direction Another approach Comparing Measure when macroscopic slippage F 0 F Normal force N F :Current tangential force 0 N0 N N :Current normal force Tangential force F 0 Prediction of macroscopic slippage

51 Shape Sensing by Vision-Based Tactile Sensor for Dexterous Handling of Robot Hands

52 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Shape Sensing Method 4. Experimental Results 5. Conclusion

53 Background Dexterous handling of object by robot hand Tactile information Contact force Slippage Shape/irregularity Temperature etc Tactile Tactile Sense Sense Object receptors sensor Feed back Object Temperature etc Actuator Vision-based tactile sensor has desirable structure.* Object Elastic body Camera capturing deformation Tactile information Analyzing Problems of other sensors (resistive, capacitive, piezoelectric, etc) Many devices and wiring are needed. Elastic body decreases sensitivity. Vision-based sensors doesn t have such problems. *:K. Kamiyama (IEEE Computer Graphics and Applications, 2005), S. Saga (Sensor Review, 2007)

54 Purpose Sense of object shape/irregularity allows to... Archive tasks requiring shape recognition Choose grasping strategy Provide detailed d information to methods sensing other tactile information etc Captured image Image processing 2D deformation Unstable Stable 3D shape of Outshoot contact surface Sensing method Helpful to sense Tactile information Problem of shape sensing by vision-based sensor Using only single camera for compactness Purpose Obtain 3D information of contact surface from 2D single image Estimating 3D shape/irregularity of objects by single camera

55 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Shape Sensing Method 4. Experimental Results 5. Conclusion

56 Structure of Vision-Based Tactile Sensor One-way mirror CCD camera LED lights Touch pad Transparent acrylic plate Translucent red water Object Analysis of dot positions contact region Metal plate Two-layered elastic membranes Outer : Black color Inner : White color, Printed dotted pattern Acquisition of tactile information Contact force* (normal, tangential, moment) Slippage degree** Object shape/irregularity *:G. Obinata (Mobile Robots: Perception & Navigation, 2007), **:Y. Ito (IEEE SENSORS, 2009)

57 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Shape Sensing Method 4. Experimental Results 5. Conclusion

58 Fundamental Principle of Shape Sensing Color in image changes depending on shape of touch pad. Captured image Change of color Side view of touch pad Change of shape

59 Fundamental Principle of Shape Sensing Main content of shape sensing Definition : Region V i is projected on Pixel P i in image. Significant relationship Depth of touch pad Dependency Light intensity in V i (Input signal of CCD) Proportionality Color intensities (RGB values) of P i CCD camera Refraction acto LED light Acrylic plate We can calculate shape of pad from RGB values. V i Depth Membrane

60 Formulation of Shape Sensing Method. Finding relationship among color, light intensity and depth 2. Eliminating parameters 3. Determining unknown parameters 4. Compensating Approximation errors

61 Relationship among Color, Light Intensity and Depth LED light reflects off membrane and travels into CCD I 0, I 2, I 3, I 4 : Intensity per unit area of LED light LED light Acrylic plate I 0 I 3 into CCD I 4 c : Extinction coefficient in colored water l l 2 coefficient R 2 : Transmission I I 2 z axis to acrylic plate R (,b) : Reflection coefficient to membrane b : Color of inner membrane Membrane

62 Relationship among Color, Light Intensity and Depth LED light reflects off red pigment and travels into CCD I 2, I 3, I 4 : Intensity per unit area of LED light into CCD LED light I 4 Acrylic plate Intensity of scattered/absorbed light II I 0 3 z 2 I 2 z 2 I z I z dz Particle of red pigment z R 2 : Transmission coefficient to acrylic plate R ( ) : Scattering coefficient to red pigment Membrane

63 Relationship among Color, Light Intensity and Depth Sum of light intensity I sum in region V i S : Area on membrane in V i S (z) : Area on plane vertical to z axis in V i at z h 2 CCD camera Geometrical relationship (Detail is abbreviated.) LED light h s : Proportional constant h 2 : Obtained from h Applying to RGB components V i z S (z) b R, b G, b B :RGB values in image R, G, B, R, G, B : Propotional constants t S Membrane l l z axis

64 Formulation of Shape Sensing Method. Finding relationship among color, light intensity and depth 2. Eliminating parameters 3. Determining unknown parameters 4. Compensating Approximation errors

65 Eliminating Parameters Approximation of scattering coefficients R ( ) R component of light is scattered. G/B component of light is absorbed. Red pigment Elimination of R depending angle and color b of membrane Solving simultaneous equations of R and G components (R and B) z axis Membrane Q, Q 2 : Polynomial function of, c R, h 2 and k

66 Formulation of Shape Sensing Method. Finding relationship among color, light intensity and depth 2. Eliminating parameters 3. Determining unknown parameters 4. Compensating Approximation errors

67 Determining Unknown Parameters Determining g( ) by using measured values Parameter identification of c R, c G and Known l (t 0 ), b R (t 0 ), b G (t 0 ) : Previously measured initial values Previously measured Solving simultaneous equations c R, c G and are obtained. Equation to calculate pad depth l from color intensities b R, b G

68 Formulation of Shape Sensing Method. Finding relationship among color, light intensity and depth 2. Eliminating parameters 3. Determining unknown parameters 4. Compensating Approximation errors

69 Compensation for Approximation Error Effect of neighboring light Initial shape Deformed shape Light in V i Neighboring light l z axis Neighboring light increases light intensity in V i. l z axis Neighboring light travels in different direction. Calculating in reference to initial shape Assumption : Error of depth l depends on Deformation of membrane Pad depth in reference to initial shape

70 Compensation for Approximation Error Parameters to represent degree of deformation/depth z axis Direction to line CCD B line C sum : Deformation of membrane Small area : Difference from initial shape line B line A line A r : Threshold value m : Scale parameter depending on depth Difference from initial shape Effect of neighboring light is maximized. Equation to compensate depth l Compensated depth : m 0 0, n : Threshold value

71 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Shape Sensing Method 4. Experimental Results 5. Conclusion

72 Experimental Description Tactile Sensor Estimation of shape of touch pad Estimated shape Object Side view Touch pad Comparing Actual shape Side camera

73 Result of Shape of Non-Contact Touch Pad Pa ad de epth l [mm m] At Actual shape Estimated shape Estimated shape with compensation x [mm] Mean error [mm] Standard deviation [mm] Without compensation With compensation

74 Result of Shape of Contacting Touch Pad () Pa ad de epth l [mm m] At Actual shape Estimated shape Estimated shape with compensation x [mm] Mean error [mm] Standard deviation [mm] Without compensation With compensation

75 mm] Pad depth l [ Result of Shape of Contacting Touch Pad (2) Actual shape Estimated shape Estimated shape with compensation x [mm] Without compensation With compensation Mean error [mm] Standard deviation [mm] [mm] Pad depth l Actual shape Estimated shape Estimated shape with compensation x [mm] Without compensation With compensation Mean Standard error deviation [mm] [mm]

76 Experimental Movie

77 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Shape Sensing Method 4. Experimental Results 5. Conclusion

78 Conclusion This paper proposed Method to obtain shape/irregularity of object by using only single camera Our sensor was developed to practical level. Sense Normal force Tangential force Moment Slippage degree Shape/irregularity Future work Implementation of developed sensor to robot hand to verify proposed method

79 Thank you for your kind attention.

80 Estimation Result of Irregularity of Object l [mm] Es stimated depth Stand dard devi iation [m mm] Contact with flat surface Contact with asperity Contact with flat surface Contact with asperity x [mm]

81 Limitation of Shape of Object Pad depth l [mm m] Actual shape Estimated shape Estimated edshape pewith compensation o x [mm] Without compensation Mean error [mm] With compensation Standard deviation [mm]

82 Limitation of Shape of Object Pad depth l [mm m] Actual shape Estimated shape Estimated shape with compensation x [mm] Without compensation Mean error [mm] With compensation Standard deviation [mm]

83 Limitation of Shape of Object Pad depth l [mm m] Actual shape Estimated shape Estimated shape with compensation x [mm] Without compensation Mean error [mm] With compensation Standard deviation [mm]

84 Calculation of Area S (z) h 2 CCD camera C Similar triangle h Acrylic plate z 2 l 2 V i z S (z) is proportional to square of (z+h 2 ). S z s z 2 h 2 S Membrane S (z) l z axis

85 Calculation of Area S (z) S z sz 2 h 2 h 2 is obtained by these relations. h x 22 cos 2 cos 2 n f h cos cos sin 2 sin 2 sin 2 cos 2 sin cos cos 2 sin 2 cos sin sin 2 2 cos 2 sin cos n f : the relative refractive coefficient between the air and the colored water h 2 h 2 2 x 2 V i similar triangle C CCD camera Refraction of light Acrylic plate

86 Parameter Identification of c R, c G, c R, c G, : 3 Unknown parameters b Q G exp cg kl br exp cr kl c expc kl Q l, c R t, b t b t f l 0 R 0, R 2 R G 0 Solving by numerical analytical approach such as Newton method Preliminarily measured l l l t, b t, b t R G t2, br t2, bg t2 t, b t b t 3 R 3, G 3 c R, c G and are set to mm -, mm - and , respectively.

87 Parameter Identification of r, m 0, n, y, t x, y, t x, y, t 2 2 m sum x k k k k 0 k m expnl x, y, t l x, y t lx y, t l x, y, t m sum x, y, t 0, 0 x k x, y k y r l x, y, t la x, y, t m sum x, y, t Comparing with ln ln S Transformed into double logarithmic equation l a : actual depth l x, y, t la x, y, t L m x, y, t ln x, y, t ln m nl x, y, t l x, y, t F sum 2 L F sum The least-square method (changing r from to 50) r, m , and n are set to mm/rad, 30 pixel and mm. 0 0

88 Solving Equation for l kl c b kl c b exp exp 0 0 0,,,, t b t b t l f g t b t b t l f G R G R R R R R R G G G R c l Q kl c c Q kl c b kl c b b b l f, exp exp exp,, 2 We approximate l by using bisection method. 0 t, l app,,, max i t f l i t l i t l sig i app app,,, t b t b i t l f G R app, 2,, f sig i app app 0 0 0,,, t b t b i t l f t b t b t l f i t f G R G R G R app sig l d N tt 4 d6 ti l 0 0 0,,,,, t b t b t l f t b t b i t l f G R G R app N t l t l app, l max and N are set to 4 mm and 6, respectively.

89 Measurement of Contact Force Normal force Tangential force Moment Contact radius (Contact t region is circle.) Relation between normal force and contact t radius Refrence Xydas, Kao:Modeling of contact mechanics and friction limit surface for soft fingers in robotics with experimental results,international te a Journal of Robotics Research,Vol.8,,No. 8,pp (999).

90 Measurement of Contact Force Normal force Tangential force Moment nom malized displ acement (m mm) tangential force (N) displacements Relation between tangential force and ddisplacements of fdots

91 Measurement of Contact Force Normal force Tangential force Moment 2 rotation angle nom malized an ngle (deg.) torque (Nmm) Relation between moment and rotation ti angle of contact t surface

92 Estimation of Slippage Degree Partial slippage by non linear pressure distribution Sensor Object Normal force N Tangential force F Contact region S c Stick region S s F<N S Incipient slippage region F>N Stick ratio indicates slippage degree. Stick ratio S s c = =0 0.5 =0 0. =0 0 Perfect sticking Partial slippage Macroscopic slippage Keeping > 0 Preventing object from slipping

93 Stick Ratio Estimation Method Discrimination of stick/slippage region d Contact region Initial image Current image Displacement d k : Each dot d 0 : Central dot Object d 0 d 0 d k d k In stick region In slippage region Stick ratio Ss S c N N s c N s : Number of dots in stick region N c : Number of dots in contact region

94 Forecasted Question Size of sensor 640 pixel CCD camera : mm pixel LED light : mm Processing speed Curvature radious : 20 mm Height : 3 mm Thickness of membrane : mm The processing speed depends on spec of PC. Although the processing speed is about Hz currently, we can process the method faster by using high spec PC in the future.

95 Forecasted Question 水平方向の分解能は? どれくらいの形状まで測れるの? 接触領域内しか推定できないんじゃないの? 接触領域はどうやって推定するの? 何故赤色にしたの? 液体じゃないと駄目なの? 同軸照明じゃないと駄目なの? なぜシリコーンゴム? 膜の厚みはどう影響する? Vision-based 以外のセンサってどんなのがあるの?electrical resistance, capacitance, electromagnetic component, piezoelectric/ ultrasonic/ component, strain gauge

96 Forecasted Question 接触領域内しか推定できないんじゃないの? Although we can estimate the entire shape of the touch pad, the estimation of the object shape is confined to the contact region. Therefore, the estimation of the contact region is also important. 接触領域はどうやって推定するの? We can estimate the contact region by using the shape of the touch pad. We will present the method to estimate the contact region at the IEEE international conference IROS in October. 何故赤色にしたの? We can estimate if we use blue or green water. However, If we use the other colored water, it is difficult to estimate, because we want to eliminate the two scattering coefficient by approximation. And the resolution depends on the difference between cr and cg. Therefore, it is desirable that cr is small and cg is large. なぜシリコーンゴム? The silicon rubber is hardly influenced by the environment.

97 Forecasted Question Vision-based 以外のセンサってどんなのがあるの? electrical resistance, capacitance, electromagnetic component, piezoelectric/ ultrasonic/ component, strain gauge. 分解能は? The resolution of br and bg are 0.00 (0~255 value ) by using smoothing (filter mask is pixel). When depth changes.7 mm, br and bg changes 8 and 3.5. Therefore, the resolution of depth is about.7/(3.5/0.00)= )= [mm]. However the actual resolution is not determined because of

98 Theory for Intensity of Light Sum of light intensity I sum in region V i S : Area on membrane in V i S (z) ) : Area on plane vertical to z axis in V i at z Geometrical relationship s : Proportional constant h 2 : Obtained from h Definition of parameter (Detail is abbreviated.) LED light h 2 h CCD camera Acrylic plate z 2 l 2 V i S (z) z S Membrane l l z axis

99 Modification of Equation Substitution of RGB values in the image Q exp I sum SR, b R2I 0 exp ckl sr R2 I0 Q c : Extinction coefficient c ckl 2 l, c exp ckl Absorbedb c R c G c B Scattered R, G, B, R, G, B : Propotional constants b R, b G, b B:RGB values in image

100 Modification of Equation Elimination of R (, b) depending and b b b R G, b R 2 I 0 R R exp crkl Qc R exp crkl R 2I 0 R R Q2 l, cr exp crkl, b R I exp c kl SR exp kl sr R SR 2 0 G G p G Solving simultaneous equations : Angle of membrane b : Color of inner membrane

101 Modification of Equation Equation to obtain pad depth l from color intensities b R, b G b R expc Rkl bg exp cg kl g Q cr expckl Q2l, cr Solving for g( ) : Calculated from position of P i k, h 2 : Obtained from, h c R, c G, : Preliminarily calculated (abbreviation) Relation between pad depth l and color intensities b R, b G l (t 0 ), b R (t 0 ), b G (t 0 ) : Preliminarily measured

102 Compensating Approximation Error Effect of neighboring light Initial shape Deformed shape Light in V i l z axis Neighboring light increases light intensity in V i. l z axis Neighboring light travels in different direction. Neighboring light Error of l depends on shape of membrane and pad depth. Equation to compensate depth l (Detail is abbreviated.) sum : Function of shape of membrane l : Compensated depth m : Function of pad depth

103 Acquisition of Tactile Information by Vision-based Tactile Sensor for Dexterous Handling of Robot Hands

104 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Contact Region Estimation 4. Object Location Estimation 5. Experimental Results 6. Conclusion

105 Background Dexterous handling of object by robot hand Tactile information Contact force Slippage Shape/irregularity Temperature etc Tactile Tactile Sense Sense Object receptors sensor Feed back Object Temperature etc Actuator Requirement of tactile sensors Simultaneous acquisition iti of multifunctional/compact many types of tactile information robot hand Many sensors and other conventional devices can obtain only single type of tactile information. cannot obtain Our method can obtain. contact region on sensor even if shape/movement 2. location of contacted object of object are complex.

106 Purpose Contact region on sensor allows to... evaluate grasp task stability estimate object shape in accurate manner evaluate contact t pressure Force Sensor Which is object shape? = Contact pressure Shape-sensing Contact region Contact region Large pressure may Stable injure sensor Unstable or object. Location of contacted t object is needed d because... manipulation tasks require exact position/angle of object deformation of elastic sensor body makes object location uncertain Purpose Efficient method to Acquire contact region and location of object

107 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Contact Region Estimation 4. Object Location Estimation 5. Experimental Results 6. Conclusion

108 Structure of Vision-Based Tactile Sensor One-way mirror CCD camera LED lights Touch pad Transparent acrylic plate Translucent red water Object Analysis of dot positions color intensity Metal plate Two-layered elastic membranes Outer : Black color Inner : White color, Printed dotted pattern Acquisition of tactile information Contact force* (normal, tangential, moment) Degree of slippage** Shape/irregularity of object *:G. Obinata (Mobile Robots: Perception & Navigation, 2007), **:Y. Ito (IEEE SENSORS, 2009)

109 Shape-sensing Method by Our Proposed Sensor Principle of shape-sensing method*** Significant relationship Depth of touch pad Dependency Light intensity in V i (Input signal of CCD) Proportionality Color intensities (RGB values) Calculation of pad shape from RGB values. ***:Y. Ito (IEEE CASE, 200) CCD camera Refraction LED light Acrylic plate Region V i Depth Shape-sensing method is used in this study. Membrane

110 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Contact Region Estimation 4. Object Location Estimation 5. Experimental Results 6. Conclusion

111 Theory of Contact Region Estimation Difficulty to estimate contact region from shape information Pad shapes are similar. Contact t regions are different. Analysis of small segment of membrane () () R() p Definition : R =RR Segment of membrane R() () ( ) () Equilibrium equation of segment () () () Center of segment P s () 2 R R 2 p p : inner pressure : tensional forces per unit length R : curvature radius 0 2 rad

112 Theory of Contact Region Estimation Curvature radius of small segment P Concave/flat segment R() () () p R =RR 0 R 2 R R 2 R 0 R() () () ( ) p 0 p 0 Not satisfied P is in contact region Necessary and sufficient condition for P to belong to contact region Touch pad Convex segment () P R =R R () R() p () () R() Assumption : Object surfaces are convex/flat. R 0 Ex) R Flat Convex P is in non-contact region 0 0 rad

113 Calculation of Curvature Radius Curvature radius defined by three reference points y z (x P, y P, 0) P 2 x P v Acrylic plate u-v plane : perpendicular to x-y plane C (x :originofu P, y P, 0) of u-v coordinate R Assumption : u P Shape of segment around P is spherical in u-v plane. Pad shape : obtained by shape-sensing method R r P 2 (r, v(r)) C (c u,, c v, ) v r : constant parameter v(u) ( ) : v value at u r u P (r, v(r)) P(0, v(0)) Substituting 3 reference points P, P, P 2 u c 2 v c 2 R x, y 2 u, v, P P, Discrimination of convex/concave shape R R R v v 0 0 cv, 0 cv, Calculation of R of all segments in various u-v planes

114 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Contact Region Estimation 4. Object Location Estimation 5. Experimental Results 6. Conclusion

115 Estimation of Object Position Analysis of dots painted on sensor surface : contacting dot : non-contacting dot Contacting dots move/rotate with object. Contact reference dot (CR dot) is always in contact. CR dot : whose displacement from initial position is biggest. CR dot Biggest displacement When CR dot doesn t slip to object, Object s CR dot s displacement = displacement rotation angle = rotation angle Initial pad shape Deformed pad shape Dot s positions/displacements are calculated by shape-sensing method.

116 Estimation of Object Orientation Rotation angle of CR dot are obtained from rotation matrix R 3 basis s vectors n, n 2 and n 3 around aou dcr dot q q 2 n q q n2 q2 q2 p i, jk pi, jk k p i k, j pik, j k n 3 n n2 : average of : average of p i,j p i,j p i,j pi p i,j p i,j p i,j p i+,j+ CR dot p i,j p i,j p i,j : position of dot (i-th from left and j-th from top) Assumption : Eight dots adjacent to CR dot are in contact with object. Positional relation of nine dots is not changed. n n n n m n n2 R n2 n n n 3 3 m m m m R n : rotation matrix from m to n m, n : sampling index

117 Estimation of Object Orientation Divide of rotation angle m n m R R n m z R n m y R n x m n z, m n y, m n x Obtain Consideration of change of CR dot CR dot changes. m R n x, m R n y, m R n z : rotation matrix around x, y, z-direction m n x, m n y, m n z : rotation angle around x, y, z-direction : CR dot : contacting dot : non-contacting dot Not satisfied m n m n m m obj x n obj y n obj z m m n obj : rotation angle of object Rotation angles of object = Sums of rotation angles of CR dot at each step m n obj x n n n k k m n k k m n,, k x obj y y obj z km km km k z (when CR dot doesn t slip to object) m x n y n z

118 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Contact Region Estimation 4. Object Location Estimation 5. Experimental Results 6. Conclusion

119 Experimental Description Tactile Sensor Proposed method calculates contact region, object location Comparing Touch pad Actual contact t region Object Actual position/angle of object is obtained Side camera

120 Estimation Result of Contact Region Object contacts with square/ring shaped object Estimated contact region Shape of object Estimated contact region Shape of object y (mm) y (mm) x (mm) x (mm)

121 Experimental Movie of Contact Region Estimation

122 Experimental Movie of Contact Region Estimation

123 Estimation Result of Object Position Object moves on x-z plane without rotation. x-c coordin ate valu ue (mm m) z-co oordina ate valu ue (mm m) Estimated position Actual position 0 Estimated position Actual position Time in number

124 Estimation Result of Rotation Angle of Rolling Object Object rolls on sensor surface. Ang gle arou und x-ax xis (deg g) y value of CR dot (mm) Eti Estimated tdangle Actual angle Time in number

125 Experimental Movie of Object Location Estimation

126 Table of Contents. Introduction 2. Vision Based i dt Tactile Sensor 3. Contact Region Estimation 4. Object Location Estimation 5. Experimental Results 6. Conclusion

127 Conclusion This paper proposed method to obtain Contact region to evaluate grasp stability, object shape and contact pressure Object location to archive complex manipulation tasks Our sensor was developed to more practical level. Normal force Tangential force Sense Moment Degree of slippage Shape/irregularity Contact region Object location Future work Implementation of developed sensor to robot hand to verify proposed method

128 Thank you for your kind attention.

129 Estima ated Depth h l [mm] Estimation Result Estimated inner shape Estimated outer shape Estima ated Depth l [mm] Estimated inner shape Estimated outer shape x [mm][ ] x [mm] 40 Contact region 40 Contact region y (mm) x (mm) x (mm) y (mm)30 20

130 Estimation Result of Object Orientation An ngle aro ound x-a axis (de eg) Estimated angle 5 Actual angle Time in number

131 Simultaneous Acquisition An ngle aroun nd x-axis (deg) y value of CR dot (mm m) y5 0 Estimated angle Actual angle Time in number Time in number=0 Time in number=20 Time in number=30 Time in number=40 Time in number=50 Time in number=60

132 Calculation of rotation angle x, y, z Divide of rotation angle R R z R R z y y x x cosz sinz 0 cos y 0 sin y 0 0 sin z cosz cos x sin x 0 0 sin y 0 cos y 0 sinx cos x cos z cos y sinz cos y sin y cos sin sin sin cos sin sin sin cos cos z z y y y x x cos sin x z z x x cos z sin y cos x sin z sin x sinz sin y cos x cosz sin x cos y cos x. We can obtain y. 2. After obtaining y, we can calculate x, z.

133 Calculation for Theory of Contact Region Estimation Calculation of curvature radius of each segment We assume that the contact surface of the object is convex/flat. Although we regard this segment as in the non contact region, it may satisfy the condition when R (θ) is a small negative number. Therefore, when R (x, y, θ) is negative, we set R (x (x, y, θ) to- that yields /R (x (x, y, θ)=0. R x, y, When R (x (x, y, )<0, R x, y, 0 R() () p () () R() () R () =R() R (+/2) =R(+/2)

134 Consideration of Calculation Error Modification of condition for P to belong to contact region Actual shape Estimated shape R 0 R 0 Condition for point P to be in contact region R x, y, 0 rad R x, y, 2 x, y min E 2, f R R Smaller are desirable. n n x, y f i R x, y, R i R n x, y 2 x y f min E f f, R x, y, 0 E R : average values of /R R : standard deviation of /R Not satisfied when touch pad doesn t contact E f : average values of f f : standard deviation of f when touch pad doesn t contact, is decreased. f R

135 Measurement of Contact Force Normal force Tangential force Moment Contact radius (Contact t region is circle.) Relation between normal force and contact t radius Refrence Xydas, Kao:Modeling of contact mechanics and friction limit surface for soft fingers in robotics with experimental results,international te a Journal of Robotics Research,Vol.8,,No. 8,pp (999).

136 Measurement of Contact Force Normal force Tangential force Moment nom malized displ acement (m mm) tangential force (N) displacements Relation between tangential force and ddisplacements of fdots

137 Measurement of Contact Force Normal force Tangential force Moment 2 rotation angle nom malized an ngle (deg.) torque (Nmm) Relation between moment and rotation ti angle of contact t surface

138 Estimation of Slippage Degree Partial slippage by non linear pressure distribution Sensor Object Normal force N Tangential force F Contact region S c Stick region S s F<N S Incipient slippage region F>N Stick ratio indicates slippage degree. Stick ratio S s c = =0 0.5 =0 0. =0 0 Perfect sticking Partial slippage Macroscopic slippage Keeping > 0 Preventing object from slipping

139 Stick Ratio Estimation Method Discrimination of stick/slippage region d Contact region Initial image Current image Displacement d k : Each dot d 0 : Central dot Object d 0 d 0 d k d k In stick region In slippage region Stick ratio Ss S c N N s c N s : Number of dots in stick region N c : Number of dots in contact region

140 Forecasted Question Size of sensor 640 pixel CCD camera : mm pixel LED light : mm Processing speed Curvature radious : 20 mm Height : 3 mm Thickness of membrane : mm Although the processing speed is about 2Hz currently, we can process the method faster by using high spec PC and parallel computing method such as CUDA.

141 Forecasted Question 回転と平行移動が重なったらどうなるの?When the movement and rotation occur simultaneously, it is hard to define the position of the object. For example, when we define the object position as the position of the weighted center of the object, we cannot estimate the displacement with the rotation. なぜシリコーンゴム?The silicon rubber is hardly influenced by the environment. Vision-based 以外のセンサってどんなのがあるの?electrical resistance, capacitance, electromagnetic component, piezoelectric/ ultrasonic/ component, strain gauge. 接触領域分解能は?We estimate the contact region, changing the center point P of the segment by 5pixel (0.4mm). It is the resolution. When we estimate changing P by smaller distance, the resolution is increased. 物体位置分解能は?The resolution of the object location depends on the shape estimation of the touch pad. the resolution of the estimated shape is about μm but mean error is 0.5 mm. Therefore, the resolution of the object position is μm but maximum error is about 0.5 mm. 物体角度分解能は?The resolution of the rotation angle is about 0.04 deg which is calculated from the resolution of the pad shape and the equation to acquire the angle. (Distance between 2 dots is.5 mm and resolution of shape is μm.) But maximum error is about 0 deg. This is due to the error of the shape estimation.

142 Forecasted Question 一般性は? スケーラブル?The shape and size of the touch pad is arbitrary. However, the sensor requires the resolution of the CCD camera to detect the position of dots. Therefore, when we change the CCD camera, we have to change the number of dots. And when we estimate the force and slippage, the elastic body must have the convex surface. r はどうやって決める?r is determined as.92 mm (25pixel) which was obtained experimentally. The size of r has trade-off between the variation of the calculated curvature radius and the estimation accuracy for a sharp curve.