Scripting Languages Fast development, extensible programs

Transcription

1 Scripting Languages Fast development, extensible programs Devert Alexandre School of Software Engineering of USTC November 30, 2012 Slide 1/60

2 Table of Contents 1 Introduction 2 Dynamic languages A Python tour Functions as objects Monkey patching 3 Languages comparison 4 Tools making Content generation 5 Glue code Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 2/60

3 Introduction You probably heard about those programming languages Javascript Perl Python Lua Ruby... Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 3/60

4 Scripting languages They are scripting languages, sharing common features Highly dynamic Automatic memory management Can compile to a virtual machine Can be embedded Can be extended with modules written in C or C++ Cross-platform Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 4/60

5 Scripting languages Scripting languages tends to help you to code faster less code to do the same things with other languages powerful default types very dynamic languages large, complete standard library Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 5/60

6 Usages They are successful for a number of applications tool-making languages glue languages to write glue code building extensible, open-ended software... Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 6/60

7 Table of Contents 1 Introduction 2 Dynamic languages A Python tour Functions as objects Monkey patching 3 Languages comparison 4 Tools making Content generation 5 Glue code Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 7/60

8 A Python tour Let s have a look at a scripting language, Python. What follows is true for most other scripting languages. Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 8/60

9 Friendly syntax List and dictionaries are parts of the language syntax a = [ 1, 2, 3, 4, 5, s i x, 7 ] b = { name : a p p l e, c o l o r : r e d, c o s t : } c = [ [ 1. 5, 3. 0 ], [ 2.0, 5. 0 ], [ 9.7, 5. 2 ] ] d = { f r u i t s : [ a p p l e, orange, peach ], l o c a t i o n : Suzhou } Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 9/60

10 Friendly syntax List manipulations are part of the languages a = [ 1, 2, 3, 4, 5, 7, 8, 9, 1 0 ] p r i n t a [ 2 : 7 ] >>> [ 3, 4, 5, 7, 8 ] p r i n t a [ 2 : ] >>> [ 3, 4, 5, 7, 8, 9, 1 0 ] p r i n t a [ : 5 ] >>> [ 1, 2, 3, 4, 5 ] p r i n t a [ 2 : 5 ] + a [ 7 : ] >>> [ 3, 4, 5, 9, 1 0 ] p r i n t [ 2, 3 ] 5 >>> [ 2, 3, 2, 3, 2, 3, 2, 3, 2, 3 ] Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 10/60

11 Friendly syntax Iterators are nearly invisible a = [ 1, 2, 3, 4, 5 ] f o r v a l u e i n a : p r i n t v a l u e Iterating over the elements of a list Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 11/60

12 Friendly syntax Iterators are nearly invisible m y F i l e = open ( i n p u t. t x t ) i = 0 f o r l i n e i n m y F i l e : i += 1 p r i n t i, l i n e Iterating over the lines of a text file Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 12/60

13 Friendly syntax You affect multiple variable in one statement a, b, c = 1, 2, 3 p r i n t a, b, c d e f myfunction ( x ) : r e t u r n x, x x a, b = myfunction ( 3 ) p r i n t a, b Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 13/60

14 Dynamic variable type Variable types is not fixed, it changes a = 42 a = a p p l e a = [ wang, f e i, yue ] a = ( 1, 2, 3, f o u r, 5. 0 ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 14/60

15 Everything is object Numbers are objects a = 2 p r i n t a + 3 p r i n t a. a d d ( 3 ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 15/60

16 Everything is object Lists are objects a = [ 1, 2, 3, 4, 5 ] p r i n t a [ 3 ] p r i n t a. g e t i t e m ( 3 ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 16/60

17 Everything is object Functions are objects d e f s q r ( x ) : r e t u r n x x p r i n t s q r ( 3 ) c l a s s sqrfunc ( o b j e c t ) : d e f c a l l ( s e l f, x ) : r e t u r n x x s q r = sqrfunc ( ) p r i n t s q r ( 3 ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 17/60

18 Standard library An extensive standard library OS-independent file and directory access data persistence XML, JSON, CSV,... parsing most common Internet protocols OS-independent GUI... Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 18/60

19 Standard library Example: defining a command-line interface i m p o r t a r g p a r s e cmdline = a r g p a r s e. ArgumentParser ( d e s c r i p t i o n = Read t h i n g s and do s t u f f s ) cmdline. add argument ( inputpath, a c t i o n = s t o r e, h e l p = path to i n p u t f i l e, t y p e = s t r ) cmdline. add argument ( s, s i z e, a c t i o n = s t o r e, d e s t = f o n t S i z e, d e f a u l t = 16, h e l p = s i z e o f r e a d b u f f e r, t y p e = i n t ) a r g s = cmdline. p a r s e a r g s ( ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 19/60

20 Standard library Example: printing the content of a ZIP archive i m p o r t i m p o r t z i p f i l e a r g p a r s e cmdline = a r g p a r s e. ArgumentParser ( d e s c r i p t i o n = L i s t t h e c o n t e n t o f a ZIP a r c h i v e ) cmdline. add argument ( inputpath, a c t i o n = s t o r e, h e l p = path to i n p u t ZIP a r c h i v e, t y p e = s t r ) a r g s = cmdline. p a r s e a r g s ( ) t r y : a r c h i v e = z i p f i l e. Z i p F i l e ( a r g s. inputpath, r ) f o r i n f o i n a r c h i v e. i n f o l i s t ( ) : p r i n t i n f o. f i l e n a m e, i n f o. d a t e t i m e, i n f o. f i l e s i z e, i n f o. c o m p r e s s s i z e e x c e p t z i p f i l e. B a d Z i p f i l e : p r i n t a r g s. inputpath, i s not a ZIP f i l e e x c e p t I O E r r o r as ( e r r n o, s t r e r r o r ) : p r i n t s t r e r r o r Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 20/60

21 Standard library Example: reading/writing JSON data i m p o r t j s o n a = j s o n. l o a d ( open ( example. j s o n ) ) p r i n t a i m p o r t j s o n a = [ f o o, { bar : ( baz, None, 1. 0, 2) } ] j s o n. dumps ( a ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 21/60

22 Fibonacci numbers Fibonacci numbers are defined as follow F n = F n 1 + F n 2, F 0 = 0, F 1 = 1 Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 22/60

23 Fibonacci numbers Let s code a function to compute Fibonacci numbers d e f f i b ( n ) : i f n < 2 : r e t u r n 1 r e t u r n f i b ( n 1) + f i b ( n 2) p r i n t f i b ( 3 6 ) On my computer, this takes about 11 sec. to compute Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 23/60

24 Memoization To make it faster, we should store intermediate results. It s called memoization fib memo = { } d e f f a s t F i b ( n ) : i f n < 2 : r e t u r n 1 i f not fib memo. h a s k e y ( n ) : fib memo [ n ] = f a s t F i b ( n 1) + f a s t F i b ( n 2) return fib memo [ n ] p r i n t f a s t F i b ( 3 6 ) Same function, but takes 80 msec. to compute! Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 24/60

25 Memoization But we might not be so happy of that solution mix the idea and the implementation have to do by hand this for any expensive recursive function Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 25/60

26 Functions are objects But remember, functions are objects! we can build a function object which contains a function f and a dictionary m if a not in m, m[a] = f (a) returns m[a] Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 26/60

27 Memoizing functions Python implementation for this (works for all arguments) c l a s s Memoize : d e f i n i t ( s e l f, f ) : s e l f. f = f s e l f.m = { } d e f c a l l ( s e l f, a r g s ) : i f not s e l f.m. h a s k e y ( a r g s ) : s e l f.m[ a r g s ] = s e l f. f ( a r g s ) r e t u r n s e l f.m[ a r g s ] Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 27/60

28 Memoizing functions And now, we have a general and clean way to do memoization d e f f i b ( n ) : i f n < 2 : r e t u r n 1 r e t u r n f i b ( n 1) + f i b ( n 2) f i b = Memoize ( f i b ) p r i n t f i b ( 3 6 ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 28/60

29 Monkey patching Methods of objects are functions too c l a s s Animal : d e f makenoise ( s e l f ) : p r i n t not d e f i n e d def dogroaaaar ( ) : p r i n t g r o a a a a r! t i g e r = Animal ( ) t i g e r. makenoise = dogroaaaar t i g e r. makenoise ( ) Dynamic replacement of methods Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 29/60

30 Monkey patching Monkey-patching allow to write very flexible and short code i m p o r t random c l a s s Animal : d e f makenoise ( s e l f ) : p r i n t not d e f i n e d c l a s s RandomNoiseMaker : d e f i n i t ( s e l f, n o i s e L i s t ) : s e l f. n o i s e L i s t = n o i s e L i s t d e f c a l l ( s e l f ) : p r i n t random. c h o i c e ( s e l f. n o i s e L i s t ) t i g e r = Animal ( ) t i g e r. makenoise = RandomNoiseMaker ( [ g r o a a a r!, g r r r r r!, graaaaouuu! ] ) t i g e r. makenoise ( ) Monkey patching with a dynamically defined function Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 30/60

31 Table of Contents 1 Introduction 2 Dynamic languages A Python tour Functions as objects Monkey patching 3 Languages comparison 4 Tools making Content generation 5 Glue code Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 31/60

32 Languages comparison Let s compare Python and Java to do common tasks Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 32/60

33 Languages comparison Printing Hello, world! p u b l i c c l a s s H e l l o W o r l d { p u b l i c s t a t i c v o i d main ( S t r i n g [ ] a r g s ) { System. out. p r i n t l n ( H e l l o, w o r l d! ) ; } } p r i n t H e l l o, w o r l d! Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 33/60

34 Languages comparison Opening a file i m p o r t j a v a. i o. ;... m y F i l e = open ( a r g F i l e n a m e ) B u f f e r e d R e a d e r m y F i l e = new B u f f e r e d R e a d e r ( new F i l e R e a d e r ( a r g F i l e n a m e ) ) ; Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 34/60

35 Languages comparison Add an integer to a list, get an other from the list p u b l i c Vector<I n t e g e r > a L i s t = new Vector<Integer >; p u b l i c i n t anumber = 5 ; p u b l i c i n t anothernumber ; a L i s t. addelement ( anumber ) ; anothernumber = a L i s t. g e t E l ement ( 0 ) ; a L i s t = [ ] anumber = 5 a L i s t. append ( anumber ) anothernumber = a L i s t [ 0 ] Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 35/60

36 Languages comparison Print a list of numbers to a file i m p o r t j a v a. i o. ; f = open ( out. t x t, wb ) p u b l i c c l a s s IOTest { f o r i i n x r a n g e ( ) : p u b l i c s t a t i c v o i d main ( S t r i n g [ ] a r g s ) { f. w r i t e ( s t r ( i ) ) t r y { F i l e f = new F i l e ( out. t x t ) ; f. c l o s e ( ) P r i n t W r i t e r ps = new P r i n t W r i t e r ( new OutputStreamWriter ( new F i l e O u t p u t S t r e a m ( f ) ) ) ; f o r ( i n t i = 0 ; i < ; i ++) ps. p r i n t ( S t r i n g. v a l u e O f ( i ) ) ; ps. c l o s e ( ) ; } c a t c h ( I O E x c e p t i o n i o e ) { i o e. p r i n t S t a c k T r a c e ( ) ; } } } Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 36/60

37 Languages comparison Defining a simple class p u b l i c c l a s s Employee { p r i v a t e S t r i n g myemployeename ; p r i v a t e i n t mytaxdeductions = 1 ; p r i v a t e S t r i n g m y M a r i t a l S t a t u s = s i n g l e ; p u b l i c Employee ( S t r i n g EmployeName ) { t h i s ( employeename, 1 ) ; } p u b l i c Employee ( String EmployeName, i n t taxdeductions ) { t h i s ( employeename, t a x D e d u c t i o n s, s i n g l e ) ; } p u b l i c Employee ( S t r i n g EmployeName, i n t t a x D e d u c t i o n s, S t r i n g m a r i t a l S t a t u s ) { t h i s. employeename = employeename ; t h i s. taxdeductions = taxdeductions ; t h i s. m a r i t a l S t a t u s = m a r i t a l S t a t u s ; }... Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 37/60

38 Languages comparison Defining a simple class c l a s s Employee ( ) : d e f i n i t ( s e l f, employeename, t a x D e d u c t i o n s =1, m a r i t a l S t a t u s= s i n g l e ) : s e l f. employeename = employeename s e l f. taxdeductions = taxdeductions s e l f. m a r i t a l S t a t u s = m a r i t a l S t a t u s... Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 38/60

39 Table of Contents 1 Introduction 2 Dynamic languages A Python tour Functions as objects Monkey patching 3 Languages comparison 4 Tools making Content generation 5 Glue code Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 39/60

40 Tools It s quite common to have to do code which will not be in the final product Build Test Content generation Source-code generation Content generation often need project-specific tools Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 40/60

41 Texture atlas In games, graphics are usually made from lot of pictures Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 41/60

42 Texture atlas Save speed & memory pack all graphics in one texture On mobile devices, it is essential Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 42/60

43 Texture atlas A texture packing many graphics elements is called a texture atlas Building a texture atlas is very device and project dependent Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 43/60

44 Texture atlas Automated texture atlas generation a big time saver artist can test many ideas does not need to be fast integration to build process Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 44/60

45 Texture atlas Texture atlas generator should 1 read a list of pictures 2 read a list of rectangle coordinates 3 pack rectangles 4 generate rectangle coordinates 5 generate a picture Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 45/60

46 Texture atlas Texture atlas generator as a script read/write pictures with one line of code read/write XML, JSON, plain text with very few code rectangle packing algorithm Personal experience 1 hour in Python, 12 hours in C Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 46/60

47 Tool making Scripting languages are wonderful to build such tools Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 47/60

48 Table of Contents 1 Introduction 2 Dynamic languages A Python tour Functions as objects Monkey patching 3 Languages comparison 4 Tools making Content generation 5 Glue code Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 48/60

49 Glue code Software are often built by gluing together various libraries project specific code some specialized processing libraries a communication library a data storage library (database, XML,... ) a GUI library Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 49/60

50 Glue code Many important libraries have Python, Ruby, Lua or Perl bindings Sqlite, MySql, Berkeley DB database wxwindow, QT, GTK graphic users interfaces Numpy, Scipy math and linear algebra Simple Direct Media Layer basic graphic, sound and inputs handling Ogre3D 3d graphic engine Cairo 2d vector graphics... Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 50/60

51 Glue code A way to build cross-platform software quickly use a scripting language to glue libraries Glue code usually goes very well with very dynamic languages Fast prototyping Good performance (most bindings are coded in C or C++) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 51/60

52 Success stories Some successful applications using this approach Python DropBox, Civilization IV, Mercurial, Django, Blender, BitTorrent Ruby Ruby On Rails, Google Sketchup Erlang Wings3D, CouchDB, Goldman Sachs robot traders Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 52/60

53 Project specific code You might want to create your own extensions for a scripting language using legacy code for the 10% code which takes 90% of the time Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 53/60

54 Project specific code All scripting languages provides a way to create extension in C/C++ Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 54/60

55 Writing Python extensions A Say hello Python extension #i n c l u d e <Python. h> s t a t i c PyObject s a y h e l l o ( PyObject s e l f, PyObject a r g s ) { const char name ; i f (! PyArg ParseTuple ( args, s, &name ) ) r e t u r n NULL ; p r i n t f ( H e l l o %s!\ n, name ) ; Py RETURN NONE ; } s t a t i c PyMethodDef HelloMethods [ ] = { { s a y h e l l o, s a y h e l l o, METH VARARGS, G r e e t somebody. }, { NULL, NULL, 0, NULL } }; PyMODINIT FUNC i n i t h e l l o ( v o i d ) { ( v o i d ) P y I n i t M o d u l e ( h e l l o, HelloMethods ) ; } Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 55/60

56 Writing Python extensions The Say hello Python extension setup from d i s t u t i l s. c o r e i m p o r t setup, E x t e n s i o n module1 = E x t e n s i o n ( h e l l o, s o u r c e s = [ h e l l o m o d u l e. c ] ) setup ( name = PackageName, v e r s i o n = 1. 0, d e s c r i p t i o n = This i s a demo package, e x t m o d u l e s = [ module1 ] ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 56/60

57 Writing Python extensions Using the Say hello extension i m p o r t h e l l o h e l l o. s a y h e l l o ( World ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 57/60

58 Writing Python extensions A Fibonacci numbers Python extension #i n c l u d e <Python. h> i n t f i b ( i n t n ) { i f ( n < 2) r e t u r n n ; e l s e r e t u r n f i b ( n 1) + f i b ( n 2); } s t a t i c PyObject f i b ( PyObject s e l f, PyObject a r g s ) { const char command ; i n t n ; i f (! PyArg ParseTuple ( args, i, &n ) ) r e t u r n NULL ; } r e t u r n P y B u i l d V a l u e ( i, f i b ( n ) ) ; s t a t i c PyMethodDef FibMethods [ ] = { { f i b, fib, METH VARARGS, Calculate the Fibonacci numbers. }, {NULL, NULL, 0, NULL} }; PyMODINIT FUNC i n i t f i b ( v o i d ) { ( v o i d ) P y I n i t M o d u l e ( f i b, FibMethods ) ; } Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 58/60

59 Writing Python extensions The Fibonacci Python extension setup from d i s t u t i l s. c o r e i m p o r t setup, E x t e n s i o n module1 = E x t e n s i o n ( f i b, s o u r c e s = [ f i b m o d u l e. c ] ) setup ( name = PackageName, v e r s i o n = 1. 0, d e s c r i p t i o n = This i s a demo package, e x t m o d u l e s = [ module1 ] ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 59/60

60 Writing Python extensions Using the Fibonacci extension i m p o r t f i b p r i n t f i b. f i b ( 1 0 ) Devert Alexandre (School of Software Engineering of USTC) Scripting Languages Slide 60/60