Scientific Programming in C XIII. Shell programming

Transcription

1 Scientific Programming in C XIII. Shell programming Susi Lehtola 11 December 2012

2 Introduction Often in scientific computing one needs to do simple tasks related to renaming of files file conversions unit conversions These are often best done using shell programming and command line tools. Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 2/22

3 sed sed performs text filtering and transformation. Examples: Replace foo with bar in file: $ sed s foo bar g f i l e > f i l e. new Delete the first 10 lines of a file $ sed 1,10 d f i l e > f i l e. new Delete the last line of a file $ sed $d f i l e > f i l e. new In-place modification of file with -i argument (no output to stdout). Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 3/22

4 awk awk is a language for processing text files. The input is read line by line, and it is split into fields (i.e., words). Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 4/22

5 awk awk is a language for processing text files. The input is read line by line, and it is split into fields (i.e., words). Awk programs are written as a series of pattern action pairs c o n d i t i o n { a c t i o n } that are run at every line of input. Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 5/22

6 awk, cont d Hello world in awk $ awk BEGIN { p r i n t H e l l o world! } H e l l o world! Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 6/22

7 awk, cont d There are also special BEGIN and END blocks that are run only once at the startand the end of the program, respectively. Full program: BEGIN { / code t h a t i s run a t t h e s t a r t / } { } / code t h a t i s run f o r e v e r y l i n e o f i n p u t / END { / code t h a t i s run a t t h e end / } Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 7/22

8 awk, cont d Example: an xyz file 18 6 m o l e c u l e water c l u s t e r O H H O H H O H H O H H O H H O H H Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 8/22

9 awk, cont d Decompose the file #!/ u s r / b i n /awk f { p r i n t f ( L i n e %i : \ %s \ \n,nr, $0 ) ; f o r ( i =1; i <=NF ; i ++) { p r i n t f ( \ tword %i : \ %s \. \ n, i, $ i ) ; } } Running gives $ c a t c l u s t e r. xyz. / decompose. awk L i n e 1 : 18 Word 1 : 1 8. L i n e 2 : Water c l u s t e r, f i r s t 6 m o l e c u l e s. Word 1 : Water. Word 2 : c l u s t e r,. Word 3 : f i r s t. Word 4 : 6. Word 5 : m o l e c u l e s.. L i n e 3 : O Word 1 : O. Word 2 : Word 3 : Word 4 : Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 9/22

10 awk, cont d $0 contains the whole input line $1 is the first word on the line $2 is the second word on the line... $NF is the last word on the line Useful special variables in awk: NR is the current line number NF contains the number of fields on the current line $NR is the amount of lines in the file (line number of last line) Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 10/22

11 awk, cont d Extract the x coordinates from the file $ c a t c l u s t e r. xyz awk { i f (NR>2) { p r i n t $2 }} Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 11/22

12 awk, cont d Find out the maximum and minimum coordinates #!/ u s r / b i n /awk f BEGIN { max [0]= max [1]= max[2]= 1 e10 ; min [0]= min [1]= min [2]=1 e10 ; } { } i f (NR>2) { f o r ( i =0; i <3; i ++) { i f ( $ ( i +2)<min [ i ] ) { min [ i ]=$ ( i +2) } ; i f ( $ ( i +2)>max [ i ] ) { max [ i ]=$ ( i +2) } } } END { f o r ( i =0; i <3; i ++) { p r i n t f ( % e... % e \n, min [ i ], max [ i ] ) } }Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 12/22

13 awk, cont d Running gives $ c a t c l u s t e r. xyz. / minmax. awk e e e e e e+00 Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 13/22

14 Bash Bash (Bourne-Again SHell) is the default shell on linux systems, and it has quite nice scripting features. For example: $ f o r i i n f o o bar ; do echo $ i ; done f o o bar $ f o r ( ( i =0; i <10; i ++)); do echo The v a l u e o f i i s $ i. ; done The v a l u e o f i i s 0. The v a l u e o f i i s 1. The v a l u e o f i i s 2. The v a l u e o f i i s 3. The v a l u e o f i i s 4. The v a l u e o f i i s 5. The v a l u e o f i i s 6. The v a l u e o f i i s 7. The v a l u e o f i i s 8. The v a l u e o f i i s 9. Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 14/22

15 Bash, cont d You can also loop over files: $ f o r i i n. t e x ; do cp a $ i $ i. o r i g ; done This will make backups of all the.tex files in the current directory. (*.tex is expanded to match all the.tex files in the directory, after which the for loop runs over the expansion) Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 15/22

16 Bash, cont d You can also loop over files: $ f o r i i n. t e x ; do cp a $ i $ i. o r i g ; done This will make backups of all the.tex files in the current directory. (*.tex is expanded to match all the.tex files in the directory, after which the for loop runs over the expansion) Advanced version: $ f o r i i n. t e x ; do # Get m o d i f i e d d a t e s u f f i x=$ ( d a t e r e f e r e n c e=$ i +%Y%m%d.%H%M. bak ) # and backup t h e f i l e cp av $ i ${ i } ${ s u f f i x } done This will suffix the backup with the time stamp of the original file. Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 16/22

17 Bash, cont d Let s say you have a bunch of files with names file1, file2,..., file199, file200,..., file1098, file1099. You want to rename these to file0001, file0002,..., file1099. How do you do this? Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 17/22

18 Bash, cont d Solution: bash and awk. $ f o r ( ( i =1; i <=1099; i ++)); do # Convert number to c o n t a i n l e a d i n g z e r o s n= echo $ i awk { p r i n t f ( %04 i, $1 ) } # Has f i l e name changed? i f [ [ $ i!= $n ] ] ; then mv f i l e $ { i } f i l e $ {n} f i done Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 18/22

19 Bash, cont d Bash also has arrays. $ a r r =( This i s an a r r a y. ) $ echo $ a r r This $ echo ${ a r r [ 0 ] } This $ echo ${ a r r [ 1 ] } i s $ echo ${ a r r [ 2 ] } an $ echo ${ a r r [ 3 ] } a r r a y. $ echo ${ a r r } This i s an a r r a y. $ f o r ( ( i =0; i <${#a r r ] } ; i ++)); do echo Element $ i : \ ${ a r r [ i ] } \. done Element 0 : This. Element 1 : i s. Element 2 : an. Element 3 : a r r a y.. Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 19/22

20 Bash, cont d You can also loop directly over the elements in the array as $ f o r i i n ${ a r r [@ ] } ; do echo $ i ; done This i s an a r r a y. since ${arr[@]} expands to the full array. Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 20/22

21 Bash, cont d For example, in conventional quantum chemistry one often needs to check the convergence with regard to the basis set. This can be nicely automatized with bash arrays $ b a s i s =({, aug }cc pv{d, T,Q, 5, 6 } Z) $ f o r i i n ${ b a s i s [@ ] } ; do echo $ i ; done cc pvdz cc pvtz cc pvqz cc pv5z cc pv6z aug cc pvdz aug cc pvtz aug cc pvqz aug cc pv5z aug cc pv6z Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 21/22

22 Bash, cont d Read more on bash programming at beginners guide advanced level Scientific Programming in C, fall 2012 Susi Lehtola Shell programming 22/22