Tox Issues with virtualenv GNU Guix guix-tox. Guix-tox. A functional version of tox. Cyril Roelandt January 30, /35

Transcription

1 A functional version of tox January 30, /35

2 OpenStack developer at Red Hat since 2013 developer 2/35

3 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 3/35

4 : Automate/standardize testing of Python software Run tests with various congurations Widely used Uses virtualenv 4/35

5 Various user-dened congurations: $ t o x l py34 # Run t h e t e s t s u i t e w i t h Python 3. 4 py27 # Run t h e t e s t s u i t e w i t h Python 2. 7 pypy # Run t h e t e s t s u i t e w i t h PyPy pep8 # F i n d s t y l i s t i c i s s u e s d o c s # B u i l d t h e doc 5/35

6 Running tests: $ t o x epep8 pep8 c r e a t e : /home/ c y r i l / o p e n s t a c k / python n o v a c l i e n t /. t o x / pep8 pep8 i n s t a l l d e p s : r /home/ c y r i l / o p e n s t a c k / python n o v a c l i e n t / r e q u i r e m e n t s. t x t, r /home/ c y r i l / o p e n s t a c k / python n o v a c l i e n t / t e s t r e q u i r e m e n t s. t x t pep8 r u n t e s t s : PYTHONHASHSEED= ' ' pep8 r u n t e s t s : commands [ 0 ] f l a k e 8. / n o v a c l i e n t / v2 / s h e l l. py : : 1 0 : E211 w h i t e s p a c e b e f o r e ' ( ' p r i n t (_(" S e r v e r s u p p o r t e d API v e r s i o n s : " ) ) ERROR : I n v o c a t i o n E r r o r : ' / home/ c y r i l / o p e n s t a c k / python n o v a c l i e n t /. t o x / pep8 / b i n / f l a k e 8 ' ERROR : pep8 : commands f a i l e d 6/35

7 creates a virtualenv for each conguration: $ l s. t o x / l o g pep8 Users may ask tox to completely recreate a given environment: $ t o x r epep8 7/35

8 Only Python packages can be installed Reproducibility One package manager per language Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 8/35

9 Only Python packages can be installed Reproducibility One package manager per language Issue: ( l i b f f i t e s t ) $ p i p i n s t a l l c f f i x86_64 l i n u x gnu gcc... c / _ c f f i _ b a c k e n d. c : 1 3 : 1 7 : f a t a l e r r o r : f f i. h : No s u c h f i l e o r d i r e c t o r y #i n c l u d e < f f i. h> ^ c o m p i l a t i o n t e r m i n a t e d. e r r o r : command ' x86_64 l i n u x gnu gcc ' f a i l e d w i t h e x i t s t a t u s 1 9/35

10 Only Python packages can be installed Reproducibility One package manager per language Fix: # apt g e t i n s t a l l l i b f f i dev Or: # yum i n s t a l l... Or... # $distro_pkg_manager i n s t a l l $missing_pkg 10/35

11 Only Python packages can be installed Reproducibility One package manager per language What version of lib will be used? Debian Stable/Testing/Unstable (+ Debian patches) Fedora X (+ Fedora patches) Package from another distribution "Vanilla" version manually installed from a tarball -> Dierent behaviours 11/35

12 Only Python packages can be installed Reproducibility One package manager per language Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 12/35

13 Only Python packages can be installed Reproducibility One package manager per language Environment variables can change the behaviour of a program: $ LANG=C p y t hon3 c " p r i n t ( ' é ' ) " Unable to decode the command from the command l i n e : U n i c o d e E n c o d e E r r o r : ' u t f 8' c o d e c can ' t e ncode c h a r a c t e r ' \ udcc3 ' i n p o s i t i o n 7 : s u r r o g a t e s n o t a l l o w e d $ LANG=en_US. UTF 8 p y thon3 c " p r i n t ( ' é ' ) " é Two developers may witness dierent behaviours depending on the value of "LANG". 13/35

14 Only Python packages can be installed Reproducibility One package manager per language External commands may be used even though the virtual environment has no control over them: $ c a t t o x. i n i [ t o x ] e n v l i s t =py34 [ t e s t e n v ] w h i t e l i s t _ e x t e r n a l s =/ b i n / echo commands= echo " S t a r t i n g t e s t s " p y t h o n t e s t. py The "echo" command might not be the exact same on all machines. A valid tox.ini might even use more complex commands instead of "echo"... 14/35

15 Only Python packages can be installed Reproducibility One package manager per language Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 15/35

16 Only Python packages can be installed Reproducibility One package manager per language Current trend: one language => one package manager Python -> pip Perl -> cpan Javascript -> npm... 16/35

17 Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 17/35

18 Based on Nix Written in GNU Guile (Scheme) Per-user package management Rollbacks Functional 18/35

19 *Functional* package manager: As in "functional programming" A package is a "function" Parameters: version, tarball URL, inputs... Return value: directory in /gnu/store 19/35

20 Example: the "hello" package ( d e f i n e p u b l i c h e l l o ( p a c k a g e ( name " h e l l o ") ( v e r s i o n " " ) ( s o u r c e ( o r i g i n ( method u r l f e t c h ) ( u r i ( s t r i n g append " m i r r o r : / / gnu / h e l l o / h e l l o " v e r s i o n ". t a r. gz ") ) ( sha256 ( b a s e 3 2 "0 s s i 1 w p a f 7 p l a s w q q j w i g p p s g 5 f y h 9 9 v d l b 9 k z l 7 c 9 l n g 8 9 n d q 1 i ") ) ) ) ( build system gnu build system ) ( s y n o p s i s " H e l l o, GNU w o r l d : An example GNU p a c k a g e ") ( d e s c r i p t i o n "GNU H e l l o p r i n t s t h e message \" H e l l o, w o r l d! \ " and t h e n e x i t s. I t s e r v e s a s an example o f s t a n d a r d GNU c o d i n g p r a c t i c e s. As such, i t s u p p o r t s command l i n e arguments, m u l t i p l e l a n g u a g e s, and s o on. " ) ( home page " h t t p : / /www. gnu. o r g / s o f t w a r e / h e l l o /") ( l i c e n s e g p l 3 +) ) ) 20/35

21 Example: installing "hello" $ g u i x p a c k a g e i h e l l o The f o l l o w i n g p a c k a g e w i l l be i n s t a l l e d : h e l l o / gnu / s t o r e /6 b 2 s x s h m s s l d m j 2 k v 1 s k k k y h e l l o p a c k a g e s i n p r o f i l e $ which h e l l o /home/ c y r i l /. g u i x p r o f i l e / b i n / h e l l o $ l s l /home/ c y r i l /. g u i x p r o f i l e / b i n / h e l l o /home/ c y r i l /. g u i x p r o f i l e / b i n / h e l l o > / gnu / s t o r e /6 b 2 s x s h m s s l d m j 2 k v 1 s k k k y h e l l o 2.10/ b i n / h e l l o 21/35

22 Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 22/35

23 Create a virtual environment (just like virtualenv) The user prole is left untouched Run a given command Interactive mode 23/35

24 $ g u i x e n v i r o n m e n t ad hoc p u r e which 24/35

25 $ g u i x e n v i r o n m e n t ad hoc p u r e which $ which g u i x which : no guix i n (/ gnu / s t o r e / scra07w6z2hnmackqj851rqznd7sznvm which 2.21/ b i n ) 24/35

26 $ g u i x e n v i r o n m e n t ad hoc p u r e which $ which g u i x which : no guix i n (/ gnu / s t o r e / scra07w6z2hnmackqj851rqznd7sznvm which 2.21/ b i n ) $ echo $PATH /gnu / s t o r e / scra07w6z2hnmackqj851rqznd7sznvm which 2.21/ bin 24/35

27 $ g u i x e n v i r o n m e n t ad hoc p u r e which $ which g u i x which : no guix i n (/ gnu / s t o r e / scra07w6z2hnmackqj851rqznd7sznvm which 2.21/ b i n ) $ echo $PATH /gnu / s t o r e / scra07w6z2hnmackqj851rqznd7sznvm which 2.21/ bin $ e x i t e x i t 24/35

28 $ g u i x e n v i r o n m e n t ad hoc p u r e python b a b e l p y t h o n 25/35

29 $ g u i x e n v i r o n m e n t ad hoc p u r e python b a b e l p y t h o n $ echo $PYTHONPATH /gnu / s t o r e /7 k8z2a1bqhj47hq4r4iz54cmb5y3hl7d python babel 1.3/ l i b / python3. 4 / s i t e p a c k a g e s : / gnu / s t o r e /9 i a n n n x g 7 m f n z 6 m i s i 4 n f 1 q 8 a v d j r l z r python 3.4.3/ l i b / p y t hon3. 4 / s i t e p a c k a g e s 25/35

30 $ g u i x e n v i r o n m e n t ad hoc p u r e python b a b e l p y t h o n $ echo $PYTHONPATH /gnu / s t o r e /7 k8z2a1bqhj47hq4r4iz54cmb5y3hl7d python babel 1.3/ l i b / python3. 4 / s i t e p a c k a g e s : / gnu / s t o r e /9 i a n n n x g 7 m f n z 6 m i s i 4 n f 1 q 8 a v d j r l z r python 3.4.3/ l i b / p y t hon3. 4 / s i t e p a c k a g e s $ e x i t e x i t 25/35

31 Complete isolation from the rest of the system using containers: $ g u i x e n v i r o n m e n t ad hoc p u r e c o r e u t i l s $ l s / u s r / b i n /make / u s r / b i n /make 26/35

32 Complete isolation from the rest of the system using containers: $ g u i x e n v i r o n m e n t ad hoc p u r e c o r e u t i l s $ l s / u s r / b i n /make / u s r / b i n /make $ g u i x e n v i r o n m e n t ad hoc p u r e c o n t a i n e r c o r e u t i l s # l s / u s r / b i n /make l s : c a n n o t a c c e s s / u s r / b i n /make : No s u c h f i l e o r d i r e c t o r y 26/35

33 Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 27/35

34 Handling multiple versions of a single package (like virtualenv) Handling!Python packages (compilers, lib*-dev) Better isolation from the rest of the system (environment variables, unspecied commands...) 28/35

35 Goal Demo Drawbacks Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 29/35

36 Goal Demo Drawbacks Keep using tox Replace virtualenv with Code : 30/35

37 Goal Demo Drawbacks Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 31/35

38 Goal Demo Drawbacks $ cd python k e y s t o n e c l i e n t $ GUIX_TOX_EXTRA=o p e n s s l guix tox env=guix epy27... py27 Guix env : g u i x environment ad hoc pure python2 pbr python2 b a b e l python2 i s o python2 d e b t c o l l e c t o r python2 n e t a d d r python2 o s l o. c o n f i g python2 o s l o. i 1 8 n python2 o s l o. s e r i a l i z a t i o n python2 o s l o. u t i l s python2 p r e t t y t a b l e python2 r e q u e s t s python2 s i x python2 s t e v e d o r e python2 h a c k i n g python2 c o v e r a g e python2 d i s c o v e r python2 f i x t u r e s python2 k e y r i n g python2 l x m l python2 mock python2 o a u t h l i b python2 o s l o t e s t python2 pycrypto python2 requests mock python2 sphinx python2 t e s t r e p o s i t o r y python2 t e s t r e s o u r c e s python2 t e s t t o o l s python2 webob python2 s e t u p t o o l s python2 p b r o p e n s s l python 2 e x e c=p y t h o n s e t u p. py t e s t r s l o w e s t t e s t r a r g s=... Ran 1131 (+937) t e s t s i n s ( s ) PASSED ( i d =42, s k i p s =4)... 32/35

39 Goal Demo Drawbacks Table of contents 1 2 Only Python packages can be installed Reproducibility One package manager per language 3 4 Goal Demo Drawbacks 33/35

40 Goal Demo Drawbacks All of PyPI is not available in (but "guix import works pretty well) Some features are not supported is alpha software (but one can use " env=virtualenv") Only available on GNU/Linux 34/35

41 Goal Demo Drawbacks Join us on Freenode: #guix! 35/35