Continuous Performance Testing Shopware Developer Conference. Kore Nordmann 08. June 2013

Transcription

1 Continuous Performance Testing Shopware Developer Conference Kore Nordmann 08. June 2013

2 About Me Kore

3 Co-founder of Helping people to create high quality web applications. Expert consulting Individual training Get a training on object oriented design for your team!

4 Outline Motivation Conclusion

5 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent

6 Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... )

7 The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas

8 A webshop Awesome Shop 0 articles 0.00 Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor.

9 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering

10 JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests)

11 Getting started Create a test plan Example: What do users actually do on your site? Random browser User registration Sign on Shopping with checkout Commenting products

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant

34 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target >

35 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target >

36 Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > < antcall target= start jmeter after hook /> 436 </ target >

37 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target >

38 Test Setup What do we actually test?

39 Test Setup iftstat, iftop jmeter Response Times Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd Profiling, xhprof, Response Times vmstat, top Slave Master Slow Query Log, Services

40 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top

41

42

43

44

45 Continuous Performance Plugins available for: Jenkins Sonar Maintaining all those servers can be expensive

46 Continuous testing with Jenkins

47 Questions Common questions: Can we survive christmas? What is the maximum we can accomplish with the current setup? Does (APC XCache MemCache) really help us?

48 Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: PIs per month sold articles per month registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops

49 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully

50 Verification Do your requests actually model customer requirements? Compare generated accesss logs with real access logs Compare user registrations / checkouts per hour with requested values

51 Outline Motivation Conclusion

52 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care

53