LECTURE 10 MONITORING

Transcription

1 SYSTEM ADMINISTRATION MTAT LECTURE 10 MONITORING Prepared By: Amnir Hadachi and Artjom Lind University of Tartu, Institute of Computer Science / 1

2 OUTLINE 1.Goal behind monitoring 2.Planning 3.Technical monitoring methods 4.Maintenance of monitoring systems 5.Monitoring tools Self-scripting Wireless networks Data collection and graphs Log collection and handling Monitoring services/servers Information security 6. E.g. System activity report 7.Conclusion 2

3 1. GOAL BEHIND MONITORIN G 3

4 GOAL BEHIND MONITORING What is monitoring: Involves: Functionality Usage statistics Environmental parameters Of the network services and systems 4

5 GOAL BEHIND MONITORING Monitoring goals: Detecting faults Using you own monitoring tools no user feedback taken into account ideally anticipate the failure Building a historical data for tracking and exploration purposes and statistics. Failure pattern extraction and analysis to determine the catalyzer of the phenomena. Detection of misuse in order to alleviate the effects. 5

6 GOAL BEHIND MONITORING Prerequisites: You need to have a clear understanding: What How MONITOR What not 6

7 2. PLANNIN G 7

8 PLANNING Consequence of bad planning Vagueness in answer the previous question can lead to: System is not monitored at all because worked too well (or poorly) and has been forgotten. Monitoring something because the one who order the action or the one who execute it has no clue about the issues and why could or should be monitored. Alarms down in a flood of false alarms. 8

9 PLANNING Moment of planning monitoring many consideration has to be taken for example: Maintenance (noticing, limiting clear faults, and reduce redundancy) Implementing a monitoring method that indicates wether the strive works or not from end user perspective. Gathering historical data for management purposes, reporting, and anomalies detection. 9

10 PLANNING General remark: Monitoring a large infrastructure or systems needs a centralised monitoring implementation with scheduled maintenance in order to make the monitoring effective and useful. 10

11 3. TECHNICAL MONITORIN G METHODS 11

12 TECHNICAL MONITORING METHODS Services and systems can be monitored by: Monitoring services status using system test to check the response to queries and file system if they are full or not. Tracking information provided by systems and services on themselves (e.g. log entries, SNMP traps, etc.) 12

13 TECHNICAL MONITORING METHODS Monitoring services status: PROS CONS Reliable information delayed alarms (runs N checks before giving an alarm) Easy to use 13

14 TECHNICAL MONITORING METHODS Tracking self generated information PROS less resource required active checking of functionalities CONS Information produced can be less reliable (loss of information) Sometime malfunctioning may occur (stop sending messages) fast detection and reporting of faults 14

15 4. MAINTENA NCE OF MONITORIN G SYSTEMS 15

16 MAINTENANCE OF MONITORING SYSTEMS The purpose behind maintenance of the monitories system is: Avoid memory lapses and mistakes Adding new device Upgrading the hardware part Remove devices from monitoring The moment they are not reachable (notification) 16

17 5. MONITOR ING TOOLS 17

18 MONITORING TOOLS Self-written scripts: Very useful when two systems can not talk directly to each other. Scripting allow the automation of certain functions: information picking converting results into human readable data. 18

19 MONITORING TOOLS Self-Scripting need planning and time. Things to be considered when scripting: which kind of information you want to generate and how be aware that assumption can be made when scripting and later they become invalid. availability, expandability and fin ration security of the scripts as the environment changes. specify the who and to what extent can use the script how is the information in the script is managed defining the scripts dependencies and its environment choosing the script language carefully (know-how and support in the future) 19

20 MONITORING TOOLS Wireless Networks [Aruba Airwave] Commercial software monitoring (wireless network access points and controllers ) user statistics works better with standalone access points than with WLAN controllers Displays a lot of data or none when it is used with controllers. More Info: 20

21 MONITORING TOOLS Wireless Networks [7signal Sapphire] Commercial software For end to end monitoring separate active sensors measure More Info: 21

22 MONITORING TOOLS Wireless Networks [Big Sister] Open source Network monitoring tool Allowing alerts of any error situations More Info: 22

23 MONITORING TOOLS Data collection and graph presentation: [Cacti] Open source code A tool for collecting packets counters + available memory Application include temperature monitoring of UPS systems and computer server rooms 23

24 MONITORING TOOLS Data collection and graph presentation: [Cricket] Open source Data collection and presentation of results using graphs Easy to take into use Downside: No alarm option, monitoring only development has stopped in

25 MONITORING TOOLS Log collection and handling [Kiwisyslog] Open source Log information collection tool good for collecting analyzing server version is commercialized 25

26 MONITORING TOOLS Log collection and handling [Splunk] Comercial software good log server that you can modify according to your needs capable of sending alarms support simultaneous browsing of many deferents logs support real time monitoring and analytics 26

27 MONITORING TOOLS Monitoring of services / servers [Nagios] open source designed for monitoring servers systems abs services running on them performs scheduled checks or only receives results scalable and modifiable no direct support to create statistics and graphs not easy to use and maintain 27

28 MONITORING TOOLS Monitoring of services / servers [MetaNav] open source Tool for monitoring extensive campus network, developed by UNINETT Easy to use 28

29 MONITORING TOOLS Monitoring of services / servers [Zabbix] open source designed for servers and applications quickly detects problems a commercial support agreement also available 29

30 MONITORING TOOLS Information security [Snort] open source An IDS/IPS tool analyzing traffic easy to take into use maintenance is laborious 30

31 MONITORING TOOLS Information security [Nfsen/Nfdump] open source designed for handling net flow Nfdump commands collects and processes netflow data Nfsen is a web-based user interface for the Nfdump tool App has tracking of traffic amounts, investigation of misuse, identification of worms, and port scanning. 31

32 6. E.G. SYSTEM ACTIVITY REPORT 32

33 SYSTEM ACTIVITY REPORT E.g we will use sar for monitoring the system gives ability to look at: current system performance historical data collected via cron jobs in files (/var/log/sa/sadd and /proc) sar can be by default in the system or you have install Gentoo package with contains it. Collects data on cpu, disk, memory, etc. Compare values over a period of time (days or weeks) system monitoring is an iterative process often when you attempt to cure one bottleneck, new problem is introduced Output not immediately meaningful more info about sar options: 33

34 SYSTEM ACTIVITY REPORT Monitoring CPU usage sar will report about CPU usage just by specifying time interval and number of samples. # sar 10 6 MBP-of-HA:~ amnir$ sar :33:28 %usr %nice %sys %idle 22:33: :33: :33: :34: :34: :34: Average: Any machine with: Zero idle time mean overloaded High system values means maybe hard drive problem 34

35 SYSTEM ACTIVITY REPORT Monitoring memory # sar -W 10 6 user@yosemite:~$ sar -W 10 6 Linux amd64 (yosemite) 11:20:55 AM pswpin/s pswpout/s 11:21:05 AM :21:15 AM :21:25 AM :21:35 AM :21:45 AM :21:55 AM Average: if swapping means not enough physical memory # sar -B /20/2016 _x86_64_ (8 CPU) user@yosemite:~$ sar -B 10 6 Linux amd64 (yosemite) 04/20/2016 _x86_64_ (8 CPU) 11:22:47 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 11:22:57 AM :23:07 AM :23:17 AM :23:27 AM :23:37 AM :23:47 AM Average: large values means not enough physical memory 35

36 SYSTEM ACTIVITY REPORT Monitoring memory # sar -q 10 6 user@yosemite:~$ sar -q 10 6 Linux amd64 (yosemite) 04/20/2016 _x86_64_ (8 CPU) 11:32:17 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 11:32:27 AM :32:37 AM :32:47 AM :32:57 AM :33:07 AM :33:17 AM Average: if queue too large means too many processes running # sar -r 10 6 user@yosemite:~$ sar -r 10 6 Linux amd64 (yosemite) 04/20/2016 _x86_64_ (8 CPU) 11:34:36 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 11:34:46 AM :34:56 AM :35:06 AM :35:16 AM :35:26 AM :35:36 AM Average: helps to look for small free swap spaces 36

37 SYSTEM ACTIVITY REPORT Monitoring disk performance # iostat -d 6 1 MacBook-Pro-of-HA:~ amnir$ iostat -d 6 1 hda disk0 disk2 disk3 disk4 KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s d option for hard rive report helps to look for uneven balance across disk and fast service time # iostat -d -p 10 6 user@yosemite:~$ iostat -d -p 10 6 Linux amd64 (yosemite) 04/20/2016 _x86_64_ (8 CPU) Device: tps kb_read/s kb_wrtn/s kb_read kb_wrtn sda sda sda sda sda sda sda p option for analyzing transfer to/from partitions a good way to look for anomalies by observing appearance of high values 37

38 Learn your system You must: understand all your system resources and their interaction who does what, when, where and why 7. C/C Accumulate full statistics when the system is idle provide you with common reference point for the system behavior can help to initiate automatic anomalies detection and analytic. Monitor performance when the system is busy this will help you to learn 'normal' values When problem strikes define the problem as you can discover the cause by suing monitoring tools fix the problem keep monitoring Monitoring and tuning processes are interactive 38