PANDAS FOUNDATIONS. pandas Foundations
|
|
|
- Samantha Jordan
- 5 years ago
- Views:
Transcription
1 PANDAS FOUNDATIONS pandas Foundations
2 What is pandas? Python library for data analysis High-performance containers for data analysis Data structures with a lot of functionality Meaningful labels Time series functionality Handling missing data Relational operations
3 What you will learn How to work with pandas Data import & export in various formats Exploratory Data Analysis using pandas Statistical & graphical methods Using pandas to model time series Time indexes, resampling
4 PANDAS FOUNDATIONS See you in the course!
5 PANDAS FOUNDATIONS Review of pandas DataFrames
6 pandas DataFrames Example: DataFrame of Apple Stock data Date Open High Low Close Volume Adj Close
7 Indexes and columns In [1]: import pandas as pd In [2]: type(aapl) Out[2]: pandas.core.frame.dataframe In [3]: AAPL.shape Out[3]: (8514, 6) In [4]: AAPL.columns Out[4]: Index(['Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'], dtype= object ) In [5]: type(aapl.columns) Out[5]: pandas.indexes.base.index
8 Indexes and columns In [6]: AAPL.index Out[6]: DatetimeIndex([' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ',... ' ', ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '], dtype='datetime64[ns]', name='date', length=8514, freq=none) In [7]: type(aapl.index) Out[7]: pandas.tseries.index.datetimeindex
9 Slicing In [8]: AAPL.iloc[:5,:] Out[8]: Open High Low Close Volume Adj Close Date In [9]: AAPL.iloc[-5:,:] Out[9]: Open High Low Close Volume Adj Close Date
10 head() In [10]: AAPL.head(5) Out[10]: Open High Low Close Volume Adj Close Date In [11]: AAPL.head(2) Out[11]: Open High Low Close Volume Adj Close Date
11 tail() In [12]: AAPL.tail() Out[12]: Open High Low Close Volume Adj Close Date In [13]: AAPL.tail(3) Out[13]: Open High Low Close Volume Adj Close Date
12 info() In [14]: AAPL.info() Out[14]: <class 'pandas.core.frame.dataframe'> DatetimeIndex: 8514 entries, to Data columns (total 6 columns): Open 8514 non-null float64 High 8514 non-null float64 Low 8514 non-null float64 Close 8514 non-null float64 Volume 8514 non-null int64 Adj Close 8514 non-null float64 dtypes: float64(5), int64(1) memory usage: KB
13 Broadcasting In [15]: import numpy as np In [16]: AAPL.iloc[::3, -1] = np.nan Assigning scalar value to column slice broadcasts value to each row. In [17]: AAPL.head(6) Out[17]: Open High Low Close Volume Adj Close Date NaN NaN NaN
14 Broadcasting In [18]: AAPL.info() Out[18]: <class 'pandas.core.frame.dataframe'> DatetimeIndex: 8514 entries, to Data columns (total 6 columns): Open 8514 non-null float64 High 8514 non-null float64 Low 8514 non-null float64 Close 8514 non-null float64 Volume 8514 non-null int64 Adj Close 5676 non-null float64 dtypes: float64(5), int64(1) memory usage: KB
15 Series In [19]: low = AAPL['Low'] In [20]: type(low) Out[20]: pandas.core.series.series In [21]: low.head() Out[21]: Date Name: Low, dtype: float64 In [22]: lows = low.values In [23]: type(lows) Out[23]: numpy.ndarray
16 PANDAS FOUNDATIONS Let s practice!
17 PANDAS FOUNDATIONS Building DataFrames from scratch
18 DataFrames from CSV files In [1]: import pandas as pd In [2]: users = pd.read_csv('datasets/users.csv', index_col=0) In [3]: print(users) Out[3]: weekday city visitors signups 0 Sun Austin Sun Dallas Mon Austin Mon Dallas 456 5
19 DataFrames from dict (1) In [1]: import pandas as pd In [2]: data = {'weekday': ['Sun', 'Sun', 'Mon', 'Mon'],...: 'city': ['Austin', 'Dallas', 'Austin', 'Dallas',...: 'visitors': [139, 237, 326, 456],...: 'signups': [7, 12, 3, 5]} In [3]: users = pd.dataframe(data) In [4]: print(users) Out[4]: weekday city visitors signups 0 Sun Austin Sun Dallas Mon Austin Mon Dallas 456 5
20 DataFrames from dict (2) In [1]: import pandas as pd In [2]: cities = ['Austin', 'Dallas', 'Austin', 'Dallas'] In [3]: signups = [7, 12, 3, 5] In [4]: visitors = [139, 237, 326, 456] In [5]: weekdays = ['Sun', 'Sun', 'Mon', 'Mon'] In [6]: list_labels = ['city', 'signups', 'visitors', 'weekday'] In [7]: list_cols = [cities, signups, visitors, weekdays] In [8]: zipped = list(zip(list_labels, list_cols))
21 DataFrames from dict (3) In [9]: print(zipped) Out[9]: [('city', ['Austin', 'Dallas', 'Austin', 'Dallas']), ('signups', [7, 12, 3, 5]), ('visitors', [139, 237, 326, 456]), ('weekday', ['Sun', 'Sun', 'Mon', 'Mon'])] In [10]: data = dict(zipped) In [11]: users = pd.dataframe(data) In [12]: print(users) Out[12]: weekday city visitors signups 0 Sun Austin Sun Dallas Mon Austin Mon Dallas 456 5
22 Broadcasting In [13]: users['fees'] = 0 # Broadcasts to entire column In [14]: print(users) Out[14]: city signups visitors weekday fees 0 Austin Sun 0 1 Dallas Sun 0 2 Austin Mon 0 3 Dallas Mon 0
23 Broadcasting with a dict In [1]: import pandas as pd In [2]: heights = [ 59.0, 65.2, 62.9, 65.4, 63.7, 65.7, 64.1 ] In [3]: data = {'height': heights, 'sex': 'M'} In [4]: results = pd.dataframe(data) In [5]: print(results) Out[5]: height sex M M M M M M M
24 Index and columns In [6]: results.columns = ['height (in)', 'sex'] In [7]: results.index = ['A', 'B', 'C', 'D', 'E', 'F', 'G'] In [8]: print(results) Out[8]: height (in) sex A 59.0 M B 65.2 M C 62.9 M D 65.4 M E 63.7 M F 65.7 M G 64.1 M
25 PANDAS FOUNDATIONS Let s practice!
26 PANDAS FOUNDATIONS Importing & exporting data
27 Original CSV file Dataset: Sunspot observations collected from SILSO 1818,01,01, , -1,1 1818,01,02, , -1,1 1818,01,03, , -1,1 1818,01,04, , -1,1 1818,01,05, , -1,1 1818,01,06, , -1,1 Source: SILSO, Daily total sunspot number (
28 Datasets from CSV files In [1]: import pandas as pd In [2]: filepath = 'ISSN_D_tot.csv' In [3]: sunspots = pd.read_csv(filepath) In [4]: sunspots.info() <class 'pandas.core.frame.dataframe'> RangeIndex: entries, 0 to Data columns (total 6 columns): non-null int non-null int non-null int non-null float non-null int non-null int64 dtypes: float64(1), int64(5) memory usage: 3.3 MB
29 Datasets from CSV files In [5]: sunspots.iloc[10:20, :] Out[5]:
30 Problems CSV file has no column headers Columns 0-2: Gregorian date (year, month, day) Column 3: Date as fraction as year Column 4: Daily total sunspot number Column 5: Definitive/provisional indicator (1 or 0) Missing values in column 4: indicated by -1 Dates representation inconvenient
31 Using header keyword In [6]: sunspots = pd.read_csv(filepath, header=none) In [7]: sunspots.iloc[10:20, :] Out[7]:
32 Using names keyword In [8]: col_names = ['year', 'month', 'day', 'dec_date',...: 'sunspots', 'definite'] In [9]: sunspots = pd.read_csv(filepath, header=none,...: names=col_names) In [10]: sunspots.iloc[10:20, :] Out[10]: year month day dec_date sunspots definite
33 Using na_values keyword (1) In [11]: sunspots = pd.read_csv(filepath, header=none,...: names=col_names, na_values='-1') In [12]: sunspots.iloc[10:20, :] Out[12]: year month day dec_date sunspots definite
34 Using na_values keyword (2) In [13]: sunspots = pd.read_csv(filepath, header=none,...: names=col_names, na_values=' -1') In [14]: sunspots.iloc[10:20, :] Out[14]: year month day dec_date sunspots definite NaN NaN NaN NaN NaN NaN 1
35 Using na_values keyword (3) In [15]: sunspots = pd.read_csv(filepath, header=none,...: names=col_names, na_values={'sunspots':[' -1']}) In [16]: sunspots.iloc[10:20, :] Out[16]: year month day dec_date sunspots definite NaN NaN NaN NaN NaN NaN 1
36 Using parse_dates keyword In [17]: sunspots = pd.read_csv(filepath, header=none,...: names=col_names, na_values={'sunspots':[' -1']},...: parse_dates=[[0, 1, 2]]) In [18]: sunspots.iloc[10:20, :] Out[18]: year_month_day dec_date sunspots definite NaN NaN NaN NaN NaN NaN 1
37 Inspecting DataFrame In [19]: sunspots.info() <class 'pandas.core.frame.dataframe'> RangeIndex: entries, 0 to Data columns (total 4 columns): year_month_day non-null datetime64[ns] dec_date non-null float64 sunspots non-null float64 definite non-null int64 dtypes: datetime64[ns](1), float64(2), int64(1) memory usage: 2.2 MB
38 Using dates as index In [20]: sunspots.index = sunspots['year_month_day'] In [21]: sunspots.index.name = 'date' In [22]: sunspots.info() Out[22]: <class 'pandas.core.frame.dataframe'> DatetimeIndex: entries, to Data columns (total 4 columns): year_month_day non-null datetime64[ns] dec_date non-null float64 sunspots non-null float64 definite non-null int64 dtypes: datetime64[ns](1), float64(2), int64(1) memory usage: 2.7 MB
39 Trimming redundant columns In [23]: cols = ['sunspots', 'definite'] In [24]: sunspots = sunspots[cols] In [25]: sunspots.iloc[10:20, :] Out[25]: sunspots definite date NaN NaN NaN NaN NaN NaN 1
40 Writing files In [26]: out_csv = 'sunspots.csv' In [27]: sunspots.to_csv(out_csv) In [28]: out_tsv = 'sunspots.tsv' In [29]: sunspots.to_csv(out_tsv, sep='\t') In [30]: out_xlsx = 'sunspots.xlsx' In [31]: sunspots.to_excel(out_xlsx)
41 PANDAS FOUNDATIONS Let s practice!
42 PANDAS FOUNDATIONS Plotting with pandas
43 AAPL stock data In [1]: import pandas as pd In [2]: import matplotlib.pyplot as plt In [3]: aapl = pd.read_csv('aapl.csv', index_col='date',...: parse_dates=true) In [4]: aapl.head(6) Out[4]: adj_close close high low open volume date
44 Plotting arrays (matplotlib) In [5]: close_arr = aapl['close'].values In [6]: type(close_arr) Out[6]: numpy.ndarray In [7]: plt.plot(close_arr) Out[7]: [<matplotlib.lines.line2d at 0x >] In [8]: plt.show()
45 Plotting arrays (Matplotlib) pandas Foundations
46 Plotting Series (matplotlib) In [9]: close_series = aapl['close'] In [10]: type(close_series) Out[10]: pandas.core.series.series In [11]: plt.plot(close_series) Out[11]: [<matplotlib.lines.line2d at 0x11801cd30>] In [12]: plt.show()
47 Plotting Series (matplotlib) pandas Foundations
48 Plotting Series (pandas) In [13]: close_series.plot() # plots Series directly In [14]: plt.show()
49 Plotting Series (pandas) pandas Foundations
50 Plotting DataFrames (pandas) In [15]: aapl.plot() # plots all Series at once Out[15]: <matplotlib.axes._subplots.axessubplot at 0x118039b38> In [16]: plt.show()
51 Plotting DataFrames (pandas) pandas Foundations
52 Plotting DataFrames (matplotlib) In [17]: plt.plot(aapl) # plots all columns at once Out[17]: <matplotlib.lines.line2d at 0x f0>, <matplotlib.lines.line2d at 0x f8>, <matplotlib.lines.line2d at 0x f0>, <matplotlib.lines.line2d at 0x e8>, <matplotlib.lines.line2d at 0x115652be0>, <matplotlib.lines.line2d at 0x115652dd8> In [18]: plt.show()
53 Plotting DataFrames (matplotlib) pandas Foundations
54 Fixing scales In [19]: aapl.plot() Out[19]: <matplotlib.axes._subplots.axessubplot at 0x118afe048> In [20]: plt.yscale('log') # logarithmic scale on vertical axis In [21]: plt.show()
55 Fixing scales pandas Foundations
56 Customizing plots In [22]: aapl['open'].plot(color='b', style='.-', legend=true) Out[22]: <matplotlib.axes._subplots.axessubplot at 0x11a17db38> In [23]: aapl['close'].plot(color='r', style=., legend=true) Out[23]: <matplotlib.axes._subplots.axessubplot at 0x11a17db38> In [24]: plt.axis(('2001', '2002', 0, 100)) Out[24]: ('2001', '2002', 0, 100) In [25]: plt.show()
57 Customizing plots pandas Foundations
58 Saving plots pandas Foundations
59 Saving plots In [26]: aapl.loc['2001':'2004',['open', 'close', 'high',...: 'low']].plot() Out[26]: <matplotlib.axes._subplots.axessubplot at 0x11ab42978> In [27]: plt.savefig('aapl.png') In [28]: plt.savefig('aapl.jpg') In [29]: plt.savefig('aapl.pdf') In [30]: plt.show()
60 PANDAS FOUNDATIONS Let s practice!
Creative Data Mining
Creative Data Mining Using ML algorithms in python Artem Chirkin Dr. Daniel Zünd Danielle Griego Lecture 7 0.04.207 /7 What we will cover today Outline Getting started Explore dataset content Inspect visually
import pandas as pd d = {"A":[1,2,np.nan], "B":[5,np.nan,np.nan], "C":[1,2,3]} In [9]: Out[8]: {'A': [1, 2, nan], 'B': [5, nan, nan], 'C': [1, 2, 3]}
![import pandas as pd d = {A:[1,2,np.nan], B:[5,np.nan,np.nan], C:[1,2,3]} In [9]: Out[8]: {'A': [1, 2, nan], 'B': [5, nan, nan], 'C': [1, 2, 3]}](/thumbs/76/74106104.jpg "import pandas as pd d = {A:[1,2,np.nan], B:[5,np.nan,np.nan], C:[1,2,3]} In [9]: Out[8]: {'A': [1, 2, nan], 'B': [5, nan, nan], 'C': [1, 2, 3]}")
In [4]: import numpy as np In [5]: import pandas as pd In [7]: d = {"":[1,2,np.nan], "B":[5,np.nan,np.nan], "C":[1,2,3]} In [8]: d Out[8]: {'': [1, 2, nan], 'B': [5, nan, nan], 'C': [1, 2, 3]} In [9]:
Propensity Score Matching
Propensity Score Matching This notebook illustrates how to do propensity score matching in Python. Original dataset available at: http://biostat.mc.vanderbilt.edu/wiki/main/datasets (http://biostat.mc.vanderbilt.edu/wiki/main/datasets)
Numpy. Luis Pedro Coelho. October 22, Programming for Scientists. Luis Pedro Coelho (Programming for Scientists) Numpy October 22, 2012 (1 / 26)
Numpy Luis Pedro Coelho Programming for Scientists October 22, 2012 Luis Pedro Coelho (Programming for Scientists) Numpy October 22, 2012 (1 / 26) Historical Numeric (1995) Numarray (for large arrays)
Quiz3_NaiveBayesTest
Quiz3_NaiveBayesTest November 8, 2018 In [1]: import numpy as np import pandas as pd data = pd.read_csv("weatherx.csv") data Out[1]: Outlook Temp Humidity Windy 0 Sunny hot high False no 1 Sunny hot high
Exploratory data analysis
Exploratory data analysis November 29, 2017 Dr. Khajonpong Akkarajitsakul Department of Computer Engineering, Faculty of Engineering King Mongkut s University of Technology Thonburi Module III Overview
EP219:Data Analysis And Interpretation
EP219:Data Analysis And Interpretation Report: Week 1 Team Poisson ous 28 July 2017 to 4 August 2017 1 Contents 1 Problem Statement 3 2 Python Program 3 3 Histograms 6 4 Inference 7 5 Team responsibilities
Combing Open-Source Programming Languages with GIS for Spatial Data Science. Maja Kalinic Master s Thesis
Combing Open-Source Programming Languages with GIS for Spatial Data Science Maja Kalinic Master s Thesis International Master of Science in Cartography 14.09.2017 Outline Introduction and Motivation Research
Linear Transformations
Lab 4 Linear Transformations Lab Objective: Linear transformations are the most basic and essential operators in vector space theory. In this lab we visually explore how linear transformations alter points
MATH 829: Introduction to Data Mining and Analysis Linear Regression: old and new
1/15 MATH 829: Introduction to Data Mining and Analysis Linear Regression: old and new Dominique Guillot Departments of Mathematical Sciences University of Delaware February 10, 2016 Linear Regression:
Stochastic processes and Data mining
Stochastic processes and Data mining Meher Krishna Patel Created on : Octorber, 2017 Last updated : May, 2018 More documents are freely available at PythonDSP Table of contents Table of contents i 1 The
Introduction to Python
Introduction to Python Luis Pedro Coelho Institute for Molecular Medicine (Lisbon) Lisbon Machine Learning School II Luis Pedro Coelho (IMM) Introduction to Python Lisbon Machine Learning School II (1
NumPy 1.5. open source. Beginner's Guide. v g I. performance, Python based free open source NumPy. An action-packed guide for the easy-to-use, high
NumPy 1.5 Beginner's Guide An actionpacked guide for the easytouse, high performance, Python based free open source NumPy mathematical library using realworld examples Ivan Idris.; 'J 'A,, v g I open source
STATISTICAL THINKING IN PYTHON I. Introduction to summary statistics: The sample mean and median
STATISTICAL THINKING IN PYTHON I Introduction to summary statistics: The sample mean and median 2008 US swing state election results Data retrieved from Data.gov (https://www.data.gov/) 2008 US swing state
Introduction to Python
Introduction to Python Luis Pedro Coelho luis@luispedro.org @luispedrocoelho European Molecular Biology Laboratory Lisbon Machine Learning School 2015 Luis Pedro Coelho (@luispedrocoelho) Introduction
Skriptsprachen. Numpy und Scipy. Kai Dührkop. Lehrstuhl fuer Bioinformatik Friedrich-Schiller-Universitaet Jena
Skriptsprachen Numpy und Scipy Kai Dührkop Lehrstuhl fuer Bioinformatik Friedrich-Schiller-Universitaet Jena kai.duehrkop@uni-jena.de 24. September 2015 24. September 2015 1 / 37 Numpy Numpy Numerische
Complex Numbers. A complex number z = x + iy can be written in polar coordinates as re i where
Lab 20 Complex Numbers Lab Objective: Create visualizations of complex functions. Visually estimate their zeros and poles, and gain intuition about their behavior in the complex plane. Representations
DATA SCIENCE SIMPLIFIED USING ARCGIS API FOR PYTHON
DATA SCIENCE SIMPLIFIED USING ARCGIS API FOR PYTHON LEAD CONSULTANT, INFOSYS LIMITED SEZ Survey No. 41 (pt) 50 (pt), Singapore Township PO, Ghatkesar Mandal, Hyderabad, Telengana 500088 Word Limit of the
timeseries talk November 10, 2015
timeseries talk November 10, 2015 1 Time series Analysis and the noise model 1.0.1 Mark Bakker 1.0.2 TU Delft Voorbeeld: Buis B58C0698 nabij Weert In [1]: import numpy as np import pandas as pd from pandas.tools.plotting
SVD and Image Compression
The SVD and Image Compression Lab Objective: The Singular Value Decomposition (SVD) is an incredibly useful matrix factorization that is widely used in both theoretical and applied mathematics. The SVD
2 Getting Started with Numerical Computations in Python
1 Documentation and Resources * Download: o Requirements: Python, IPython, Numpy, Scipy, Matplotlib o Windows: google "windows download (Python,IPython,Numpy,Scipy,Matplotlib" o Debian based: sudo apt-get
Univariate Analysis of Variance
Univariate Analysis of Variance Output Created Comments Input Missing Value Handling Syntax Resources Notes Data Active Dataset Filter Weight Split File N of Rows in Working Data File Definition of Missing
CS 237 Fall 2018, Homework 06 Solution
0/9/20 hw06.solution CS 237 Fall 20, Homework 06 Solution Due date: Thursday October th at :59 pm (0% off if up to 24 hours late) via Gradescope General Instructions Please complete this notebook by filling
Complex Numbers. Visualize complex functions to estimate their zeros and poles.
Lab 1 Complex Numbers Lab Objective: Visualize complex functions to estimate their zeros and poles. Polar Representation of Complex Numbers Any complex number z = x + iy can be written in polar coordinates
CS 237 Fall 2018, Homework 07 Solution
CS 237 Fall 2018, Homework 07 Solution Due date: Thursday November 1st at 11:59 pm (10% off if up to 24 hours late) via Gradescope General Instructions Please complete this notebook by filling in solutions
LECTURE 3 LINEAR ALGEBRA AND SINGULAR VALUE DECOMPOSITION
SCIENTIFIC DATA COMPUTING 1 MTAT.08.042 LECTURE 3 LINEAR ALGEBRA AND SINGULAR VALUE DECOMPOSITION Prepared by: Amnir Hadachi Institute of Computer Science, University of Tartu amnir.hadachi@ut.ee OUTLINE
Learning Deep Broadband Hongjoo LEE
Learning Deep Broadband Network@HOME Hongjoo LEE Who am I? Machine Learning Engineer Software Engineer Fraud Detection System Software Defect Prediction Email Services (40+ mil. users) High traffic server
Perceptron. by Prof. Seungchul Lee Industrial AI Lab POSTECH. Table of Contents
Perceptron by Prof. Seungchul Lee Industrial AI Lab http://isystems.unist.ac.kr/ POSTECH Table of Contents I.. Supervised Learning II.. Classification III. 3. Perceptron I. 3.. Linear Classifier II. 3..
Applied Machine Learning Lecture 2, part 1: Basic linear algebra; linear algebra in NumPy and SciPy. Richard Johansson
Applied Machine Learning Lecture 2, part 1: Basic linear algebra; linear algebra in NumPy and SciPy Richard Johansson linear algebra linear algebra is the branch of mathematics that deals with relationships
Supplement. July 27, The aims of this process is to find the best model that predict the treatment cost of type 2 diabetes for individual
Supplement July 27, 2017 In [1]: import pandas as pd import numpy as np from scipy import stats import collections import matplotlib.pyplot as plt import seaborn as sns %matplotlib inline The aims of this
Data Analysis of NYC Cab Services
Data Analysis of NYC Cab Services Abhinandan Dubey Stony Brook University New York, USA Raju Khanal Stony Brook University New York, USA [adubey, rkhanal, lmadiraju]@cs.stonybrook.edu Teja Madiraju Stony
ECE521 W17 Tutorial 1. Renjie Liao & Min Bai
ECE521 W17 Tutorial 1 Renjie Liao & Min Bai Schedule Linear Algebra Review Matrices, vectors Basic operations Introduction to TensorFlow NumPy Computational Graphs Basic Examples Linear Algebra Review
L14. 1 Lecture 14: Crash Course in Probability. July 7, Overview and Objectives. 1.2 Part 1: Probability
L14 July 7, 2017 1 Lecture 14: Crash Course in Probability CSCI 1360E: Foundations for Informatics and Analytics 1.1 Overview and Objectives To wrap up the fundamental concepts of core data science, as
IAST Documentation. Release Cory M. Simon
IAST Documentation Release 0.0.0 Cory M. Simon November 02, 2015 Contents 1 Installation 3 2 New to Python? 5 3 pyiast tutorial 7 3.1 Import the pure-component isotherm data................................
DM559 Linear and integer programming
Department of Mathematics and Computer Science University of Southern Denmark, Odense March 8, 2018 Marco Chiarandini DM559 Linear and integer programming Sheet 4, Spring 2018 [pdf format] Included. Exercise
LEC1: Instance-based classifiers
LEC1: Instance-based classifiers Dr. Guangliang Chen February 2, 2016 Outline Having data ready knn kmeans Summary Downloading data Save all the scripts (from course webpage) and raw files (from LeCun
Exercise 1. In the lecture you have used logistic regression as a binary classifier to assign a label y i { 1, 1} for a sample X i R D by
Exercise 1 Deadline: 04.05.2016, 2:00 pm Procedure for the exercises: You will work on the exercises in groups of 2-3 people (due to your large group size I will unfortunately not be able to correct single
STATISTICAL THINKING IN PYTHON I. Probabilistic logic and statistical inference
STATISTICAL THINKING IN PYTHON I Probabilistic logic and statistical inference 50 measurements of petal length Statistical Thinking in Python I 50 measurements of petal length Statistical Thinking in Python
f = Xw + b, We can compute the total square error of the function values above, compared to the observed training set values:
Linear regression Much of machine learning is about fitting functions to data. That may not sound like an exciting activity that will give us artificial intelligence. However, representing and fitting
DM534 - Introduction to Computer Science
Department of Mathematics and Computer Science University of Southern Denmark, Odense October 21, 2016 Marco Chiarandini DM534 - Introduction to Computer Science Training Session, Week 41-43, Autumn 2016
STATISTICAL THINKING IN PYTHON I. Probability density functions
STATISTICAL THINKING IN PYTHON I Probability density functions Continuous variables Quantities that can take any value, not just discrete values Michelson's speed of light experiment measured speed of
Edo Liberty Principal Scientist Amazon Web Services. Streaming Quantiles
Edo Liberty Principal Scientist Amazon Web Services Streaming Quantiles Streaming Quantiles Manku, Rajagopalan, Lindsay. Random sampling techniques for space efficient online computation of order statistics
Image Processing in Numpy
Version: January 17, 2017 Computer Vision Laboratory, Linköping University 1 Introduction Image Processing in Numpy Exercises During this exercise, you will become familiar with image processing in Python.
Hot spot Analysis with Clustering on NIJ Data Set The Data Set:
Hot spot Analysis with Clustering on NIJ Data Set The Data Set: The data set consists of Category, Call groups, final case type, case desc, Date, X-coordinates, Y- coordinates and census-tract. X-coordinates
The Metropolis Algorithm
16 Metropolis Algorithm Lab Objective: Understand the basic principles of the Metropolis algorithm and apply these ideas to the Ising Model. The Metropolis Algorithm Sampling from a given probability distribution
WS27: Unleash the power of GRASS GIS 7. Session 6 Temporal data analysis (TGRASS)
WS27: Unleash the power of GRASS GIS 7 Session 6 Temporal data analysis (TGRASS) Markus Neteler mundialis GmbH & Co KG, Germany Luca Delucchi Fondazione Edmund Mach, Italy Martin Landa Czech Technical
Analyzing the Earth Using Remote Sensing
Analyzing the Earth Using Remote Sensing Instructors: Dr. Brian Vant- Hull: Steinman 185, 212-650- 8514 brianvh@ce.ccny.cuny.edu Ms. Hannah Aizenman: NAC 7/311, 212-650- 6295 haizenman@ccny.cuny.edu Dr.
Lecture Unconstrained optimization. In this lecture we will study the unconstrained problem. minimize f(x), (2.1)
Lecture 2 In this lecture we will study the unconstrained problem minimize f(x), (2.1) where x R n. Optimality conditions aim to identify properties that potential minimizers need to satisfy in relation
What s Cooking? Predicting Cuisines from Recipe Ingredients
What s Cooking? Predicting Cuisines from Recipe Ingredients Kevin K. Do Department of Computer Science Duke University Durham, NC 27708 kevin.kydat.do@gmail.com Abstract Kaggle is an online platform for
Linear Classification
Linear Classification by Prof. Seungchul Lee isystems Design Lab http://isystems.unist.ac.kr/ UNIS able of Contents I.. Supervised Learning II.. Classification III. 3. Perceptron I. 3.. Linear Classifier
R STATISTICAL COMPUTING
R STATISTICAL COMPUTING some R Examples Dennis Friday 2 nd and Saturday 3 rd May, 14. Topics covered Vector and Matrix operation. File Operations. Evaluation of Probability Density Functions. Testing of
Hypothesis Tests Solutions COR1-GB.1305 Statistics and Data Analysis
Hypothesis Tests Solutions COR1-GB.1305 Statistics and Data Analysis Introduction 1. An analyst claims to have a reliable model for Twitter s quarterly revenues. His model predicted that the most recent
In order to save time, the following data files have already been preloaded to the computer (most likely under c:\odmc2012\data\)
ODMC2012 QGIS Ex1 Schools and Public Transport In this exercise, users will learn how to a) map the location of secondary schools in and around the Southampton area; b) overlay the school location map
Outline Python, Numpy, and Matplotlib Making Models with Polynomials Making Models with Monte Carlo Error, Accuracy and Convergence Floating Point Mod
Outline Python, Numpy, and Matplotlib Making Models with Polynomials Making Models with Monte Carlo Error, Accuracy and Convergence Floating Point Modeling the World with Arrays The World in a Vector What
GRASS GIS Development APIs
GRASS GIS Development APIs Lifting the fog on the different ways to develop with and for GRASS Moritz Lennert Member of the GRASS GIS Project Steering Comittee PostGISomics Over 30 years of development
The Source Finder (SoFi)
20.12.2016 Wir schaffen Wissen heute für morgen Paul Scherrer Institute F. Canonaco, C. Bozzetti, K. Dällenbach, Y. Sosedova, J. G. Slowik, I. ElHaddad, M. Crippa, U. Baltensperger,A. S. H. Prévôt and
HOMEWORK #4: LOGISTIC REGRESSION
HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2018 Due: Friday, February 23rd, 2018, 11:55 PM Submit code and report via EEE Dropbox You should submit a
HOMEWORK #4: LOGISTIC REGRESSION
HOMEWORK #4: LOGISTIC REGRESSION Probabilistic Learning: Theory and Algorithms CS 274A, Winter 2019 Due: 11am Monday, February 25th, 2019 Submit scan of plots/written responses to Gradebook; submit your
Level 1 Science Data Description Version 1.0
Miniature X-ray Solar Spectrometer (MinXSS) CubeSat Level 1 Science Data Description Version 1.0 Contacts: Principal Investigator: tom.woods@lasp.colorado.edu Data access: james.mason@lasp.colorado.edu
SKLearn Tutorial: DNN on Boston Data
SKLearn Tutorial: DNN on Boston Data This tutorial follows very closely two other good tutorials and merges elements from both: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/skflow/boston.py
Matrices: 2.1 Operations with Matrices
Goals In this chapter and section we study matrix operations: Define matrix addition Define multiplication of matrix by a scalar, to be called scalar multiplication. Define multiplication of two matrices,
epistasis Documentation
epistasis Documentation Release 0.1 Zach Sailer May 19, 2017 Contents 1 Table of Contents 3 1.1 Setup................................................... 3 1.2 Basic Usage...............................................
1. Determine by inspection which of the following sets of vectors is linearly independent. 3 3.
1. Determine by inspection which of the following sets of vectors is linearly independent. (a) (d) 1, 3 4, 1 { [ [,, 1 1] 3]} (b) 1, 4 5, (c) 3 6 (e) 1, 3, 4 4 3 1 4 Solution. The answer is (a): v 1 is
INTRODUCTION TO DATA SCIENCE
INTRODUCTION TO DATA SCIENCE JOHN P DICKERSON Lecture #13 3/9/2017 CMSC320 Tuesdays & Thursdays 3:30pm 4:45pm ANNOUNCEMENTS Mini-Project #1 is due Saturday night (3/11): Seems like people are able to do
CSE546 Machine Learning, Autumn 2016: Homework 1
CSE546 Machine Learning, Autumn 2016: Homework 1 Due: Friday, October 14 th, 5pm 0 Policies Writing: Please submit your HW as a typed pdf document (not handwritten). It is encouraged you latex all your
pyiast Documentation Release Cory M. Simon
pyiast Documentation Release 0.0.0 Cory M. Simon Apr 15, 2018 Contents 1 Installation 5 2 New to Python? 7 3 pyiast tutorial 9 3.1 Import the pure-component isotherm data................................
Tutorial for reading and manipulating catalogs in Python 1
Tutorial for reading and manipulating catalogs in Python 1 First, download the data The catalogs that you will need for this exercise can be downloaded here: https://www.dropbox.com/sh/le0d8971tmufcqx/aaaxivdo2p_pg63orhkdudr7a?dl=0
Manual for some LCLS Matlab GUIs. Henrik Loos SLAC, 2575 Sand Hill Road, Menlo Park, CA 94025
Manual for some LCLS Matlab GUIs Henrik Loos SLAC, 2575 Sand Hill Road, Menlo Park, CA 94025 October 8, 2007 2 Emittance GUI Scope This graphical user interface (Fig. 1) for Matlab enables the measurement
Data Canopy. Accelerating Exploratory Statistical Analysis. Abdul Wasay Xinding Wei Niv Dayan Stratos Idreos
Accelerating Exploratory Statistical Analysis Abdul Wasay inding Wei Niv Dayan Stratos Idreos Statistics are everywhere! Algorithms Systems Analytic Pipelines 80 Temperature 60 40 20 0 May 2017 80 Temperature
Graphical User Interfaces for Emittance and Correlation Plot. Henrik Loos
Graphical User Interfaces for Emittance and Correlation Plot Common GUI Features Overview Files Configs Measure Data Point Export Common GUI Features Save Saves the present data. Auto-generated file name
TF Mutiple Hidden Layers: Regression on Boston Data Batched, Parameterized, with Dropout
TF Mutiple Hidden Layers: Regression on Boston Data Batched, Parameterized, with Dropout This is adapted from Frossard's tutorial (http://www.cs.toronto.edu/~frossard/post/tensorflow/). This approach is
5615 Chapter 3. April 8, Set #3 RP Simulation 3 Working with rp1 in the Notebook Proper Numerical Integration 5
5615 Chapter 3 April 8, 2015 Contents Mixed RV and Moments 2 Simulation............................................... 2 Set #3 RP Simulation 3 Working with rp1 in the Notebook Proper.............................
Module 5.2: nag sym lin sys Symmetric Systems of Linear Equations. Contents
Linear Equations Module Contents Module 5.2: nag sym lin sys Symmetric Systems of Linear Equations nag sym lin sys provides a procedure for solving real or complex, symmetric or Hermitian systems of linear
Solving Systems of ODEs in Python: From scalar to TOV. MP 10/2011
Solving Systems of ODEs in Python: From scalar to TOV. MP 10/2011 Homework: Integrate a scalar ODE in python y (t) =f(t, y) = 5ty 2 +5/t 1/t 2 y(1) = 1 Integrate the ODE using using the explicit Euler
Unit IV: Introduction to Vector Analysis
Unit IV: Introduction to Vector nalysis s you learned in the last unit, there is a difference between speed and velocity. Speed is an example of a scalar: a quantity that has only magnitude. Velocity is
-ASTR 204 Application of Astroimaging Techniques
Lab 5 - JPL Ephemeris, Binary Maker 3 In Lab 5 we ll look at another ephemeris generating tool; Horizons Web-Interface from JPL, and Binary Maker 3 program for generating radial velocity curves and 3-D
Assignment No A-05 Aim. Pre-requisite. Objective. Problem Statement. Hardware / Software Used
Assignment No A-05 Aim Implement Naive Bayes to predict the work type for a person. Pre-requisite 1. Probability. 2. Scikit-Learn Python Library. 3. Programming language basics. Objective 1. To Learn basic
<br /> D. Thiebaut <br />August """Example of DNNRegressor for Housing dataset.""" In [94]:
![<br /> D. Thiebaut <br />August Example of DNNRegressor for Housing dataset. In [94]:](/thumbs/83/87238525.jpg "<br /> D. Thiebaut <br />August Example of DNNRegressor for Housing dataset. In [94]:")
sklearn Tutorial: Linear Regression on Boston Data This is following the https://github.com/tensorflow/tensorflow/blob/maste
@SoyGema GEMA PARREÑO PIQUERAS
@SoyGema GEMA PARREÑO PIQUERAS WHAT IS AN ARTIFICIAL NEURON? WHAT IS AN ARTIFICIAL NEURON? Image Recognition Classification using Softmax Regressions and Convolutional Neural Networks Languaje Understanding
Exercise_set7_programming_exercises
Exercise_set7_programming_exercises May 13, 2018 1 Part 1(d) We start by defining some function that will be useful: The functions defining the system, and the Hamiltonian and angular momentum. In [1]:
Crash Course on TensorFlow! Vincent Lepetit!
Crash Course on TensorFlow Vincent Lepetit 1 TensorFlow Created by Google for easily implementing Deep Networks; Library for Python, but can also be used with C and Java; Exists for Linux, Mac OSX, Windows;
DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University January 17 2019 Logistics HW 1 is on Piazza and Gradescope Deadline: Friday, Jan. 25, 2019 Office
DESIGN OF SYSTEM LEVEL CONCEPT FOR TERRA 25994
DESIGN OF SYSTEM LEVEL CONCEPT FOR TERRA 25994 Prof. S.Peik Group Number : 14 Submitted by: Deepthi Poonacha Machimada : 5007754 Shridhar Reddy : 5007751 1 P a g e Table of Contents 1. Introduction...
Extracting Canadian Climate Data from Environment Canada dataset
University of British Columbia Department of Statistics Technical Report #244 May 2009 Extracting Canadian Climate Data from Environment Canada dataset by Reza Hosseini 1 1 University of British Columbia
COMS 6100 Class Notes
COMS 6100 Class Notes Daniel Solus September 20, 2016 1 General Remarks The Lecture notes submitted by the class have been very good. Integer division seemed to be a common oversight when working the Fortran
Package weathercan. July 5, 2018
Type Package Package weathercan July 5, 2018 Title Download Weather Data from the Environment and Climate Change Canada Website Version 0.2.7 Provides means for downloading historical weather data from
CHAPTER 10. Regression and Correlation
CHAPTER 10 Regression and Correlation In this Chapter we assess the strength of the linear relationship between two continuous variables. If a significant linear relationship is found, the next step would
YEAR 10 GENERAL MATHEMATICS 2017 STRAND: BIVARIATE DATA PART II CHAPTER 12 RESIDUAL ANALYSIS, LINEARITY AND TIME SERIES
YEAR 10 GENERAL MATHEMATICS 2017 STRAND: BIVARIATE DATA PART II CHAPTER 12 RESIDUAL ANALYSIS, LINEARITY AND TIME SERIES This topic includes: Transformation of data to linearity to establish relationships
Sea Ice Thickness from Altimetry - Practical Exercise for the ESA Cryosphere Training Course 2018
Link to this document: https://tinyurl.com/seaicepractical Sea Ice Thickness from Altimetry - Practical Exercise for the ESA Cryosphere Training Course 2018 The purpose of this practical is to get familiar
A COMMON DATA MODEL APPROACH TO NETCDF AND GRIB DATA HARMONISATION
A COMMON DATA MODEL APPROACH TO NETCDF AND GRIB DATA HARMONISATION Alessandro Amici, B-Open, Rome @alexamici - http://bopen.eu Workshop on developing Python frameworks for earth system sciences, 2017-11-28,
Iris. A python package for the analysis and visualisation of Meteorological data. Philip Elson
Iris A python package for the analysis and visualisation of Meteorological data Philip Elson 30 th Sept 2015 Outline What is Iris? Iris demo Using Iris for novel analysis Opportunities for combining Iris
Appendix C Grids in DSS
Appendix C Grids in DSS Appendix C Grids in DSS C.1 Identifying Grid Records in DSS Grid records in DSS are named according to a naming convention that differs slightly from the convention for time-series
Worksheet 1: Integrators
Simulation Methods in Physics I WS 2014/2015 Worksheet 1: Integrators Olaf Lenz Alexander Schlaich Stefan Kesselheim Florian Fahrenberger Peter Košovan October 22, 2014 Institute for Computational Physics,
pygaps Release 1.2.0
pygaps Release 1.2.0 Feb 19, 2018 Contents 1 Contents 3 1.1 Overview................................................. 3 1.2 Installation................................................ 5 1.3 Quickstart................................................
Pysynphot: A Python Re Implementation of a Legacy App in Astronomy
Pysynphot: A Python Re Implementation of a Legacy App in Astronomy Vicki Laidler 1, Perry Greenfield, Ivo Busko, Robert Jedrzejewski Science Software Branch Space Telescope Science Institute Baltimore,
Instructions for Mapping 2011 Census Data
Instructions for Mapping 2011 Census Data To map 2011 census data, you must download the census boundary files and the census data separately, then join the two files in ArcMap. In this guide, we will
You have 3 hours to complete the exam. Some questions are harder than others, so don t spend too long on any one question.
Data 8 Fall 2017 Foundations of Data Science Final INSTRUCTIONS You have 3 hours to complete the exam. Some questions are harder than others, so don t spend too long on any one question. The exam is closed
ESRI Object Models and Data Capture 2/1/2018
Number 123 124 125 ESRI* Object Models; Data Capture Feature Class (spatial table) Age_Ma 142 136 1_sigma 1.5 2.0 Rx_Type B_schist G_schist Object Class (nonspatial table) Size_kg 3.4 1.3 Y Control Point
pyorbital Documentation
pyorbital Documentation Release v1.3.1 The Pytroll crew Sep 14, 2018 Contents 1 Installation 3 2 TLE files 5 3 Use actual TLEs to increase accuracy 7 3.1 TLE files.................................................
INF 5860 Machine learning for image classification. Lecture 5 : Introduction to TensorFlow Tollef Jahren February 14, 2018
INF 5860 Machine learning for image classification Lecture 5 : Introduction to TensorFlow Tollef Jahren February 14, 2018 OUTLINE Deep learning frameworks TensorFlow TensorFlow graphs TensorFlow session