Data QA Using Python¶
A data quality analysis (QA) problem over a table using pandas without drowning in code.Let’s discover a systematic QA analysis, step-by-step.
You have a table ith these columns:
| A | B | C | width | D | Region |
Each row represents record. Region groups related records geografically.
read moreSome DL.
Deep Learning Tools¶
Machine Learning¶
Difference/similarities between DL and ML. In ML model receives data and get patterns and make a representation that fits best the data. Then when you input the new data model figures out a class or label for each datapoint. So learning is the storing collections of patterns that are used to make assumption about new input. DL is a subtype of ML.
read moreSome DL Pt2. Back Propagation.
Backpropagation¶
To unrderstand backpropagation we need to understand the forward propagation: the process of moving the input data through the net to get the output. On the contrary, backpropagation is the process of moving the error back through the net to adjust the weights and biases. Backpropagation involves calculating the gradient of the error with respect to the weights and biases of the neural network.
read moreRegression
Intro¶
Regression is a supervised learning problem: labelled data passed to a model, once model is instantiated some new input data can be passed in to predict what may happen next is a continuious sequence.
The simplest model is a line. A line is a rough generalization that gives the ability to explain and predict variables that have a linear relationship with each other. A line that fits a set of data best is a Linear Regression. In regression problems, we are trying to predict a continuous-valued output: housing price is the most known dataset.
read moreScikit-Learn Chunks
Data Visualization (Python)
Example 1¶
I have written a sript a while ago to plot a long sequence of data. In this first example I plot data represented LIDAR scanning of a road with segments having either cleanded ditches or not, while registering a lot I am intrested here in representing three quantitative variables extracted from LIDAR pointcloud.
read moreHTML (From 100Days of Web)
Intro¶
Numpy (Cool library)
Numpy Intro¶
No need to say a lot here, NumPy is useful and you need it all over the place. In my favorite book about Data Science Jake says "provides an efficient interface to store and operate on dense data buffers." and then make a survey of many goodies fo numpy library. VanderPlas, Jake. Python Data Science Handbook read more
Clustering
Intro¶
Cluster analysis is the method in data analysis that is used to classify data points. Clustering pick out pattern in unlabeled data and group items in meaningful way. As a programmer you have to write scripts that learns the inherent structure of the data with no labeled examples provided (unsupervised learning). The program under the hood analyzes the data it encounters and tries to identify patterns and group the data on output.
read more