What is big Query ? Second generation of big data at google. It started with GFS and mapreduce (aka sharding of the data). Then Map operation with nodes and then reduce aggregates the data. Mapreduce is a prior step, you have to split and shard the data and store it Read more…
Slides : Module 1: Introduction to Cloud Dataproc What Qualifies as Unstructured Data? Data without a schema is unstructured. However, if data has a schema or partial schema but it is not helpful to the purposes of analysis or query, that data is considered unstructured also. Two options. The second Read more…
Author : Tim Marshall Geopolitics looks at the ways in which international affairs can be understood through geographical factors:Climate, demographics, cultural regions, and access to natural resources. China and India never have fought each other. Why? Because between them is the highest mountain range in the world. Leaders, ideas, technology, Read more…
Author : Guillaume Pitron P17 : Moins de pollution, mais beaucoup plus d’énérgie dans le même temps. Ce n’est dès lors pas un hasard si l’un de ces éléments fut baptisé prométhium à sa découverte par le chimiste Charles Coryell dans les années 1940 : c’est Grace Marie, son épouse, Read more…
Data Science for Beginners Numpy Basics Matplotlib : Plotting in Python Pandas
Data Pipelines Ingesting real-time data poses a number of challenges: Have to scale in real time Have to deal with data being late Have to deal with bad data coming in real time (duplicates, missing data, etc.) Cloud Pub/Sub Distributed messaging system to handle real-time messaging. Serverless Big Data Pipeline Read more…
Notes for the Google Cloud Platform Big Data and Machine Learning Fundamentals course. https://www.coursera.org/learn/gcp-big-data-ml-fundamentals/home/welcome https://console.cloud.google.com WEEK 1 1. Google Cloud Architecture -2. Abstract away scaling, infrastructure -1. Process, store, and deliver pipelines, ML models, etc. -0. Auth 2. Creating a VM on Compute Engine 2.1 Setting up the VM https://cloud.google.com Read more…
Summary of course topics Supervised learning – labeled data Linear regression Logistic regression Neural networks Support vector machines Unsupervised learning – unlabeled data K-means PCA Anomaly detection Special applications/topics Recommender systems Large scale machine learning Advice on building machine learning systems Bias and variance Regularization What to do next when developing a Read more…
Problem description and pipeline Case study focused around photo OCR Three reasons to do this 1) Look at how a complex system can be put together 2) The idea of a machine learning pipeline What to do next How to do it 3) Some more interesting ideas Applying machine learning to tangible problems Artificial data synthesis What is the photo OCR Read more…
Learning with large datasets This set of notes look at large scale machine learning – how do we deal with big datasets? If you look back at 5-10 year history of machine learning, ML is much better now because we have much more data However, with this increase in data comes Read more…