AWS EMR Info including Hadoop, Map Reduce and Hive along with Machine Learning
| Algorithm/Compression Type | Splittable | Compression Ratio | Compress_Decompress Speed |
|---|---|---|---|
| GZIP | No | High | Medium |
| bzip2 | Yes | Very High | Slow |
| LZO | Yes | Low | Fast |
| Snappy | No | Low | Very Fast |
| HBase | Dynamodb |
|---|---|
| Wide Column Store | Key-Value Store |
| No row size restrictions | Item Size Restricted |
| Flexible Row Key Data Types | Scalar Types |
| Index Creation is more manual | Easier Index Creation |
| Name of interface | URI |
|---|---|
| YARN ResourceManager | http://master-public-dns-name:8088/ |
| YARN NodeManager | http://slave-public-dns-name:8042/ |
| Hadoop HDFS NameNode | http://master-public-dns-name:50070/ |
| Hadoop HDFS DataNode | http://slave-public-dns-name:50075/ |
| Spark HistoryServer | http://master-public-dns-name:18080/ |
| Zeppelin | http://master-public-dns-name:8890/ |
| Hue | http://master-public-dns-name:8888/ |
| Ganglia | http://master-public-dns-name/ganglia/ |
| HBase UI | http://master-public-dns-name:16010/ |
| Long Running Cluster | Transient Cluster |
|---|---|
| Cluster stays up and runningfor queries against HBase | Temporary Cluster |
| Jobs in cluster run frequently | Shouts Down after Processing |
| Data may be large | Batch jobs when needed |
| Keeps HDFS Data on Core Nodes | Input,Output and Code Stored in S3 |
| Termination Protection | Easy Recovery |
| - | Hive metastore stored in MySQL RDS |
| - | Reduces Costs |
| Large Nodes and Small Cluster | Small Nodes and Large Cluster |
|---|---|
| 500 Large Nodes | 750 small nodes |
| AWS recommends a small cluster with large nodes due to low maintanence | It requires High maintanence |
Default replication factor for EMR
Setting can be found in hdfs-site.xml
** Use HDFS for high I/O requirements.**
For example, 10 nodes * 800 GB = 8 TB
For a Replication Factor of 3.
HDFS capacity => 8 TB/3 = 2.6 TB
Comparing the Data Size and HDFS capcity, we get,
Data Size => 3 TB
HDFS Capacity => 2.6 TB
Conclusion is NOT enough Space
Hue UI Browser
Ref - https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hue.html LDAP Integration - https://docs.aws.amazon.com/emr/latest/ReleaseGuide/hue-ldap.html
Advantages -
Disadvantages -
Data in-transit between nodes is encrypted.
This process involves transferring data from node to node within the cluster, and if you want that data to be encrypted in-transit between nodes, then Hadoop encrypted shuffle has to be setup. Encrypted Shuffle capability allows encryption of the MapReduce shuffle using HTTPS. When you select the in-transit encryption checkbox in the EMR security configuration, Hadoop Encrypted Shuffle is automatically setup for you upon cluster launch.
It allows encryption of the MapReduce shuffle using HTTPS and with optional client authentication(HTTPS with client certificates). It includes -
For enabling the shuffle, the hadoop files need to changed as per the properties file mentioned in the link below.
Ref - https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html
Note
Condition
We need to use EMR with EMRFS. However, your security team requires that you both encrypt all data before sending it to S3 and that you maintain the keys.
Answer Would use CSE-Custom, where you would encrypt the data before sending it to S3 and also manage the the client-side master key. The other encryption options available are: S3 Server-Side Encryption (SSE-S3), S3 manages keys for you; Server-Side Encryption with KMS–Managed Keys (SSE-KMS), S3 uses a customer master key that is managed in the Key Management Service to encrypt and decrypt the data before saving it to an S3 bucket; Client-Side Encryption with KMS-Managed Keys (CSE-KMS), the EMR cluster uses a customer master key to encrypt data before sending it to Amazon S3 for storage and to decrypt the data after it is downloaded.
Key concepts in greater detail:
Types of ML-
Under training and evaluation settings, we can select Default Settings or Custom Settings -> Create a ML Model
| Models | Purpose | Example | Algorithm | Measuring Unit for Quality of Histogram |
|---|---|---|---|---|
| Binary Classification Model | Predict a binary outcome (one of two possible classes) | “Is this email spam or not spam?” | Logistic regression | AUC(Area Under Curve) [Higher the value, better the quality] |
| Multiclass Classification Model | multiple classes (predict one of more than two outcomes) | “Is this product a book, movie, or clothing?” | multinomial logistic regression | Confusion Matrix |
| Regression Model | predict a numeric value. | “What price will this house sell for?” | Linear regression | RMSE [Lower the value, better the quality] |
Ref - https://docs.aws.amazon.com/machine-learning/latest/dg/types-of-ml-models.html
0 - Most Likely 1 - Unlikely
The confusion matrix gives some insights about the performance of the model as a way to visualize the accuracy of multiclass classification predictive models. The confusion matrix illustrates in a table the number or percentage of correct and incorrect predictions for each class by comparing an observation’s predicted class and its true class.
Blue Color - Correct Predictions Light Brownish Color - Incorrect Predictions
F1 measures the quality of the model. Ranges from 0 to 1. Higher the F1 Score, better the Quality of ML model.
If the value is near to 1 or above 0.5, then the binary model prediction is correct else high chances of it being wrong.It comes under default settings. It measures prediction accuracy.
The trade off threshold value provides a business approach to alter the false positives and negatives based on a business descision from the output of the outcome graph.
Obtains output of a data source batch and creates a csv file with results into S3.Output CSV will contain sample like -
| Best Answer | Score |
|---|---|
| 0 | 0.32 |
| 1 | 0.98 |
Here the Best Answer will be 0 or 1, depending upon the threshold value set by user in the settings.
Under Evaluations -> Try-Real Time Predictions -> Paste a Record
Zeppelin
Jupyter
D3.js