diff options
Diffstat (limited to 'chapter/8/big-data.md')
| -rw-r--r-- | chapter/8/big-data.md | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/chapter/8/big-data.md b/chapter/8/big-data.md index b3dc0a9..8044f70 100644 --- a/chapter/8/big-data.md +++ b/chapter/8/big-data.md @@ -384,7 +384,7 @@ The Hive execution model as shown above composes of the below important componen <figure class="main-container"> <img src="./Hive-transformation.png" alt="Hive transformation" /> </figure> -*Figure from:* %cite thusoo2010hive --file big-data %} +*Figure to depict the transformation flow during optimization, from:* %cite thusoo2010hive --file big-data %} Some of the important transformations are : @@ -395,7 +395,7 @@ The Hive execution model as shown above composes of the below important componen - Join Reordering - Large tables are streamed and not materialized in-memory in the reducer to reduce memory requirements.Some optimizations are not enabled by default but can be activated by setting certain flags. - Repartitioning data to handle skew in GROUP BY processing.This is achieved by performing GROUP BY in two MapReduce stages - first where data is distributed randomly to the reducers and partial aggregation is performed. In the second stage, these partial aggregations are distributed on GROUP BY columns to different reducers. - Hash bases partial aggregations in the mappers to reduce the data that is sent by the mappers to the reducers which help in reducing the amount of time spent in sorting and merging the resulting data. - + - Execution Engine : Execution Engine executes the tasks in order of their dependencies. A MapReduce task first serializes its part of the plan into a plan.xml file. This file is then added to the job cache and mappers and reducers are spawned to execute relevant sections of the operator DAG. The final results are stored to a temporary location and then moved to the final destination (in the case of say INSERT INTO query). |