10.5120/ijais12-450799
- Prabin R. Sahoo 2012. Performance Overhead on Relational Join in Hadoop using Hive/Pig/Streaming - A Comparative Analysis. International Journal of Applied Information Systems. 4, 7 (December 2012), 15-20. DOI=http://dx.doi.org/10.5120/ijais450799
-
@article{10.5120/ijais2017451568, author = {Prabin R. Sahoo}, title = {Performance Overhead on Relational Join in Hadoop using Hive/Pig/Streaming - A Comparative Analysis}, journal = {International Journal of Applied Information Systems}, issue_date = {December 2012}, volume = {4}, number = {}, month = {December}, year = {2012}, issn = {}, pages = {15-20}, numpages = {}, url = {/archives/volume4/number7/370-0799}, doi = { 10.5120/ijais12-450799}, publisher = { xA9 2012 by IJAIS Journal}, address = {} } -
%1 450799 %A Prabin R. Sahoo %T Performance Overhead on Relational Join in Hadoop using Hive/Pig/Streaming - A Comparative Analysis %J International Journal of Applied Information Systems %@ %V 4 %N %P 15-20 %D 2012 %I xA9 2012 by IJAIS Journal
Abstract
Hadoop Distributed File System (HDFS) is quite popular in the big data world. It not only provides a framework for storing data in a distributed environment, but also has set of tools to retrieve and process these data using map-reduce concept. This paper discusses the result of evaluation of major tools such as Hive, Pigand hadoop streaming for solving problems from a relational prospective and comparing their performances. Though big data cannot be compared to the strength of relational database in solving relational problems, but as big data is about data so the relational nature of data access cannot be eliminated altogether. Fortunately, there are ways to deal with this which has been discussed in this paper from a performance prospective. This may help the big data community in understanding the performance challenges so that further optimization can be done and the application developers' community can learn how strategically the relational operations need to be used.
References
- Lucene Hadoop, "Hadoop Map-Reduce Tutorial", http://hadoop. apache. org/docs/r0. 15. 2/mapred_tutorial. html, retrieved online November 2012
- Viglas,S. D,Niazi,S, "SAND Join — A skew handling join algorithm for Google's Map/Reduce framework", http://ieeexplore. ieee. org/xpl/articleDetails. jsp?tp=&arnumber=6151466&contentType=Conference+Publications&queryText%3Djoin+in+hadoop, Dec 2011
- Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Ning Zhang, Suresh Antony, Hao Liu, and Raghotham Murthy, "Hive – A Petabyte Scale Data Warehouse Using Hadoop", infolab. stanford. edu/~ragho/Hive-icde2010. pdf,ICDE 2010
- Christer A. Hansen, "Optimizing Hadoop for the cluster", Institue for Computer Science, University of Troms0, Norway, http://oss. csie. fju. edu. tw/~tzu98/Optimizing%20Hadoop%20for%20the%20cluster. pdf, Retrieved online October 2012
- Nils Braden, "The Hadoop Framework", http://homepages. thm. de/~hg51/Veranstaltungen/MasterSeminar1011/NielsBraden. pdf, Retrieved September 2012
- Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler, The Hadoop Distributed File System, http://storageconference. org/2010/Papers/MSST/Shvachko. pdf, retrieved online October, 2012
- Apache Hadoop,"Hadoop 0. 20 documentation", http://hadoop. apache. org/docs/r0. 17. 1/mapred_tutorial. html, August 2008, retrieved online August 2012
- Atlassian Confluence, "Hive Tutorial", https://cwiki. apache. org/Hive/tutorial. html, Feb 2011
- Apache Hadoop, "Pig 0. 7. 0 Documentation", http://Pig. apache. org/docs/r0. 7. 0/tutorial. html, retrieved online August, 2012
- HDFS Architecture Guide, http://hadoop. apache. org/docs/hdfs/current/hdfs_design. html, retrieved online October, 2012
- Map/Reduce Tutorial, http://hadoop. apache. org/docs/r0. 20. 2/mapred_tutorial. html, retrieved online October, 2012
Keywords
Hive, Pig, Hadoop, HDFS, Map-Reduce, streaming
