Sparsh Kumar's Blog

Decoding Hadoop: An In-Depth Analysis of Parallelism and Velocity in Big Data Processing

Sparsh Kumar's photo
Sparsh Kumar
·

3 min read

Decoding Hadoop: An In-Depth Analysis of Parallelism and Velocity in Big Data Processing

Introduction :

In the ever-evolving landscape of big data management, the significance of Hadoop cannot be overstated. This article dives into the intricate workings of Hadoop, exploring the utilization of parallelism to address the velocity problem in data upload. Authored by a seasoned technical blogger, this piece unravels the complexities of Hadoop's data flow mechanism through a comprehensive AWS-based experiment.

The Experiment :

Embarking on a journey to unravel the mysteries of Hadoop, I conducted an insightful experiment on AWS, involving the setup of four EC2 instances – one as the NameNode, one as the Client, and the remaining two as DataNodes. The objective was clear: delve into the nuances of data upload at the DataNode level and dissect the intricacies of Hadoop's parallel processing capabilities.

Steps Taken:

  1. AWS Configuration:

    • Created an AWS account and launched four EC2 instances with designated roles.

  2. Hadoop Environment Setup:

    • Installed JDK and Hadoop packages across all instances.

    • Configured "hdfs-site.xml" and "core-site.xml" files in DataNodes and NameNode, adhering to best practices.

  3. Cluster Initialization:

    • Formatted the NameNode to prepare for Hadoop operations.

    • Ensured seamless startup of Hadoop daemon services across all instances.

  4. Cluster Verification:

    • Checked DataNodes' availability within the Hadoop cluster using the command hadoop dfsadmin -report.

  5. File Upload and Monitoring:

    • Executed the Hadoop Client command hadoop fs -put <file_name> / to initiate file upload.

    • Monitored the Hadoop cluster using the command hadoop fs -ls / to verify the successful upload.

  6. Packet Analysis with Tcpdump:

    • Installed the powerful Tcpdump package using yum install tcpdump.

    • Ran Tcpdump commands on both the NameNode and DataNodes to capture and analyze TCP/IP packets.

Observations and Analysis:

  • Request-Reply Dynamics: Tcpdump revealed a fascinating interplay between the Client and the NameNode. The Client requested the NameNode for DataNodes' IP addresses, and the NameNode responded, orchestrating the subsequent data transfer.

  • Serialism in Data Flow: The analysis of data packets using tcpdump -i eth0 port 50010 -n -x on both DataNodes and the NameNode showcased a sequential flow of data packets. This serialized data transfer ensures efficient uploadation, with alternating packets reaching DataNode1 and DataNode2.

Conclusion:

Contrary to the notion of parallelism, the experiment demonstrated that Hadoop employs a form of "serialism" in data packet flow, strategically optimizing the velocity of data upload. This revelation challenges conventional wisdom and emphasizes the nuanced approach Hadoop takes in managing the intricacies of big data.

For further insights into my experiments and expertise in the realm of big data and cloud computing, feel free to connect with me on LinkedIn: Sparsh Kumar - LinkedIn.

Stay tuned for more in-depth explorations into the ever-evolving world of technology and data management!