Hadoop

A comprehensive Big Data cheatsheet covering

1. What is Big Data?

Big Data refers to extremely large and complex datasets that cannot be processed, stored, or analyzed using traditional database systems.

5 V’s of Big Data:

2. What is Hadoop?

Hadoop = Distributed Storage (HDFS) + Distributed Processing (MapReduce)

Created by: Doug Cutting Named after his son’s toy elephant 🐘

3. Hadoop Architecture

A) HDFS Architecture

1. NameNode

2. DataNode

3. Secondary NameNode

B) MapReduce Architecture

Components:

4. HDFS Important Concepts

Block Size

Default:

Replication Factor

Default:

5. Important Hadoop Configuration Files

Location:

$HADOOP_HOME/etc/hadoop/
FilePurpose
core-site.xmlCore configuration
hdfs-site.xmlHDFS settings
mapred-site.xmlMapReduce settings
yarn-site.xmlYARN configuration

6. Hadoop Startup Commands

Format HDFS

hdfs namenode -format

Start Hadoop

start-dfs.sh
start-yarn.sh

Stop Hadoop

stop-dfs.sh
stop-yarn.sh

Check running daemons

jps

Expected output:

7. HDFS Commands (Very Important 🔥)

Basic Commands

Check HDFS

hdfs dfs -ls /

Create directory

hdfs dfs -mkdir /data

Upload file

hdfs dfs -put file.txt /data

Download file

hdfs dfs -get /data/file.txt

View file

hdfs dfs -cat /data/file.txt

Delete file

hdfs dfs -rm /data/file.txt

Delete directory

hdfs dfs -rm -r /data

🗑 Trash Policy (Exam Important)

Enable in core-site.xml:

<property>
<name>fs.trash.interval</name>
<value>60</value>
</property>

Access trash:

hdfs dfs -ls /user/username/.Trash

Restore:

hdfs dfs -mv /user/username/.Trash/Current/data/file.txt /data/

8. HDFS File System Check

Check file block details

hdfs fsck /data/file.txt -files -blocks

Check replication

hdfs dfs -stat %r /data/file.txt

9. MapReduce Example (WordCount – Practical)

Step 1: Create input file

echo "hello hadoop hello big data" > input.txt

Step 2: Put in HDFS

hdfs dfs -mkdir /input
hdfs dfs -put input.txt /input

Step 3: Run WordCount

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output

Step 4: View output

hdfs dfs -cat /output/part-r-00000

10. YARN Architecture

YARN = Yet Another Resource Negotiator

Components:

11. Hadoop Web UI

ServiceURL
NameNodehttp://localhost:9870
ResourceManagerhttp://localhost:8088

12. Important Hadoop Ports

ServicePort
NameNode9870
DataNode9864
ResourceManager8088

13. Common Hadoop Ecosystem Tools

Apache Hive :- SQL on Hadoop

Apache Pig :- Data flow scripting

Apache HBase :- NoSQL DB on HDFS

Apache Spark :- Fast processing engine

14. Important HDFS Commands (Exam Special)

stat formats

hdfs dfs -stat %b file
FormatMeaning
%bfile size in bytes
%ggroup name
%nfile name
%oblock size

15. Checksum Concept

16. Hadoop Modes

ModeDescription
StandaloneDefault
Pseudo-distributedSingle machine cluster
Fully distributedReal cluster

17. Interview Important Points

18. Directory Management Commands

List files

hdfs dfs -ls /

Detailed listing

hdfs dfs -ls -h /

Recursive listing

hdfs dfs -ls -R /

Create directory

hdfs dfs -mkdir /data

Create multiple directories

hdfs dfs -mkdir -p /data/input/2026

Remove empty directory

hdfs dfs -rmdir /data

Remove non-empty directory

hdfs dfs -rm -r /data

Force delete (skip trash)

hdfs dfs -rm -r -skipTrash /data

19. File Management Commands

Upload local → HDFS

hdfs dfs -put file.txt /data

Alternative upload

hdfs dfs -copyFromLocal file.txt /data

Download HDFS → local

hdfs dfs -get /data/file.txt

Alternative download

hdfs dfs -copyToLocal /data/file.txt

Move file

hdfs dfs -mv /data/file.txt /input/

Copy file inside HDFS

hdfs dfs -cp /data/file.txt /backup/

Append data (Important 🔥)

hdfs dfs -appendToFile newdata.txt /data/file.txt

(HDFS normally write-once-read-many, but append is supported.)

20. Viewing File Content

View full file

hdfs dfs -cat /data/file.txt

View first lines

hdfs dfs -head /data/file.txt

View last lines

hdfs dfs -tail /data/file.txt

Count lines/words/chars

hdfs dfs -cat /data/file.txt | wc

21 File Information Commands (Exam Special 🔥)

Check disk usage

hdfs dfs -du /data

Human readable size

hdfs dfs -du -h /data

Check total disk usage summary

hdfs dfs -df -h

File statistics

hdfs dfs -stat %b /data/file.txt
FormatMeaning
%bfile size
%nfile name
%ggroup
%oblock size
%rreplication factor
%ymodification date

Check blocks & replication

hdfs fsck /data/file.txt -files -blocks -locations

22 Permission & Ownership Commands

Change permission

hdfs dfs -chmod 755 /data/file.txt

Change owner

hdfs dfs -chown user:group /data/file.txt

Change group

hdfs dfs -chgrp groupname /data/file.txt

Permission Format:

rwx r-x r-x

23. Replication Management

Check replication

hdfs dfs -stat %r /data/file.txt

Change replication

hdfs dfs -setrep 2 /data/file.txt

Force replication immediately

hdfs dfs -setrep -w 2 /data/file.txt

24. Trash Management (Very Important for Exam)

Enable Trash (core-site.xml)

<property> <name>fs.trash.interval</name> <value>60</value> </property>

View trash

hdfs dfs -ls /user/username/.Trash

Restore file

hdfs dfs -mv /user/username/.Trash/Current/data/file.txt /data/

25. Advanced Commands (Admin Level)

Check safe mode

hdfs dfsadmin -safemode get

Enter safe mode

hdfs dfsadmin -safemode enter

Leave safe mode

hdfs dfsadmin -safemode leave

Report cluster info

hdfs dfsadmin -report

Shows:

Balancer (balance data across nodes)

hdfs balancer

26. Practical Practice Task (Do This 🔥)

Try this sequence:

hdfs dfs -mkdir /practice echo "big data hadoop practice" > test.txt hdfs dfs -put test.txt /practice hdfs dfs -ls /practice hdfs dfs -cat /practice/test.txt hdfs dfs -setrep 2 /practice/test.txt hdfs fsck /practice/test.txt -blocks hdfs dfs -rm /practice/test.txt

If this works without error → your Hadoop is properly configured ✅

27 Difference: hdfs dfs vs hadoop fs

CommandMeaning
hdfs dfsHDFS specific
hadoop fsWorks for all Hadoop-supported file systems

Example:

hadoop fs -ls /