In a Hadoop cluster, find how to
contribute limited/specific amount
of storage as slave to the cluster
In this blog we will do learn and practical how we can limited storage share from datanode to namenode.
I have already Created a hadoop cluster, which consist one Namenode and two Datanode.
Click this URL and Setup hadoop Cluster:
URL: https://weeklyrealindustrytask.blogspot.com/2020/11/hadoop-cluster-setupconfigure-with-hdfs.html
Pre-requisite :
- I have already configured Hadoop Cluster.
- I have added 20 GB or 20 GB Drive datanode 1 & datanode2. ______________________________________________________________
dfsadmin:
dfsadmin (distributed file system administration) command is used for file system administration activities like getting file system report, enter/leave safemode, refreshing nodes in the cluster and HDFS upgrade etc.
-report — Reports basic file system information and statistics.
check how many Datanode connected on Namenode Please execute this command.
| # hadoop dfsadmin -report
Step-1: Go to DataNode1 and and check how many drives are connected on datanode1 Using fdisk -l command.
| # fdisk -l
Step-2: Create Partition/Volume on DataNode1:
First we need to Create a Partition Using fdisk command.
| # fdisk /dev/sdb
| # fdisk /dev/sdb
Step-3: udevadm settle command :
Changing udev attributes happens in the background and might take a long time. The udevadm settle command waits until the change is fully registered, which ensures that your next command will be able to utilize the new attribute correctly.
| # udevadm settle
Step-4: Format the Partition :
Formatting: Formatting is the act of creating a file system on a volume, so that the operating system can store and retrieve data on that volume.
| # mkfs.ext4 /dev/sdb1
Step-5: The Linux mount mechanism:
This section explains basic concepts of mounting file systems on Linux.
On Linux, UNIX, and similar operating systems, file systems on different partitions and removable devices (CDs, DVDs, or USB flash drives for example) can be attached to a certain point (the mount point) in the directory tree, and then detached again. While a file system is mounted on a directory, the original content of the directory is not accessible.
Note: that Linux does not prevent you from mounting a file system to a directory with a file system already attached to it.
Step-6: Go to DataNode2 and and check how many drives are connected on datanode2 Using fdisk -l command.
| # fdisk -l
Step-7: Create Partition/Volume on DataNode2:
First we need to Create a Partition Using fdisk command.
| # fdisk /dev/sdb
Step-8: udevadm settle command :
Changing udev attributes happens in the background and might take a long time. The udevadm settle command waits until the change is fully registered, which ensures that your next command will be able to utilize the new attribute correctly.
| # udevadm settle
Step-9: Format the Partition :
Formatting: Formatting is the act of creating a file system on a volume, so that the operating system can store and retrieve data on that volume.
| # mkfs.ext4 /dev/sdb1
Step-10: The Linux mount mechanism:
This section explains basic concepts of mounting file systems on Linux.
On Linux, UNIX, and similar operating systems, file systems on different partitions and removable devices (CDs, DVDs, or USB flash drives for example) can be attached to a certain point (the mount point) in the directory tree, and then detached again. While a file system is mounted on a directory, the original content of the directory is not accessible.
Note: that Linux does not prevent you from mounting a file system to a directory with a file system already attached to it.
Step-11: Go to NameNode and check how many Data Nodes are connected on NameNode:
| # hadoop dfsadmin -report
Note: DataNode 1 and DataNode 2 I Have shared 19 GB Storage to the NameNode.
Written By: Aman Gangwar
