Elasticity Task

Arth Task7.1

🔅Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

🔅Increase or Decrease the Size of Static Partition in Linux.

🔅Automating LVM Partition using Python-Script.

LVM Creation

We have successfully added two storage drives on our virtual machine of size 3GB and 2GB.

Using the below command create a new physical volume for the disks attached.

# pvcreate <diskname> <diskname>

See all the PVs :

Now create Volume Group or VG by :

# vgcreate <name of VG> <pv1 name> <pv2 name>

Display all VGs :

Now from our VG of 5GB we are going to create a Logical volume of size 4GB using command :

# lvcreate — name <name for LV> — size <lv size> <VG name>

See all the LVs :

Using command # fdisk -l we can see the logical volume of size 4GB that we have created.

To use our Logical Volume, we have to format the volume first to create a fresh Inode Table. We will be using the ‘ext4’ file system to format our Logical Volume. To format, we use the command:

# mkfs.ext4 /dev/<VG Name>/<LV Name>

After formatting, we mount the storage on one directory called “task7” using the command:

# mount /dev/<VG Name>/<LV Name>   /<mount point>

Now we have to use this Logical Volume as our Data Node storage. So go to the Data Node configuration file “hdfs-site.xml” and set the <value> tag value to the Logical Volume directory.

Restart the hadoop cluster services and then we can see that our datanode is contributing 4GB storage.

We can create or store various files in this directory. What if we require more storage as our file size exceeds our LV size?

Now we want more storage but using static volume increment would require shutting down the system which we cannot afford here. So what to do?

LVM provides facility to increase or decrease the storage size on the fly easily without losing any data or shutting down the storage using the command:

# lvextend --size +<size to increase> /dev/<VG Name>/<LV Name>

we can see that our Logical Volume size has increased from 4GiB to 4.5GiB.

After increasing our Logical Volume size, we have to format it to create a fresh inode table. But we cannot lose our data while formatting, so there is a special command that helps us update our inode table without disturbing previous entries.

# resize2fs /dev/<VG Name>/<LV Name>

The above image shows the disk space before reformatting and after reformating.

Now we can see our datanode is now contributing 4.5GB of space.

Python Script to automate the whole process

When we run commands manually, it is prone to experience some error as we might forget some steps or there might be some typing mistake. Hence here is one python script making our task easier.

Here is the link to python script for automating LVM.

Thanks for reading !!