Monitoring & Filesystems In Linux

Ayush Bhat

12 min readSep 19, 2021

Hello Folks! In this blog I will go through the various Tools provided by Linux Distributions.

I have briefly described the topics that will be covered in this blog:

Process and Load Monitoring utilities
Memory Monitoring utilities
I/O monitoring utilities
Linux Filesystems
Disk Partitioning
Filesystems : ext4, XFS and btrfs
Encrypting Disks
Logical Volume Management

So let’s get started ✨✨

Introduction

Most of the these utilities make use of mounted pseudo-filesystems that are /proc and /sys , these contains a lot of information about the system. Pseudo-filesystem totally exists in memory.

SAR stands for System Activity reporter. It is a tool that creates system reports that are readable by humans.

Syntax for sar

 sar [options] [interval] [count]

If I don’t provide any option it will show the report of CPU usage.
-w : this option shows report on context switching
-b : shows I/O and transfer rate statistics

Log Files : are essential for monitoring and troubleshooting. If you want to see the logs you can view all these in /var/log/syslog (in Ubuntu). If you are interested only in viewing the last few logs you can use the command shown in below screenshot.

You can view only kernel related messages by:

dmesg -w

Process Monitoring

In this part we will go through Process Monitoring Tools like ps, pstree and top utilities.

ps

Displays characteristics and statistics associated with the process.

Common Choice for ps 
ps -aux

Customizing ps output

You can print out a customized list of ps field with -o option.

pstree

This utility provides visual description of process ancestry.

top

Provides a dynamic real-time view of running process
Used to display processes with highest CPU usage
To interrupt use Ctrl-C

Memory Monitoring and Usage

Vocabulary that we will require here are:

Paging : It is a function of memory management where a computer will store and retrieve data from a device’s secondary storage to the primary storage.
Dirty memory : memory that needs to be written back to disk.
Swap Memory : It is a space in the Hard Disk of your computer that Operating Systems will use to put the info that is actually on the RAM to free it for another application.

Utilities which we will be using are free, vmstat and pmap.

free : gives brief summary memory usage, -m option displays the amount of memory in mebibytes.

/proc/meminfo : stores the information about how memory is being used.

/proc/sys/vm : It contains the tunable parameters to control virtual memory system.

vmstat : It reports virtual memory statistics. It is a multi-purpose tool that displays information about memory, paging, I/O ,processor and many other things.

#SYNTAX#
vmstat [option] [delay] [count]

option -S m is given, memory statistics will be in MB instead of KB.
-a option, vmstat displays information about active and inactive memory

Active memory is memory used by currently-running programs. Inactive memory was used recently by programs which have now been terminated

I/O Monitoring and Tuning

In this section we will explore utilities that will help to monitor I/O and also in tuning. To name few utilities, these are iostat, iotop and ionice.

iostat : It is used for monitoring I/O activity on the device.

Detailed Information

-k option will show result in KB
-x option will generate a detailed report
if %util approaches 100 that means the system is saturated

iotop: It displays table of current I/O usage and updates periodically. It needs to be run as root.

To see more option you can use iotop --help

ionice : This program has the capability to set or get the I/O scheduling class and priority for a process.

To get the scheduling class and priority of current shell, use:

I/O Scheduling

System performance depends heavily on optimizing I/O scheduling. Various aspects that we need to work so that system performance can be enhanced are:

Minimizing hardware access time
Ensuring data integrity
Able to prioritize important tasks

I/O scheduling also called disk scheduling. It is a method that computer use to decide in which order the block I/O operations will be submitted to storage volumes.

Requirements that I/O scheduling algorithms needs to satisfy are:

Requests should be satisfied with as low as latency
Process should share I/O bandwidth in a fair
Should favor reads over writes

IO operations per second (IOPS) is a measure of how many IO requests can be completed by storage device in a second.
Throughput is used to measure how much amount of data transferred from/to a storage device in a second. It measure no. of bits read or written per second.

I/O Scheduler Choices

To check which I/O schedulers are available use this command:

In iosched folder there are many parameters which we can change per scheduler.

Linux Filesystem and the VFS

Filesystem Basics

In Unix-like file system, every path or node is under the root (/) directory. Linux uses VFS to communicate with the filesystem software.

Inode : is the data structure on a disk that describes and stores file attributes, including it’s location. Every file in Linux filesystem is associated with it’s own inode.

Inode is used by operating system to keep track of properties like:

Size
Permission
User and Group ownership
Timestamps, last access and modification time

Hard and Soft Links

There are 2 ways to associate a file name and inodes.

Hard links points to an inode. All hard linked files have to be on the same filesystem.
Soft link also called symbolic link : It points to a file name which has associated inode. It maybe on different filesystem.

Screenshots below show how we can create hard and soft links.

Additional links can be created using ln command.

Virtual Filesystem : Application contacts to VFS abstraction layer to get the file and VFS provides the required file.

To get a list of supported filesystem in your OS, it can be found in /proc/filesystems.

Journaling filesystem

It recovers from system crashes or ungraceful shutdowns with little or no corruption and can do so very rapidly.

Some Journaling Filesystems are ext3, ext4, reiserfs, JFS, XFS and btrfs

XFS : RHEL 7 uses XFS as its default filesystem.
BTRFS : It is default filesystem for SUSE and OpenSUSE projects.

Special Filesystem

Linux uses special filesystem for accessing kernel data structure and for tuning kernel behavior. Some filesystem have mountpoints such as proc at /proc or sys at /sys and some do not like sockfs and pipefs .

Disk Partitioning

Common Disk types are USB, SATA, SCSI and SAS(better suited for servers)

Brief Introduction

We store a data into file and file is stored in directory which is inside the storage. Block storage is persistent storage. We can have 4 physical partitions and 64 bytes is reserved to store the partition table. Each P.T is of 16 bytes.

When we delete a file from windows or Linux it is not deleted , OS removed that file only from the inode table.

In electromechanical hard disk , we have platter, track ,sectors and head. When we create data as the head make holes (for 0 or for 1) and data gets stored.

We can access files in the system so quickly as internally OS goes to the inode table which tells the OS that in which sector the data is stored. When we reformat the hard disk all the entries in the inode table get deleted. But data still remains on the hard disk.

Partition Organization

As disks are divided into partitions. Partition is a contiguous region on the disk.

There are two partition schemes MBR(Master Boot Record) and GPT (GUID Partition Table) , MBR can have 4 primary partitions while GPT is on all modern devices, it may have up to 128 partitions.

Why Partitions ?

Security enhancements
Increase in performance
Separation of User and App data from operating system files

MBR Partition Table

The information in each partition table is:

Active bit
Beginning address in head
Partition type
Ending address
Start sector
Number of sectors in partition.

GPT Partition Table

There are 2 copies of the GPT header, at the beginning and at the end of the disk.

GPT Disk Partition Table

blkid utility shows information about partition. Used to locate block devices.

Unique UUID : describes the filesystem on the system.

Part UUID : describes partition and stays same even if filesystem is reformatted.

lsblk : shows block devices information in a tree format.

Partition Table Editors

There are a number of utilities which can be used to manage partition tables.

fdisk : I will be using this and is mostly available in all Linux distros.
sfdisk : It is useful for scripting
parted: GNU partition manipulation program
gparted: GUI for parted
gdisk : used for GPT system
sgdisk : can be used in script or CLI

Here I have provided the link for brief demonstration of working with fdisk. I have used it in RHEL 8. Click on the link below to see the demo.

RHEL 8 fdisk demo video

lsattr and chattr

We can set flag values to a file, these values are stored in file inode. These flags can be viewed by lsattr and set with chattr.

Flags:

Immutable
Append-Only
No-dump
No atime(access time) update

Below is screenshot of the working.

+a in chattr command adds the append only attribute in file.

mkfs : It is used to build a Linux filesystem on a device mostly on hard disk partition.

General format for mkfs

mkfs [-t fstype] [options] [device name]

fsck : Utility designed to check for errors and plus fix any that are found.
Note: fsck should only be run on unmounted filesystems.

mount : This program allows attaching at any point in the tree structure.

General form for mount

mount <options> <source> <directory>

Filesystem can be unmounted by:

umount [mount-point]

During system initialization, the following command mount -a is executed in order to mount all filesystems listed in /etc/fstab file.

Filesystem Features

df : This utility is used to look at filesystem usage

To check disk capacity and disk usage du utility is used. Below screenshot displays disk usage of current directory in human readable format.

Filesystem Quotas

Disk Quotas allow administrators to control the maximum space particular users are allowed.

Utilities that helps in managing quotas:

quotacheck : generates and updates quota accounting files.
quotaon: enables quota accounting
quotaoff : disable quota accounting
edquota : editing user/group quotas
quota: reports on usage and limits

In the below screenshot

First we check the limit of /dev/sdb1 for a user
Then created a file of 400 MiB approx.
Again checked the limit
Added more data to the file
It showed Disk Quota exceeded

If you want to edit the limits of a user use edquota

edquota -u [username]

EXT4 filesystem

ext4 was the default choice of Linux distributions because of it’s performance, integrity and stability. Some of features are:

Max. volume size1 EiB (for 4 KiB block size)
Max. file size16 TiB (for 4 KiB block size)
Max. number of files 4 billion (specified at filesystem creation time)
Max. filename length 255 bytes

EXT4 Superblock

Superblock : contains information about the filesystem, like:

Mount count and maximum mount count.
Block Size
Blocks Per group
Free Block count
Free inode count

Using dumpe2fs and tune2fs

dumpe2fs : It is used to get information about a particular partition.

tune2fs can be used to change filesystem parameters.

-l is used to list the contents of the superblock, including the current values of parameters which can be changed.

XFS and btrfs Filesystem

XFS feature:

Up to 16 EB (exabytes) for the total filesystem
Up to 8 EB for an individual file

XFS utilities:

xfsdump : for backup
xfsrestore : for restore
xfs_freeze : to make the filesystem for snapshots

btrfs

Created by Chris Mason
It is default root filesystem on openSUSE
Solves the issue of snapshots , checksums

Encrypting Disks

Encryption should be used wherever sensitive data is being stored and transmitted.

LUKS(Linux Unified Key Setup)

Linux distribution provides block device level encryption through the use of LUKS. LUKS is installed on top of Cryptsetup. In order to use LUKS format to encrypt block devices we need to install cryptsetup-bin.

Let’s encrypt /dev/sdb1, in my case this partition already exists but it must be unmounted.

Steps

Giving partition to LUKS, here you will be prompted for passphrase that will be used to open the encrypted volume later.

In case default encryption does not work , you can check available encryptions available in /proc/crypto

We can make volume available at any time with

Now we can format the partition

Mount it

Use it and store data

If you want to remove the encryption from the block device. First unmount it and then use luksClose flag in cryptsetup command.

umount /mnt

Logical Volume Management (LVM)

Let me give you an example so it would be easier to understand why to use LVM ?

Suppose I have movie which is 20 GB in size but one of disk is of 10 GB size and other is of 12 GB . Now I can’t use these two disks to store 20GB. To solve this problem we can use LVM. LVM helps to plug these 2 hard disk together and we can use them.

Steps to create Logical Volume