Monitoring & Filesystems In Linux
Hello Folks! In this blog I will go through the various Tools provided by Linux Distributions.
I have briefly described the topics that will be covered in this blog:
- Process and Load Monitoring utilities
- Memory Monitoring utilities
- I/O monitoring utilities
- Linux Filesystems
- Disk Partitioning
- Filesystems : ext4, XFS and btrfs
- Encrypting Disks
- Logical Volume Management
So let’s get started ✨✨
Introduction
Most of the these utilities make use of mounted pseudo-filesystems that are /proc and /sys , these contains a lot of information about the system. Pseudo-filesystem totally exists in memory.
SAR stands for System Activity reporter. It is a tool that creates system reports that are readable by humans.
Syntax for sar
sar [options] [interval] [count]
- If I don’t provide any option it will show the report of CPU usage.
- -w : this option shows report on context switching
- -b : shows I/O and transfer rate statistics
Log Files : are essential for monitoring and troubleshooting. If you want to see the logs you can view all these in /var/log/syslog (in Ubuntu). If you are interested only in viewing the last few logs you can use the command shown in below screenshot.
You can view only kernel related messages by:
dmesg -w
Process Monitoring
In this part we will go through Process Monitoring Tools like ps, pstree and top utilities.
ps
- Displays characteristics and statistics associated with the process.
Common Choice for ps
ps -aux
Customizing ps output
- You can print out a customized list of ps field with -o option.
pstree
This utility provides visual description of process ancestry.
top
- Provides a dynamic real-time view of running process
- Used to display processes with highest CPU usage
- To interrupt use Ctrl-C
Memory Monitoring and Usage
Vocabulary that we will require here are:
- Paging : It is a function of memory management where a computer will store and retrieve data from a device’s secondary storage to the primary storage.
- Dirty memory : memory that needs to be written back to disk.
- Swap Memory : It is a space in the Hard Disk of your computer that Operating Systems will use to put the info that is actually on the RAM to free it for another application.
Utilities which we will be using are free, vmstat and pmap.
- free : gives brief summary memory usage, -m option displays the amount of memory in mebibytes.
- /proc/meminfo : stores the information about how memory is being used.
- /proc/sys/vm : It contains the tunable parameters to control virtual memory system.
- vmstat : It reports virtual memory statistics. It is a multi-purpose tool that displays information about memory, paging, I/O ,processor and many other things.
#SYNTAX#
vmstat [option] [delay] [count]
- option -S m is given, memory statistics will be in MB instead of KB.
- -a option, vmstat displays information about active and inactive memory
Active memory is memory used by currently-running programs. Inactive memory was used recently by programs which have now been terminated
I/O Monitoring and Tuning
In this section we will explore utilities that will help to monitor I/O and also in tuning. To name few utilities, these are iostat, iotop and ionice.
iostat : It is used for monitoring I/O activity on the device.
Detailed Information
- -k option will show result in KB
- -x option will generate a detailed report
- if %util approaches 100 that means the system is saturated
iotop: It displays table of current I/O usage and updates periodically. It needs to be run as root.
To see more option you can use iotop --help
ionice : This program has the capability to set or get the I/O scheduling class and priority for a process.
To get the scheduling class and priority of current shell, use:
I/O Scheduling
System performance depends heavily on optimizing I/O scheduling. Various aspects that we need to work so that system performance can be enhanced are:
- Minimizing hardware access time
- Ensuring data integrity
- Able to prioritize important tasks
I/O scheduling also called disk scheduling. It is a method that computer use to decide in which order the block I/O operations will be submitted to storage volumes.
Requirements that I/O scheduling algorithms needs to satisfy are:
- Requests should be satisfied with as low as latency
- Process should share I/O bandwidth in a fair
- Should favor reads over writes
IO operations per second (IOPS) is a measure of how many IO requests can be completed by storage device in a second.
Throughput is used to measure how much amount of data transferred from/to a storage device in a second. It measure no. of bits read or written per second.
I/O Scheduler Choices
To check which I/O schedulers are available use this command:
In iosched folder there are many parameters which we can change per scheduler.
Linux Filesystem and the VFS
Filesystem Basics
In Unix-like file system, every path or node is under the root (/) directory. Linux uses VFS to communicate with the filesystem software.
Inode : is the data structure on a disk that describes and stores file attributes, including it’s location. Every file in Linux filesystem is associated with it’s own inode.
Inode is used by operating system to keep track of properties like:
- Size
- Permission
- User and Group ownership
- Timestamps, last access and modification time
Hard and Soft Links
There are 2 ways to associate a file name and inodes.
- Hard links points to an inode. All hard linked files have to be on the same filesystem.
- Soft link also called symbolic link : It points to a file name which has associated inode. It maybe on different filesystem.
Screenshots below show how we can create hard and soft links.
Additional links can be created using ln command.
Virtual Filesystem : Application contacts to VFS abstraction layer to get the file and VFS provides the required file.
To get a list of supported filesystem in your OS, it can be found in /proc/filesystems.
Journaling filesystem
- It recovers from system crashes or ungraceful shutdowns with little or no corruption and can do so very rapidly.
Some Journaling Filesystems are ext3, ext4, reiserfs, JFS, XFS and btrfs
- XFS : RHEL 7 uses XFS as its default filesystem.
- BTRFS : It is default filesystem for SUSE and OpenSUSE projects.
Special Filesystem
Linux uses special filesystem for accessing kernel data structure and for tuning kernel behavior. Some filesystem have mountpoints such as proc at /proc or sys at /sys and some do not like sockfs and pipefs .
Disk Partitioning
Common Disk types are USB, SATA, SCSI and SAS(better suited for servers)
Brief Introduction
We store a data into file and file is stored in directory which is inside the storage. Block storage is persistent storage. We can have 4 physical partitions and 64 bytes is reserved to store the partition table. Each P.T is of 16 bytes.
When we delete a file from windows or Linux it is not deleted , OS removed that file only from the inode table.
In electromechanical hard disk , we have platter, track ,sectors and head. When we create data as the head make holes (for 0 or for 1) and data gets stored.
We can access files in the system so quickly as internally OS goes to the inode table which tells the OS that in which sector the data is stored. When we reformat the hard disk all the entries in the inode table get deleted. But data still remains on the hard disk.
Partition Organization
As disks are divided into partitions. Partition is a contiguous region on the disk.
There are two partition schemes MBR(Master Boot Record) and GPT (GUID Partition Table) , MBR can have 4 primary partitions while GPT is on all modern devices, it may have up to 128 partitions.
Why Partitions ?
- Security enhancements
- Increase in performance
- Separation of User and App data from operating system files
MBR Partition Table
The information in each partition table is:
- Active bit
- Beginning address in head
- Partition type
- Ending address
- Start sector
- Number of sectors in partition.
GPT Partition Table
There are 2 copies of the GPT header, at the beginning and at the end of the disk.
GPT Disk Partition Table
- blkid utility shows information about partition. Used to locate block devices.
Unique UUID : describes the filesystem on the system.
Part UUID : describes partition and stays same even if filesystem is reformatted.
- lsblk : shows block devices information in a tree format.
Partition Table Editors
There are a number of utilities which can be used to manage partition tables.
- fdisk : I will be using this and is mostly available in all Linux distros.
- sfdisk : It is useful for scripting
- parted: GNU partition manipulation program
- gparted: GUI for parted
- gdisk : used for GPT system
- sgdisk : can be used in script or CLI
Here I have provided the link for brief demonstration of working with fdisk. I have used it in RHEL 8. Click on the link below to see the demo.
lsattr and chattr
We can set flag values to a file, these values are stored in file inode. These flags can be viewed by lsattr and set with chattr.
Flags:
- Immutable
- Append-Only
- No-dump
- No atime(access time) update
Below is screenshot of the working.
- +a in chattr command adds the append only attribute in file.
mkfs : It is used to build a Linux filesystem on a device mostly on hard disk partition.
General format for mkfs
mkfs [-t fstype] [options] [device name]
fsck : Utility designed to check for errors and plus fix any that are found.
Note: fsck should only be run on unmounted filesystems.
mount : This program allows attaching at any point in the tree structure.
General form for mount
mount <options> <source> <directory>
Filesystem can be unmounted by:
umount [mount-point]
- During system initialization, the following command mount -a is executed in order to mount all filesystems listed in /etc/fstab file.
Filesystem Features
- df : This utility is used to look at filesystem usage
- To check disk capacity and disk usage du utility is used. Below screenshot displays disk usage of current directory in human readable format.
Filesystem Quotas
Disk Quotas allow administrators to control the maximum space particular users are allowed.
Utilities that helps in managing quotas:
- quotacheck : generates and updates quota accounting files.
- quotaon: enables quota accounting
- quotaoff : disable quota accounting
- edquota : editing user/group quotas
- quota: reports on usage and limits
In the below screenshot
- First we check the limit of /dev/sdb1 for a user
- Then created a file of 400 MiB approx.
- Again checked the limit
- Added more data to the file
- It showed Disk Quota exceeded
If you want to edit the limits of a user use edquota
edquota -u [username]
EXT4 filesystem
ext4 was the default choice of Linux distributions because of it’s performance, integrity and stability. Some of features are:
- Max. volume size1 EiB (for 4 KiB block size)
- Max. file size16 TiB (for 4 KiB block size)
- Max. number of files 4 billion (specified at filesystem creation time)
- Max. filename length 255 bytes
EXT4 Superblock
Superblock : contains information about the filesystem, like:
- Mount count and maximum mount count.
- Block Size
- Blocks Per group
- Free Block count
- Free inode count
Using dumpe2fs and tune2fs
- dumpe2fs : It is used to get information about a particular partition.
- tune2fs can be used to change filesystem parameters.
-l is used to list the contents of the superblock, including the current values of parameters which can be changed.
XFS and btrfs Filesystem
XFS feature:
- Up to 16 EB (exabytes) for the total filesystem
- Up to 8 EB for an individual file
XFS utilities:
- xfsdump : for backup
- xfsrestore : for restore
- xfs_freeze : to make the filesystem for snapshots
btrfs
- Created by Chris Mason
- It is default root filesystem on openSUSE
- Solves the issue of snapshots , checksums
Encrypting Disks
Encryption should be used wherever sensitive data is being stored and transmitted.
LUKS(Linux Unified Key Setup)
Linux distribution provides block device level encryption through the use of LUKS. LUKS is installed on top of Cryptsetup. In order to use LUKS format to encrypt block devices we need to install cryptsetup-bin.
Let’s encrypt /dev/sdb1, in my case this partition already exists but it must be unmounted.
Steps
- Giving partition to LUKS, here you will be prompted for passphrase that will be used to open the encrypted volume later.
- In case default encryption does not work , you can check available encryptions available in /proc/crypto
- We can make volume available at any time with
- Now we can format the partition
- Mount it
Use it and store data
- If you want to remove the encryption from the block device. First unmount it and then use luksClose flag in cryptsetup command.
umount /mnt
Logical Volume Management (LVM)
Let me give you an example so it would be easier to understand why to use LVM ?
Suppose I have movie which is 20 GB in size but one of disk is of 10 GB size and other is of 12 GB . Now I can’t use these two disks to store 20GB. To solve this problem we can use LVM. LVM helps to plug these 2 hard disk together and we can use them.
Steps to create Logical Volume
- Install required packages
- Create a partition
- Create physical volumes from the partition
- Create Volume group.
- Allocate logical volume from volume group. I have set the logical volume size to 500MB
- Format the logical volume
- Mount the logical volume
- Show the physical volume
- Show the volume group
- Show the logical volume
That’s all for now. Thanks for reading.