Monitoring & Filesystems In Linux

Ayush Bhat
12 min readSep 19, 2021

--

Hello Folks! In this blog I will go through the various Tools provided by Linux Distributions.

I have briefly described the topics that will be covered in this blog:

  • Process and Load Monitoring utilities
  • Memory Monitoring utilities
  • I/O monitoring utilities
  • Linux Filesystems
  • Disk Partitioning
  • Filesystems : ext4, XFS and btrfs
  • Encrypting Disks
  • Logical Volume Management
Source Internet

So let’s get started ✨✨

Introduction

Most of the these utilities make use of mounted pseudo-filesystems that are /proc and /sys , these contains a lot of information about the system. Pseudo-filesystem totally exists in memory.

SAR stands for System Activity reporter. It is a tool that creates system reports that are readable by humans.

Syntax for sar

 sar [options] [interval] [count]
Used Ubuntu image
  • If I don’t provide any option it will show the report of CPU usage.
  • -w : this option shows report on context switching
  • -b : shows I/O and transfer rate statistics

Log Files : are essential for monitoring and troubleshooting. If you want to see the logs you can view all these in /var/log/syslog (in Ubuntu). If you are interested only in viewing the last few logs you can use the command shown in below screenshot.

You can view only kernel related messages by:

dmesg -w

Process Monitoring

In this part we will go through Process Monitoring Tools like ps, pstree and top utilities.

ps

  • Displays characteristics and statistics associated with the process.
Common Choice for ps 
ps -aux

Customizing ps output

  • You can print out a customized list of ps field with -o option.

pstree

This utility provides visual description of process ancestry.

top

  • Provides a dynamic real-time view of running process
  • Used to display processes with highest CPU usage
  • To interrupt use Ctrl-C
Output of top command

Memory Monitoring and Usage

Vocabulary that we will require here are:

  1. Paging : It is a function of memory management where a computer will store and retrieve data from a device’s secondary storage to the primary storage.
  2. Dirty memory : memory that needs to be written back to disk.
  3. Swap Memory : It is a space in the Hard Disk of your computer that Operating Systems will use to put the info that is actually on the RAM to free it for another application.

Utilities which we will be using are free, vmstat and pmap.

  • free : gives brief summary memory usage, -m option displays the amount of memory in mebibytes.
  • /proc/meminfo : stores the information about how memory is being used.
  • /proc/sys/vm : It contains the tunable parameters to control virtual memory system.
  • vmstat : It reports virtual memory statistics. It is a multi-purpose tool that displays information about memory, paging, I/O ,processor and many other things.
#SYNTAX#
vmstat [option] [delay] [count]
Used ubuntu
  • option -S m is given, memory statistics will be in MB instead of KB.
  • -a option, vmstat displays information about active and inactive memory

Active memory is memory used by currently-running programs. Inactive memory was used recently by programs which have now been terminated

I/O Monitoring and Tuning

In this section we will explore utilities that will help to monitor I/O and also in tuning. To name few utilities, these are iostat, iotop and ionice.

iostat : It is used for monitoring I/O activity on the device.

Detailed Information

  • -k option will show result in KB
  • -x option will generate a detailed report
  • if %util approaches 100 that means the system is saturated

iotop: It displays table of current I/O usage and updates periodically. It needs to be run as root.

To see more option you can use iotop --help

ionice : This program has the capability to set or get the I/O scheduling class and priority for a process.

To get the scheduling class and priority of current shell, use:

I/O Scheduling

System performance depends heavily on optimizing I/O scheduling. Various aspects that we need to work so that system performance can be enhanced are:

  • Minimizing hardware access time
  • Ensuring data integrity
  • Able to prioritize important tasks

I/O scheduling also called disk scheduling. It is a method that computer use to decide in which order the block I/O operations will be submitted to storage volumes.

Requirements that I/O scheduling algorithms needs to satisfy are:

  • Requests should be satisfied with as low as latency
  • Process should share I/O bandwidth in a fair
  • Should favor reads over writes

IO operations per second (IOPS) is a measure of how many IO requests can be completed by storage device in a second.

Throughput is used to measure how much amount of data transferred from/to a storage device in a second. It measure no. of bits read or written per second.

I/O Scheduler Choices

To check which I/O schedulers are available use this command:

Ubuntu

In iosched folder there are many parameters which we can change per scheduler.

In RHEL 8

Linux Filesystem and the VFS

Filesystem Basics

In Unix-like file system, every path or node is under the root (/) directory. Linux uses VFS to communicate with the filesystem software.

Inode : is the data structure on a disk that describes and stores file attributes, including it’s location. Every file in Linux filesystem is associated with it’s own inode.

Inode is used by operating system to keep track of properties like:

  • Size
  • Permission
  • User and Group ownership
  • Timestamps, last access and modification time

Hard and Soft Links

There are 2 ways to associate a file name and inodes.

  • Hard links points to an inode. All hard linked files have to be on the same filesystem.
  • Soft link also called symbolic link : It points to a file name which has associated inode. It maybe on different filesystem.

Screenshots below show how we can create hard and soft links.

Additional links can be created using ln command.

Virtual Filesystem : Application contacts to VFS abstraction layer to get the file and VFS provides the required file.

To get a list of supported filesystem in your OS, it can be found in /proc/filesystems.

Journaling filesystem

Source intenet
  • It recovers from system crashes or ungraceful shutdowns with little or no corruption and can do so very rapidly.

Some Journaling Filesystems are ext3, ext4, reiserfs, JFS, XFS and btrfs

  • XFS : RHEL 7 uses XFS as its default filesystem.
  • BTRFS : It is default filesystem for SUSE and OpenSUSE projects.

Special Filesystem

Linux uses special filesystem for accessing kernel data structure and for tuning kernel behavior. Some filesystem have mountpoints such as proc at /proc or sys at /sys and some do not like sockfs and pipefs .

Disk Partitioning

Common Disk types are USB, SATA, SCSI and SAS(better suited for servers)

Brief Introduction

We store a data into file and file is stored in directory which is inside the storage. Block storage is persistent storage. We can have 4 physical partitions and 64 bytes is reserved to store the partition table. Each P.T is of 16 bytes.

When we delete a file from windows or Linux it is not deleted , OS removed that file only from the inode table.

In electromechanical hard disk , we have platter, track ,sectors and head. When we create data as the head make holes (for 0 or for 1) and data gets stored.

We can access files in the system so quickly as internally OS goes to the inode table which tells the OS that in which sector the data is stored. When we reformat the hard disk all the entries in the inode table get deleted. But data still remains on the hard disk.

Partition Organization

As disks are divided into partitions. Partition is a contiguous region on the disk.

There are two partition schemes MBR(Master Boot Record) and GPT (GUID Partition Table) , MBR can have 4 primary partitions while GPT is on all modern devices, it may have up to 128 partitions.

Why Partitions ?

  • Security enhancements
  • Increase in performance
  • Separation of User and App data from operating system files

MBR Partition Table

Source Internet

The information in each partition table is:

  • Active bit
  • Beginning address in head
  • Partition type
  • Ending address
  • Start sector
  • Number of sectors in partition.

GPT Partition Table

Source Internet

There are 2 copies of the GPT header, at the beginning and at the end of the disk.

GPT Disk Partition Table

  • blkid utility shows information about partition. Used to locate block devices.

Unique UUID : describes the filesystem on the system.

Part UUID : describes partition and stays same even if filesystem is reformatted.

  • lsblk : shows block devices information in a tree format.

Partition Table Editors

There are a number of utilities which can be used to manage partition tables.

  • fdisk : I will be using this and is mostly available in all Linux distros.
  • sfdisk : It is useful for scripting
  • parted: GNU partition manipulation program
  • gparted: GUI for parted
  • gdisk : used for GPT system
  • sgdisk : can be used in script or CLI

Here I have provided the link for brief demonstration of working with fdisk. I have used it in RHEL 8. Click on the link below to see the demo.

RHEL 8 fdisk demo video

lsattr and chattr

We can set flag values to a file, these values are stored in file inode. These flags can be viewed by lsattr and set with chattr.

Flags:

  • Immutable
  • Append-Only
  • No-dump
  • No atime(access time) update

Below is screenshot of the working.

  • +a in chattr command adds the append only attribute in file.

mkfs : It is used to build a Linux filesystem on a device mostly on hard disk partition.

General format for mkfs

mkfs [-t fstype] [options] [device name]

fsck : Utility designed to check for errors and plus fix any that are found.
Note: fsck should only be run on unmounted filesystems.

mount : This program allows attaching at any point in the tree structure.

General form for mount

mount <options> <source> <directory>

Filesystem can be unmounted by:

umount [mount-point]
  • During system initialization, the following command mount -a is executed in order to mount all filesystems listed in /etc/fstab file.

Filesystem Features

  • df : This utility is used to look at filesystem usage
  • To check disk capacity and disk usage du utility is used. Below screenshot displays disk usage of current directory in human readable format.

Filesystem Quotas

Disk Quotas allow administrators to control the maximum space particular users are allowed.

Utilities that helps in managing quotas:

  • quotacheck : generates and updates quota accounting files.
  • quotaon: enables quota accounting
  • quotaoff : disable quota accounting
  • edquota : editing user/group quotas
  • quota: reports on usage and limits

In the below screenshot

  • First we check the limit of /dev/sdb1 for a user
  • Then created a file of 400 MiB approx.
  • Again checked the limit
  • Added more data to the file
  • It showed Disk Quota exceeded

If you want to edit the limits of a user use edquota

edquota -u [username]

EXT4 filesystem

ext4 was the default choice of Linux distributions because of it’s performance, integrity and stability. Some of features are:

  • Max. volume size1 EiB (for 4 KiB block size)
  • Max. file size16 TiB (for 4 KiB block size)
  • Max. number of files 4 billion (specified at filesystem creation time)
  • Max. filename length 255 bytes

EXT4 Superblock

Superblock : contains information about the filesystem, like:

  • Mount count and maximum mount count.
  • Block Size
  • Blocks Per group
  • Free Block count
  • Free inode count

Using dumpe2fs and tune2fs

  • dumpe2fs : It is used to get information about a particular partition.
  • tune2fs can be used to change filesystem parameters.

-l is used to list the contents of the superblock, including the current values of parameters which can be changed.

XFS and btrfs Filesystem

XFS feature:

  • Up to 16 EB (exabytes) for the total filesystem
  • Up to 8 EB for an individual file

XFS utilities:

  • xfsdump : for backup
  • xfsrestore : for restore
  • xfs_freeze : to make the filesystem for snapshots

btrfs

  • Created by Chris Mason
  • It is default root filesystem on openSUSE
  • Solves the issue of snapshots , checksums

Encrypting Disks

Encryption should be used wherever sensitive data is being stored and transmitted.

LUKS(Linux Unified Key Setup)

Linux distribution provides block device level encryption through the use of LUKS. LUKS is installed on top of Cryptsetup. In order to use LUKS format to encrypt block devices we need to install cryptsetup-bin.

Let’s encrypt /dev/sdb1, in my case this partition already exists but it must be unmounted.

Steps

  • Giving partition to LUKS, here you will be prompted for passphrase that will be used to open the encrypted volume later.
  • In case default encryption does not work , you can check available encryptions available in /proc/crypto
  • We can make volume available at any time with
  • Now we can format the partition
  • Mount it

Use it and store data

  • If you want to remove the encryption from the block device. First unmount it and then use luksClose flag in cryptsetup command.
umount /mnt

Logical Volume Management (LVM)

Let me give you an example so it would be easier to understand why to use LVM ?

Suppose I have movie which is 20 GB in size but one of disk is of 10 GB size and other is of 12 GB . Now I can’t use these two disks to store 20GB. To solve this problem we can use LVM. LVM helps to plug these 2 hard disk together and we can use them.

Steps to create Logical Volume

  • Install required packages
  • Create a partition
  • Create physical volumes from the partition
  • Create Volume group.
  • Allocate logical volume from volume group. I have set the logical volume size to 500MB
  • Format the logical volume
  • Mount the logical volume
  • Show the physical volume
  • Show the volume group
  • Show the logical volume

That’s all for now. Thanks for reading.

--

--

Ayush Bhat
Ayush Bhat

Written by Ayush Bhat

AWS SAA-C02 | Certified Kubernetes Administrator | Linux Foundation Certified Sysadmin | Ex294 RedHat Certified Engineer

No responses yet