Reading MAN Pages
SYNOPSIS
ls [OPTION]... [FILE]...
- [brackets] - Optional arguments/switches
...- Represents an arbitrary length of items of a specific type- [option1 | option2] - Pipe | symbol is OR
e.g.
ls -la /home/ /etc/
SYNOPSIS
mv [OPTION]... [-T] SOURCE DEST
mv [OPTION]... SOURCE... DIRECTORY
mv [OPTION]... -t DIRECTORY SOURCE...
mv [A number of arguments (...)] , One source and one destination
mv [OPTION]... [-T] SOURCE DEST
mv [n arguments], n sources, and only ONE directory
mv [OPTION]... SOURCE... DIRECTORY
date -u or --utc or --universal
date [-u|--utc|--universal]
Standard Output
Concatenates motd and issue and sends to standard output
cat motd issue
> Redirect standard output to file
>> Redirect stdout and append to file
< Redirect file into a command that will accept standard input e.g.
mysql table_name < backup.sql
2> Redirect standard error
&> Redirect standard error and standard output to file
2>&1 Redirect Standard error to standard output - This is useful if you want to send the output to a command that would normally only accept standard output e.g. the pipe command |
`$ asasdjasjk 2>&1 | grep command
Additional useful terminal commands
head, tail, less, more
Grep and Regex (Regular Expressions)
grep '^#' file.txt - Line starts with '#'
grep 'endingword$' file.txt - '$' Line ends with 'endingword'
grep [lL]inux file.txt - [] word start with lower case OR upper case 'L'
grep '[^linux]' file.txt - '^' inside [] = Do not match
grep -E '(a)+' file.txt - '-E' Extended grep - '+' means match the preceding character 'a' one or more times
grep 'l...x' file.txt - Match 'l' then any 3 characters and then 'x'
SSH
touch {file1,file2,file3} - '{}' allows you to create multiple files at once
ssh user@ip ls - SSH's to remote system and issues ls command and disconnects
scp - Send receive files between remote systems
sftp - get/put/mkdir etc
Login and switch users on Multi-User Target
systemctl get-default - Outputs current target `multi-user.target' or 'graphical.target'
su root - Login as root user, but hasn't loaded profile customisations. This is just an 'interactive' shell. A 'login' shell requires loading of customisation file and environment variables
su -|-l|--login <username> - Represents loading login shell
.bash_profile - Executed on login to Login Shell e.g. su - ubuntu
.bash_logout - Executed on logout of the Login Shell
.basrc - Executed on login to an Interactive shell e.g. su root
/etc/profile - Global system wide customisation for any Login Shell
Archiving
tar - Tape Archiving - Doesn't natively support compression, but 3rd party compression can be accessed from it with the appropriate switch
gzip - gzip Compress an individual file - You can pass multiple files but each file passed will be compressed individually
n.b. You cannot compress directories e.g.
$ gzip hello1 hello2
$ ls -la h*
-rw-r--r-- 1 as as 27 Aug 3 08:52 hello1.gz
-rw-r--r-- 1 as as 27 Aug 3 08:52 hello2.gz
$ gzip directory1/
gzip: directory1/ is a directory -- ignored
gunzip - gunzip - Decompress individual files or you can use gzip -d <filename>
Compress multiple files and directories, list, then compress with gzip
tar -cvf
-c - Create Archive
-v - Verbosely - Show all operations e.g. adding of files directories
-tf - List tar archive contents as specified with -f
-f - Specify the file to create - myarchive1.tar
$ mkdir directory1 && cd $_ && touch {file1,file2,file3,file4} && cd - && touch {hello1,hello2}
$ tar -cvf myarchive1.tar directory1/ hello1 hello2
$ tar -tf myarchive1.tar
directory1/
directory1/file2
directory1/file3
directory1/file4
directory1/file1
hello1
hello2
$ gzip myarchive1.tar
$ ls -la myarchi*
-rw-r--r-- 1 as as 216 Aug 3 09:08 myarchive1.tar.gz
Pass the -z flag to pass the created archive into gzip in one command
$ tar -cvzf myarchive1.tar.gz directory1/ hello1 hello2
directory1/
directory1/file2
directory1/file3
directory1/file4
directory1/file1
hello1
hello2
$ ls -l myarchive1*
-rw-r--r-- 1 as as 201 Aug 3 09:15 myarchive1.tar.gz
Alternative compression algorithm bzip2 (block-sorting file compressor) can be invoked with -j instead of -z
When extracting files using tar into a directory with files of the same name, tar will overwrite those files in the current directory - Inspect the files first -d before extracting
$ echo 'hello world' > hello1
$ tar -dvf myarchive1.tar.gz
directory1/
directory1/file2
directory1/file3
directory1/file4
directory1/file1
hello1
hello1: Mod time differs
hello1: Size differs
hello2
Happy with overwriting the files? - Extract and decompress -x
$ cat hello1
hello world
$ tar -xvzf myarchive1.tar.gz
directory1/
directory1/file2
directory1/file3
directory1/file4
directory1/file1
hello1
hello2
$ cat hello1
$
Show compression information -l for the created archive
$ gzip -l myarchive1.tar.gz
compressed uncompressed ratio uncompressed_name
201 10240 98.2% myarchive1.tar
star - standard tape archiver is a useful alternative to tar with some useful features, such as extracting individual files or warning you if you're about to overwrite a file in you current directory that has the same name as in the archive - This seems to be a centOS/RHEL only package and is outside the scope of the certification
star -c -f=myarchive.tar directory1/ hello1 hello2
star -t -f=myarchive.tar
star -x -f=myarchive.tar - Won't override newer operating system files - Gives us more protection
star -x -f=myarchive.tar hello1 - Extract individual file
Create and Edit Text Files
3rd Party - i.e. non-default installed text editors
nano
emacs
vi - Always on a system by default
vim - vi improved - Sometimes installed by default and with additional features
vi Opens in command mode by default - 3 Modes available:
commandmode -ESCinsertmode -ixmode -shift:
i - Insert text
a - Append text, so move cursor to right if at the end of a line
ESC - Move from insert mode to command mode
q! - quit ! immediately
l - Move cursor to the right
h - Move cursor to the left
j - Move cursor down
k - Move cursor up
x - Delete character the cursor is on
yy - Yank i.e. copy the whole line
p - Paste the yanked text in line below the cursor
u - Undo
R - Replace text from point cursor is located
dd - Cut the line, that can then be pasted elsewhere
5dd - Cut 5 lines starting with line cursor is on
3yy - Copy 5 lines starting with line the cursor is on
1G - 1 = line number, shift g - Navigates cursor to line 1 of the file, can also use e.g. 3gg to navigate to line 3 of the file
G - Shift g - Navigate to end of file
cw - Cut word from where the cursor is placed and transition to insert mode
cc - Same as dd but will transition to insert mode
:?hello - shift+:,?<word to search for> - Search for text. If text entered in lower case, all text regardless of case is returned, but if specified with upper case characters words only matching the given combination of upper and lower case characters will be returned
/ - /hello will also search for the word specified
:%s/hello/world - In x mode, replace %s the word /hello with the new word specified /world - Only the first occurance of the word on a given line
:%s/hello/world/g - Same as above, but all instances /g (global) of the word in a file
:w - Save file changes
:wq - Save and quit
:!ls /etc - In x mode, issue command ! ls /etc outside the file for the directory specified
:!docker ps -a - Run the docker ps -a command from the vi application
Create, Delete, Copy and Move File and Directories
mv - Move & rename files and directories
mkdir - Create directory -p for directory hierarchy
rmdir - Remove directory but only if empty
rm -rf - Remove directory and recursively all files within it
cp - Copy files
cp -R - Copy directory and all files within it -R recursively
Hard Links and Soft Links on a System
ln -s /etc/motd motd - Create a symlink
~$ ln -s /etc/os-release os-release
~$ ls -l os-release
lrwxrwxrwx 1 as as 15 Aug 3 11:16 os-release -> /etc/os-release
~$ ls -l /etc/os-release
lrwxrwxrwx 1 root root 21 Jul 23 20:42 /etc/os-release -> ../usr/lib/os-release
~$ ls -l /usr/lib/os-release
-rw-r--r-- 1 root root 386 Jul 23 20:42 /usr/lib/os-release
n.b. Permissions on symlink don't matter - If in doubt look at the source file permissions e.g. If you create a symlink to a root root file in your home directory as a non-root user and try and edit that file with vi, it'll tell you that the file is read-only
~$ echo 'Append this text to the end of the file' >> os-release
bash: os-release: Permission denied
~$ sudo sh -c "echo 'Append this text to the end of the file' >> os-release'"
~$ tail -n 3 os-release
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Append this text to the end of the file
Hard Links are a link to a specific inode location on a filesystem. On a filesystem an inode provides information about files and directories. Hard links cannot link across filesystems, symlinks you can.
~$ touch hardlinktestfile && ls -l hardlinktestfile
-rw-r--r-- 1 as as 0 Aug 3 11:41 hardlinktestfile
~$ echo 'Text in hardlinktestfile' > hardlinktestfile && ls -l hardlinktestfile && cat hardlinktestfile
-rw-r--r-- 1 as as 25 Aug 3 11:44 hardlinktestfile
Text in hardlinktestfile
n.b. links to inodes is represented by the number after the permissions -rw-r--r--. In the example above it's currently 1
If we now create a hardlink, the link will be created and the link number will increment to 2 - This is now showing there's 2x links to the same inode on disk
~$ ln hardlinktestfile HARDLINKTESTFILE && ls -l hardlinktestfile HARDLINKTESTFILE
-rw-r--r-- 2 as as 25 Aug 3 11:44 hardlinktestfile
-rw-r--r-- 2 as as 25 Aug 3 11:44 HARDLINKTESTFILE
Any editing in either file will be represented in both files
~$ echo "This is a newline in the HARDLINKTESTFILE" >> HARDLINKTESTFILE
~$ cat hardlinktestfile
Text in hardlinktestfile
This is a newline in the HARDLINKTESTFILE
Whereas permissions for symlinks don't apply on the linked file compared to the source, hardlink permissions are the same on both files as shown here when the execute permission is applied
~$ ls -l hardlinktestfile HARDLINKTESTFILE
-rw-r--r-- 2 as as 67 Aug 3 11:50 hardlinktestfile
-rw-r--r-- 2 as as 67 Aug 3 11:50 HARDLINKTESTFILE
~$ chmod u+x hardlinktestfile && ls -l hardlinktestfile HARDLINKTESTFILE
-rwxr--r-- 2 as as 67 Aug 3 11:50 hardlinktestfile
-rwxr--r-- 2 as as 67 Aug 3 11:50 HARDLINKTESTFILE
If we remove the original source file, the hardlinked file will persist along with all it's contents. The number of links to the inode will decrease
~$ rm -rf hardlinktestfile
~$ ls -l HARDLINKTESTFILE
-rwxr--r-- 1 as as 67 Aug 3 11:50 HARDLINKTESTFILE
~$ cat HARDLINKTESTFILE
Text in hardlinktestfile
This is a newline in the HARDLINKTESTFILE
- Hard links link directly to the inode source on the hard drive.
- If the source or target file is removed the created link still exists and the data is available on the inode until all links to the inode are no longer available or no longer exist
- All hard links linking to the same files will share the same datetimes as well as permissions
- After a new hardlink is created it is not possible to determine which was the original file
List, Set and Change Standard Permissions
umask - Defines the default set of permissions that are assigned when a user creates a file
~$ umask
0022
as@ubuntu:~$ umask -S
u=rwx,g=rx,o=rx
Understanding the bits assigned to files, directories, symlinks
~$ ls -l | egrep 'directory1|HARDLINKTESTFILE|os-release'
drwxr-xr-x 2 as as 4096 Aug 3 08:52 directory1
-rwxr--r-- 1 as as 67 Aug 3 11:50 HARDLINKTESTFILE
lrwxrwxrwx 1 as as 15 Aug 3 11:16 os-release -> /etc/os-release
-rwxr--r-- - Divided into 3x slots after first bit -|rwx|r--|r--
d - Directory
- - File
l - Symlink
All bits after the first one above apply to permissions. When you're changing permissions you're 'flipping' bits
Using this as an example:
-rwxr--r-- 1 as as 67 Aug 3 11:50 HARDLINKTESTFILE
- - This is a file
rwx - This is the permissions for the owner of the file {read, write, execute}
r-- - This is the permissions for the group that owns the file {read, denied write, denied execute}
r-- - This is other or everyone else on the system {read, denied write, denied execute}
1 - The number of links to the file
as - The owner of the file
as - The group owner of the file - Always user when file created under logged in user unless they have a different default group
Symbolic Notation to change permissions on a file, directory or symlink - Octal Notation uses the bit value e.g. 7
chmod u+x my_script.sh - Give user execute permissions
chmod g-rwx my_script.sh - Remove group read, write, execute permissions
-u | - User (owner) permissions
-g | - Group permissions
-o | - Other permissions
+|- r - Read permission
+|- w - Write permission
+|- x - Execute permission
getent group - Output all the created groups on the system
cat /etc/group - Same as above
groupadd finance - Create a new group called finance
chown user:group - Change the user or group owner of the file or directory
Example - Create a group called finance, create a directory as current user, and change only the group ownership of that directory to the group finance
~$ sudo groupadd finance
[sudo] password for as:
~$ getent group | grep finance
finance:x:1001:
~$ mkdir finance
~$ ls -l | grep finance
drwxr-xr-x 2 as as 4096 Aug 3 12:37 finance
~$ sudo chown :finance finance
as@ubuntu:~$ ls -l | grep finance
drwxr-xr-x 2 as finance 4096 Aug 3 12:37 finance
n.b. x - Execute permissions are required to be able to navigate into a directory
Change permissions on directory to root as the owner and finance as the group owner
~$ sudo chown root:finance finance
~$ ls -l | grep finance
drwxr-x--- 2 root finance 4096 Aug 3 12:37 finance
as@ubuntu:~$ cd finance/
bash: cd: finance/: Permission denied
Give access so the as user can cd into and read the contents of the finance directory
sudo usermod -G finance as
You cannot log into the directory until you've opened a new shell. You can switch to root and the switch back to as
~$ cd finance/
bash: cd: finance/: Permission denied
~$ su - root
Password:
~# su - as
~$ cd finance/
~/finance$
Files inherit the permissions from their parent directory. Any new files created in the directory would inherit the correct permissions, but the existing files would need to have the permission recursively chmod g+w -R written to them e.g.
root:/home/as# ls -l | grep finance && ls -l finance/ | grep -v total
drwxr-x--- 2 root finance 4096 Aug 3 14:09 finance
-rw-r--r-- 1 root root 15 Aug 3 14:10 myfile
root:/home/as# chmod g+w finance/
root:/home/as# ls -l | grep finance && ls -l finance/ | grep -v total
drwxrwx--- 2 root finance 4096 Aug 3 14:09 finance
-rw-r--r-- 1 root root 15 Aug 3 14:10 myfile
as:~/finance$ touch myfile2 && ls -l
total 4
-rw-r--r-- 1 root root 15 Aug 3 14:10 myfile
-rw-rw-r-- 1 as as 0 Aug 3 14:20 myfile2
root:/home/as# chmod g+w -R finance/ && ls -l finance/
total 4
-rw-rw-r-- 1 root root 15 Aug 3 14:10 myfile
-rw-rw-r-- 1 as as 0 Aug 3 14:20 myfile2
Set all permissions for a file a
as:~/finance$ chmod ugo-rw myfile2
as:~/finance$ ls -l
-rw-rw-r-- 1 root root 15 Aug 3 14:10 myfile
---------- 1 as as 0 Aug 3 14:20 myfile2
:~/finance$ chmod a+r myfile2
:~/finance$ ls -l
-rw-rw-r-- 1 root root 15 Aug 3 14:10 myfile
-r--r--r-- 1 as as 0 Aug 3 14:20 myfile2
Set execute privileges for directories recursively +X when used with -R
as:~$ ls -l | grep finance && ls -l finance/ | grep -v total
drwxrwx--- 2 root finance 4096 Aug 3 14:20 finance
-rw-rw-r-- 1 root root 15 Aug 3 14:10 myfile
-r--r--r-- 1 as as 0 Aug 3 14:20 myfile2
as:~$ mkdir finance/{dir1,dir2,dir3}
as:~$ chmod a-x finance/dir*
as@ubuntu:~$ chmod a-r finance/myfile2 && ls -l finance/ | grep -v total
drw-rw-r-- 2 as as 4096 Aug 3 14:38 dir1
drw-rw-r-- 2 as as 4096 Aug 3 14:38 dir2
drw-rw-r-- 2 as as 4096 Aug 3 14:38 dir3
-rw-rw-r-- 1 root root 15 Aug 3 14:10 myfile
---------- 1 as as 0 Aug 3 14:20 myfile2
~/finance$ sudo chmod ugo-x -R ../finance/
as:~/finance$ ls -l
ls: cannot open directory '.': Permission denied
as:~$ ls -l finance/
ls: cannot access 'finance/myfile': Permission denied
ls: cannot access 'finance/dir2': Permission denied
ls: cannot access 'finance/dir1': Permission denied
ls: cannot access 'finance/dir3': Permission denied
total 0
d????????? ? ? ? ? ? dir1
d????????? ? ? ? ? ? dir2
d????????? ? ? ? ? ? dir3
-????????? ? ? ? ? ? myfile
as@ubuntu:~$ sudo chmod ugo+X -R finance/
as@ubuntu:~$ ls -l finance/
total 16
drwxrwxr-x 2 as as 4096 Aug 3 14:38 dir1
drwxrwxr-x 2 as as 4096 Aug 3 14:38 dir2
drwxrwxr-x 2 as as 4096 Aug 3 14:38 dir3
---------- 1 root root 15 Aug 3 14:10 myfile
Octal Notation
read=4
write=2
execute=1
Total = 7
chmod 4|0|0 - 4 = Read of owner slot, 0 or no permissions for group and other slots
as:~/finance$ chmod ugo-r myfile && ls -l myfile
---------- 1 as finance 15 Aug 3 14:10 myfile
as@ubuntu:~/finance$ chmod 400 myfile && ls -l myfile
-r-------- 1 as finance 15 Aug 3 14:10 myfile
setuid, setgid
setgid = Permission bit, that forks a new process as the person executing the file. An example would be; say I want the file to be executed with the same privileges as the user that owns the file and not the user that's running the file
Sticky Bit s = setuid - Whenever anybody calls the passwd program it will be executed as the user who owns the file, so even though my user is user it will execute it as though it's root. Very big security issue if you try to set it on a script. Setuid on Bash scripts has been disabled as it's such a major security hole
as:~/finance$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 59640 Jan 25 2018 /usr/bin/passwd
Setgid = Execute as permissions of group who owns the file, not as user executing the file
chmod u+s test
as:~/finance$ chmod u+s myfile
as@ubuntu:~/finance$ ls -l myfile
-r-S------ 1 as finance 15 Aug 3 14:10 myfile
In Octal Notation
chmod 4500 test - 4 when put in front of group of 3 is for user
chmod 2500 test - 2 = group
chmod 6500 test - Everyone
Set Sticky bit - Another sticky bit?
chmod +t test - I couldn't get this to stop allowing me to delete the file!?
Understanding umask (user mask)
If using say mkdir, the process that's performing the task is setting the initial default permissions on the end result.
There are multiple ways of setting umask. When setting umask for current logged in shell the setting isn't persistent. If you log out then in again, the umask settings will be lost and will revert to the persistent setting.
Current Shell default permissions
as@ubuntu:~$ umask
0022
0|0|2|2 - Therefore in order we have
0= Do not mask ??0= User - Do not mask any permissions2= Group - Maskwritepermission (in Octal write == 2)2= Other - Maskwritepermission
e.g.
as@ubuntu:~/Documents/tmp$ touch file1
as@ubuntu:~/Documents/tmp$ ls -l
total 0
-rw-r--r-- 1 as as 0 Aug 10 09:06 file1
-rw-r--r-- - touch doesn't by default set execute permissions on a file so even though it's not umasked, the permission isn't set, w is masked for group and w is masked for other
When files are created they start at 666, when directories are created they start at 777. This is because you need x permissions on a directory to be able to browse into it. 2(r) + 4(w) = 6
umask is the reverse of chmod, similar to subnet mask on IOS Cisco
as@ubuntu:~/Documents/tmp/dir1$ umask 266
as@ubuntu:~/Documents/tmp/dir1$ touch file3
as@ubuntu:~/Documents/tmp/dir1$ ls -l
total 0
-r-------- 1 as as 0 Aug 10 09:47 file3
umask 266
2= Mask write(2) permissions forusergroup6= Mask read(2) + write(4) ==6permissions for the owning group of the file6= Mask read + write for everyone else on the system
As it was a file that was created with touch, permissions started at 666
Persistent Setting of umask
Both these need updating in RHEL for persistent change
vim /etc/bashrc
vim /etc/profile
/etc/login.defs <- Think it's this one in ubuntu
root@ubuntu:/etc# vim login.defs
UMASK 022 # Changed from 022 to 026
root@ubuntu:/etc# grep -rwi umask login.defs | grep -vE '^#'
UMASK 026
as@ubuntu:~/Documents/tmp/dir1$ su root
root@ubuntu:/home/as/Documents/tmp/dir1# umask
0026
root@ubuntu:/home/as/Documents/tmp/dir1# ls -la
total 8
drwxrwx--x 2 as as 4096 Aug 10 09:47 .
drwxr-xr-x 3 as as 4096 Aug 10 09:46 ..
-r-------- 1 as as 0 Aug 10 09:47 file3
root@ubuntu:/home/as/Documents/tmp/dir1# touch file4
root@ubuntu:/home/as/Documents/tmp/dir1# ls -l
total 0
-r-------- 1 as as 0 Aug 10 09:47 file3
-rw-r----- 1 root root 0 Aug 10 10:17 file4
file4 is now created with all permissions for the other group masked out
n.b. Users UID are always > 199
0 -> 200 = System Users for Redhat Processes
201 -> 200 = Other Systemd processes
as@ubuntu:~$ echo $UID
1000
Locate, Read and Use System Documentation with man, info and /usr/share/doc
In Exam there will be an objective to download a 3rd party program and figure out how to use the program by finding documentation installed with the program.
Could be:
- man
- --help
- linux info pages
- /usr/doc/share
- Inside rpm package
Index everything in /usr/share/man directory
as@ubuntu:~$ mandb
0 man subdirectories contained newer manual pages.
0 manual pages were added.
0 stray cats were added.
0 old database entries were purged.
Search apropos the man pages for information you would like to read about:
as@ubuntu:~$ apropos passwd
chgpasswd (8) - update group passwords in batch mode
chpasswd (8) - update passwords in batch mode
fgetpwent_r (3) - get passwd file entry reentrantly
getpwent_r (3) - get passwd file entry reentrantly
gpasswd (1) - administer /etc/group and /etc/gshadow
grub-mkpasswd-pbkdf2 (1) - generate hashed password for GRUB
mkpasswd (1) - Overfeatured front end to crypt(3)
openssl-passwd (1ssl) - compute password hashes
pam_localuser (8) - require users to be listed in /etc/passwd
passwd (1) - change user password
passwd (1ssl) - compute password hashes
passwd (5) - the password file
passwd2des (3) - RFS password encryption
update-passwd (8) - safely update /etc/passwd, /etc/shadow and /etc/group
Navigate straight to the man page that you're interested in e.g. passwd (5) - the password file
as@ubuntu:~$ man 5 passwd
man is considered 'old' as it harks back to the unix days. Most GNU Linux distributions are migrating to the info utility
info searches the directory /usr/share/info - If there's no new documentation within this directory for the program you're interested in reading about info will fall-back to the man utility. The following can be seen at the bottom of the documentation if for instance you issue:
info passwd-----Info: (*manpages*)passwd
Commands to navigate info
? - Show help on RHEL H for Ubuntu
u - Takes you to top level of the info program. So if you entered the program @ info screen pressing u will take you to the very top level. Use t to get back to start page/node of just the pages about the screen command
n - Next node
p - Previous node
T - Top of node you entered the info utility with
pgup/pgdn - Navigate current page
info provides the ability to navigate around a text based GUI
Using Cursor keys you navigate on the page to a link then by pressing <\CR> you can navigate to that node/section
You can pass apropos= to search the info utility similar to searching the man pages e.g.
info --apropos=tee
/usr/share/doc might include templates for programs
locate passwd - Searches a cached db for all files on the system with passwd in the filename
updatedb will update the cache for locate - Best to run after press package installation, but will normally run via cron once a day
which passwd - Shows path for the program
whatis passwd - Searches man pages descriptions - Cut down version of apropos
whereis passwd - Locate source binary, and source files and manual files for a specific command
rpm -qd packagename - Query for documentation for the package if installed via rpm
Finding files with Locate and Find
locate - Searches all files from cache / updatedb - Update cache manually
find ~ -name file* - Find all files with name starting file
find /etc/ -user root - Find all files owned by user 'root'
find . -mtime -3 - Find in the current directory all files that have been modified in the last 3 days
find ~/ -daystart -type f -mtime 1 | grep -vE '\.' - List the regular files in your home directory that were modified yesterday
You can then use stat to show all properties of the file you've just found e.g.
as@ubuntu:~$ stat /home/as/MEGA/linuxNotes/Ubuntu-Server-Setup-Notes
File: /home/as/MEGA/linuxNotes/Ubuntu-Server-Setup-Notes
Size: 7976 Blocks: 16 IO Block: 4096 regular file
Device: 801h/2049d Inode: 3692817 Links: 1
Access: (0600/-rw-------) Uid: ( 1000/ as) Gid: ( 1000/ as)
Access: 2018-08-10 08:28:59.551456428 +0100
Modify: 2018-08-09 20:44:12.000000000 +0100
Change: 2018-08-10 08:28:59.551456428 +0100
Birth: -
Find all files owned by root:
as@ubuntu:~$ find ~ -uid 0
/home/as/Documents/tmp/dir1/file4
as@ubuntu:~$ find ~ -user root
/home/as/Documents/tmp/dir1/file4
root@ubuntu:~$ echo "Hello find command" >> /home/as/Documents/tmp/dir1/file4
as@ubuntu:~$ sudo find ~ -user root -type f -exec cat {} \;
Hello find command
sudo find ~ -user root -type f -exec cat {} \;
find ~- Find in the current users home directory-user root- All files owned by root-type f- Of type file-exec cat {}- Then run the cat command against the returned{}files\;- Terminate the whole command
Similar example but copying the returned files into the directory specified. Required sudo as the current logged in user as didn't have permission to read the file
as@ubuntu:~$ find ~ -user root -type f -exec cp {} /home/as/Documents \;
cp: cannot open '/home/as/Documents/tmp/dir1/file4' for reading: Permission denied
as@ubuntu:~$ sudo find ~ -user root -type f -exec cp {} /home/as/Documents \;
as@ubuntu:~$ ls -l Documents/file*
-rw-r----- 1 root root 19 Aug 10 11:47 Documents/file4
Boot, Reboot and Shutdown
Systemdhandles initialisation of services using unit configuration filesSystemdadvantage oversystemvis that services can be started in parallelSystemdhas replaced the concept of run levelsinit 0- Shutdown,init 6- Restart
systemctl reboot - Equivalent of init 6, shutdown -r now or simply reboot
shutdown -r 00:00 - Reboot at midnight
Targetreplacedrunlevels
Boot into Different Targets Manually
- A
Targetis a grouping of configuration files
systemctl list-units --type=target - List available targets
systemctl -t help - List all the types of unit files available. Will only really be concerned with service & target
/usr/systemd/system - RHEL Unit files - other systems can vary e.g. to find on ubuntu
as@ubuntu:/usr/lib/systemd/user$ systemctl list-unit-files | grep -i avahi
avahi-daemon.service enabled
as@ubuntu:/usr/lib/systemd/user$ locate avahi-daemon.service
/etc/systemd/system/multi-user.target.wants/avahi-daemon.service
/lib/systemd/system/avahi-daemon.service
/var/lib/lxcfs/cgroup/blkio/system.slice/avahi-daemon.service
<text omitted>
as@ubuntu:/usr/lib/systemd/user$ sudo systemctl status avahi-daemon.service
[sudo] password for as:
● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
Loaded: loaded (/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2018-08-09 09:07:37 BST; 1 day 3h ago
Main PID: 745 (avahi-daemon)
Status: "avahi-daemon 0.7 starting up."
ls -l /etc/systemd/system/multi-user.target.wants/avahi-daemon.service
lrwxrwxrwx 1 root root 40 Dec 4 2017 /etc/systemd/system/multi-user.target.wants/avahi-daemon.service -> /lib/systemd/system/avahi-daemon.service
systemctl list-dependencies multi-user.target - List multi-user dependencies. This lists all the services or other targets that will start once the system enters the multi-user.target
You can create dependencies on other targets
**
systemctl get-default - Show the system default target e.g.
Ubuntu Laptop
as@ubuntu:/usr/lib/systemd/user$ systemctl get-default
graphical.target
Headless Raspberry Pi
pi@rp-01:~ $ systemctl get-default
multi-user.target
multi-user.target - Generally cli
graphical.target - GUI with the dependency of multi-user.target as shown here:
as@ubuntu:/usr/lib/systemd/user$ systemctl list-dependencies graphical.target | grep multi-user.target● └─multi-user.target
emergency.target - Boots system into CLI single user environment as root user and with read-only filesystem
Move Between Targets
systemctl isolate multi-user.target - This would close the GUI if open and return you to a shell
n.b. Enter ctrl+alt+F2 if you just see a blinking cursor and no login prompt
systemctl isolate graphical-user.target - This would load all the services required to operate a GUI and place you at the console
systemctl set-default multi-user.target - Set the new default target
List Targets:
ls -l /lib/systemd/system | grep -E *.target$
To set boot target e.g. to rescue an installation
- Interrupt the boot sequence at GRUB
- Navigate to the linux16 line
- Append to the end of the line
systemd.unit=multi-user.target - or `systemd.unit=rescue.target
Interrupt Boot Process to Reset Root Password
- In Grub - append to end of linux16 line
rd.break
mount -oremount,rw /sysrootchroot /sysrootpasswd roottouch /.autorelabel
Identify CPU / Memory Intensive Processes, Adjust Process Priority and Kill Processes
- pgrep / pkill (Installed from procps)
pgrep httpd -l - List process ID's and owning process name
pgrep -u as -l vim - List process ID's owned by user as
pgrep -v -u root -l - Long list processes not owned by root
pkill httpd - Greps for all processes identified by httpd and kills them
kill -l - List all kill signals e.g. 15) SIGTERM
kill -15 - Nice kill signal - "Please terminate, but terminate cleanly" - Generally the best way to kill a program
kill -9 - SIGKILL - Immediately terminate a process e.g. a hung or malicious process - Emergency situations
kill -SIGHUP - Similar to closing a terminal window via GUI x - Hangup the session to our terminal
kill -SIGINT - Keyboard Interrupt (^C)
kill -SIGQUIT - Request process to quit
kill -SIGTERM - Default when executing kill - Asks program to run all cleanup processes and then terminate
kill -SIGCONT - Stop a process and start it again using kill signals
kill -SIGSTOP - Stop process and make it that we can start it again. This signal cannot be ignored
kill -SIGTSTP - Sends a stop that can be ignored
Cleanly Boot a user from the system
-
Create a user and ensure sshd is running
sudo useradd testuser
sudo systemctl enable sshd.service
sudo systemctl start sshd.service -
ssh into the box:
ssh testuser@localhost -
In the as/root terminal run
w
[as@localhost ~]$ w
10:36:20 up 1:00, 2 users, load average: 0.00, 0.00, 0.04
USER TTY LOGIN@ IDLE JCPU PCPU WHAT
as tty2 09:37 59:53 8:55 5:23 /usr/lib64/firefox/firefox-contentproc -childID 3 -isFo
testuser pts/2 10:36 16.00s 0.02s 0.02s -bash
- Launch
viin testuser terminal and re-runwin as/root terminal. Note the process vim is now showing as running
[as@localhost ~]$ w
10:41:35 up 1:05, 2 users, load average: 0.03, 0.12, 0.09
USER TTY LOGIN@ IDLE JCPU PCPU WHAT
as tty2 09:37 1:05m 10:00 6:08 /usr/lib64/firefox/firefox -contentproc -childID 3 -isFo
testuser pts/2 10:36 7.00s 0.04s 0.01s vim
TTY ID is the terminal window ID
- Kill a users process that's running in the TTY
[as@localhost ~]$ w
10:52:15 up 1:15, 2 users, load average: 0.83, 0.57, 0.33
USER TTY LOGIN@ IDLE JCPU PCPU WHAT
as tty2 09:37 1:15m 13:27 0.43s /usr/lib64/firefox/firefox -contentproc -childID 7 -isFo
testuser pts/2 10:36 10:47 0.04s 0.04s -bash
Vim: Caught deadly signal TERM
Vim: Finished.
- Boot the user from the system
[as@localhost ~]$ sudo pkill -u testuser sshd
Terminated
[testuser@localhost ~]$ Connection to localhost closed by remote host.
Connection to localhost closed.
Create a process, send it to the background, kill -SIGSTOP on the process and then start the process again
(while true; do echo -n "My Program" >> ~/output.file; sleep 1; done) &
& - This sends whatever you've called into the background
[as@localhost ~]$ (while true; do echo -n "My Program" >> ~/output.file; sleep 1; done) &
[1] 30667
[as@localhost ~]$ jobs
[1]+ Running ( while true; do
echo -n "My Program" >> ~/output.file; sleep 1;
done ) &
[as@localhost ~]$ kill -SIGSTOP %1
[1]+ Stopped ( while true; do
echo -n "My Program" >> ~/output.file; sleep 1;
done )
[as@localhost ~]$ jobs
[1]+ Stopped ( while true; do
echo -n "My Program" >> ~/output.file; sleep 1;
done )
[as@localhost ~]$ kill -SIGCONT %1
[as@localhost ~]$ jobs
[1]+ Running ( while true; do
echo -n "My Program" >> ~/output.file; sleep 1;
done ) &
[as@localhost ~]$ kill %1
[1]+ Terminated ( while true; do
echo -n "My Program" >> ~/output.file; sleep 1;
done )
ps
pkill httpd - Kill all httpd processes
Use ps to view nice level - nice level is the niceness of a process, which means the priority of the process. With a lower nice level a process is allocated more priority on CPU time/utilisation than processes with higher nice levels
ps axo pid,comm,nice - -o Allows you to output particular headers
w | who - Similar commands, but w provides what processes the user is running
[root@localhost ~]# ps -u testuser
PID TTY TIME CMD
32420 ? 00:00:00 systemd
32428 ? 00:00:00 (sd-pam)
32437 ? 00:00:00 pulseaudio
32438 ? 00:00:00 sshd
32444 pts/2 00:00:00 bash
32507 ? 00:00:00 dbus-daemon
Nice, Renice & niceness
Range:: -20 ... +19
-20 is most favourable (when competing for resources)
+19 is least favourable cpu priority
- Using
psoutput the niceness of a process
[root@localhost ~]# ps axo pid,comm,nice | grep httpd
32601 httpd 0
32602 httpd 0
32604 httpd 0
32605 httpd 0
32607 httpd 0
- Stop the process and re-launch with a higher process priority
[root@localhost ~]# systemctl stop httpd
[root@localhost ~]# nice -n 10 httpd
[root@localhost ~]# ps axo pid,comm,nice | grep httpd
32830 httpd 10
32831 httpd 10
32833 httpd 10
32835 httpd 10
32836 httpd 10
- Change
renicethe priority of an individual proccess
[root@localhost ~]# renice -n 0 32830
32830 (process ID) old priority 10, new priority 0
[root@localhost ~]# ps axo pid,comm,nice | grep httpd
32830 httpd 0
32831 httpd 10
32833 httpd 10
32835 httpd 10
32836 httpd 10
renicethe priority of all processes
[root@localhost ~]# renice -n 0 $(pgrep httpd)
32830 (process ID) old priority 0, new priority 0
32831 (process ID) old priority 10, new priority 0
32833 (process ID) old priority 10, new priority 0
32835 (process ID) old priority 10, new priority 0
32836 (process ID) old priority 10, new priority 0
Demonstrate a process allocated more CPU time
- Create a 5GB file
[root@localhost ~]# dd if=/dev/zero of=/root/test.file bs=1M count=5120
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 4.29275 s, 1.3 GB/s
[root@localhost ~]# ls -lh
total 5.1G
-rw-------. 1 root root 946 Aug 10 17:56 anaconda-ks.cfg
-rw-r--r--. 1 root root 5.0G Aug 17 12:21 test.file
timehow long thetarwith compression-zprogram takes to run when given the lowest CPU priority
[root@localhost ~]# time nice -n 19 tar -czvf test19.tar.gz test.file
test.file
real 0m57.321s
user 0m21.324s
sys 0m38.580s
- Compare against the
timetaken to runtarwith the highest CPU priority
[root@localhost ~]# time nice -n -20 tar -czvf test-20.tar.gz test.file
test.file
real 0m51.369s
user 0m16.974s
sys 0m36.933s
System Load Averages
w
uptime
top
[root@localhost ~]# uptime
12:34:04 up 2:57, 1 user, load average: 0.17, 0.47, 0.47
load average: - Represents the percentage of CPU being used. If we had a system with 1 processor, if any of the following numbers were 1.00 we'd be using 100% of CPU time
0.17, - Load average for the last 60 seconds
0.47, - Load average for past 5 mins
0.47 - Load average for the last 15 mins
How many CPU's does the system we're working on have?
[root@localhost ~]# cat /proc/cpuinfo | grep -E 'processor|core id'
processor : 0
core id : 0
processor : 1
core id : 0
processor : 2
core id : 0
processor : 3
core id : 0
[root@localhost ~]# cat /proc/cpuinfo | grep 'processor' | wc -l
4
Current Process usage is 60sec load average / number of processors e.g.
(0.17 / 4) * 100 = 8.5% of Total CPU
Another Example
[as@localhost ~]$ $(while true; do echo -n "hello world" > ~/file.output; done;) &
[as@localhost ~]$ uptime
13:16:28 up 3:40, 1 user, load average: 3.39, 2.58, 1.48
(3.39 / 4) * 100 = 85%
[as@localhost ~]$ stress-ng --cpu 4 --cpu-load 66
stress-ng: info: [34472] defaulting to a 86400 second run per stressor
stress-ng: info: [34472] dispatching hogs: 4 cpu
[root@localhost ~]# uptime
13:35:55 up 3:59, 1 user, load average: 2.86, 2.66, 2.34
(2.86 / 4) * 4 = 71.5%
top - Display Linux Processes
l - Toggle Load Stats
m - Toggle Memory Stats
t- Toggle Tasks/CPU Stats
P - Sort by %CPU
M - Order by %MEM - A tasks currently used share of available physical memory
TIME+ - Total CPU time, task has used since starting
RES - Non-swap physical memory a task is using
SHR - Shared memory size, amount of shared memory available to the task
r - renice a task priority
k - Kill a task (Signal 15)
top -n 2 - Update 2 times and exit top
top -d 1 - Refresh the screen every 1 second
Locate and Interpret System Log Files and Journals
/var/log/ - All logs generated via rsyslogd or syslogd
/etc/logrotate.conf - Modify how logs are stored and overwritten
In newer systems all event logs are captured via the service systemd-journald
man systemd-journald for information.
The tool to view what's captured is: journalctl
journalctl - By default journald logs are not persistent across reboots
/run/log/journal/ - The directory where the logs are written
n.b. The /run/ directory is ephemeral and upon reboot, the directory is cleared
To make the logs persistent, create a newline and set Storage=persistent. Once you've updated to persistent storage the log directory is updated to point to /var/log/journal/
vim /etc/systemd/journal.conf
[Journal]
#Storage=auto
Storage=persistent
To understand the journald.conf parameters you can search the man pages
[as@localhost ~]$ apropos journald
journald.conf (5) - Journal service configuration files
journald.conf.d (5) - Journal service configuration files
systemd-journald (8) - Journal service
systemd-journald-audit.socket (8) - Journal service
systemd-journald-dev-log.socket (8) - Journal service
systemd-journald.service (8) - Journal service
systemd-journald.socket (8) - Journal service
[as@localhost ~]$ man 5 journald.conf
journalctl -n - Last 10 lines outputted
journalctl -f - Follow the log - Same as tail -f /var/log/syslog
journalctl -nx - Show extended logging information if it's available. This is indicated by ... at the end of a logging line e.g.
journalctl -u - Filter logs by unit e.g httpd.service
journalctl _SYSTEM_UNIT=httpd.service - Same as above - systemctl -t help to list available units
journalctl -p info - Filter based on syslog priority - emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug
journalctl --since=yesterday" - All logs since yesterday
[as@localhost ~]$ systemctl status httpd
...
Aug 17 15:16:52 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
[as@localhost ~]$ journalctl -fxu httpd.service
...
Aug 17 15:16:52 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up.
In the above example you can see the extended messages as denoted by --
/etc/rsyslog.conf
Show system startup information
systemd-analyze
systemd-analyze blame - Shows all the unit files and how long they took to load at startup
Securely Transfer Files Between Systems
Copy File tuning-info.txt from remote system to local directory .
scp sl-nl-ub-vps-01:/root/tuning-info.txt .
Use sftp to copy files from remote directory
as@as-x1:~$ sftp sl-nl-ub-vps-01
Connected to sl-nl-ub-vps-01.
sftp> ls t*
target tuning-info.txt
sftp> get tuning-info.txt
Fetching /root/tuning-info.txt to tuning-info.txt
/root/tuning-info.txt 100% 31KB 377.5KB/s 00:00
sftp> exit
as@as-x1:~$ ls t*
tuning-info.txt
Or similar to SCP:
as@as-x1:~$ sftp sl-nl-ub-vps-01:tuning-info.txt .
Connected to sl-nl-ub-vps-01.
Fetching /root/tuning-info.txt to ./tuning-info.txt
/root/tuning-info.txt 100% 31KB 310.4KB/s 00:00
List, Create and Delete Partitions on MBR and GPT Disks
The default username is user and default password is 123456.
Once logged in with "user" type "su"
Enter password "123456"
Confirm password "123456"
Enter your own customer password
Root login is disabled by default so you will always need to login as a regular user first
MBR developed in 1982
MBR is 32 bit
MBR can only contain 4x Primary Paritions with a limit of 2TiB ~ 2048GB
GPT is 64bit
GTP Based Partitions run on a UEFI Device. These can be run on backwards compatible BIOS
128 Primary Partitions
8ZiB Max parition size
fdisk is the tool to manage MBR based paritions
Create 2x Partitions
[root@apsoul1 dev]# ls xvd*
xvda xvda1 xvda2 xvdf
[root@apsoul1 dev]# fdisk /dev/xvdf
Command (m for help): n
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-2097151, default 2048):
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): +500M
Command (m for help): t
Hex code (type L to list all codes): L
Hex code (type L to list all codes): 83
Command (m for help): w
[root@apsoul1 dev]# ls xvd*
xvda xvda1 xvda2 xvdf xvdf1
[root@apsoul1 dev]# fdisk /dev/xvdf
Command (m for help): n
Select (default p): p
Partition number (2-4, default 2):
First sector (1026048-2097151, default 1026048):
Last sector, +sectors or +size{K,M,G} (1026048-2097151, default 2097151): +500M
Command (m for help): w
[root@apsoul1 dev]# ls xvd*
xvda xvda1 xvda2 xvdf xvdf1 xvdf2
Create Useable Filesystem To Start Storing Data
Format the Partition with a filesystem. On Redhat the most common filesystem to use is: xfs
[root@apsoul1 dev]# mkfs -t xfs xvdf1
meta-data=xvdf1 isize=512 agcount=4, agsize=32000 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=128000, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=855, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@apsoul1 dev]# blkid
/dev/xvda2: UUID="668dbd02-c201-44bc-be76-f606fc9ab8db" TYPE="xfs" PARTUUID="9146b810-9a31-4c10-a206-01b0bbaca807"
/dev/xvda1: PARTUUID="c32f79de-9ad9-4d19-a8f2-f3a61628211b"
/dev/xvdf1: UUID="9f50c98c-fb16-4c47-88d2-807beb0797a1" TYPE="xfs"
xvdf2 is not available yet as we haven't formatted it with a block filesystem.
Mount & Unmount a Device
[root@apsoul1 mnt]# mount /dev/xvdf1 /mnt/mymount/
[root@apsoul1 mnt]# df -h | grep mymount
/dev/xvdf1 497M 26M 472M 6% /mnt/mymount
[root@apsoul1 mnt]# umount /mnt/mymount
Mount by UUID
This is considered best practice to avoid conflicts or mounting an incorrect device. It's also the best practice for adding mounts at startup through
/etc/fstab
[root@apsoul1 ~]# mount -U 9f50c98c-fb16-4c47-88d2-807beb0797a1 /mnt/mymount
Delete a Partition
n.b. After a create and destroy on any paritions it's good practice to run
partprobe
[root@apsoul1 ~]# ls /dev/xvdf*
/dev/xvdf /dev/xvdf1 /dev/xvdf2
Command (m for help): d
Partition number (1,2, default 2): 2
Command (m for help): w
[root@apsoul1 ~]# partprobe
[root@apsoul1 ~]# ls /dev/xvdf*
/dev/xvdf /dev/xvdf1
Create Mount, Unmount & Delete GPT Partitions
[root@apsoul2 user]# gdisk /dev/xvdf
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-2097118, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-2097118, default = 2097118) or {+-}size{KMGTP}: +500M
Hex code or GUID (L to show codes, Enter = 8300): 8300
Command (? for help): w
Do you want to proceed? (Y/N): Y
[root@apsoul2 user]# ls /dev/xvd*
/dev/xvda /dev/xvda1 /dev/xvda2 /dev/xvdf /dev/xvdf1
[root@apsoul2 user]# mkfs -t xfs /dev/xvdf1
[root@apsoul2 user]# mkdir /mnt/mymount
[root@apsoul2 user]# mount /dev/xvdf1 /mnt/mymount/
[root@apsoul2 user]# df -h | grep mymount
/dev/xvdf1 497M 26M 472M 6% /mnt/mymount
[root@apsoul2 user]# blkid
/dev/xvda2: UUID="668dbd02-c201-44bc-be76-f606fc9ab8db" TYPE="xfs" PARTUUID="9146b810-9a31-4c10-a206-01b0bbaca807"
/dev/xvda1: PARTUUID="c32f79de-9ad9-4d19-a8f2-f3a61628211b"
/dev/xvdf1: UUID="6375b5c8-04d2-47ea-bfa6-2630f4c7e0d9" TYPE="xfs" PARTLABEL="Linux filesystem" PARTUUID="f30ab0f4-b609-4216-a8dc-850c046661d1"
[root@apsoul2 user]# umount /mnt/mymount/
[root@apsoul2 user]# mount -U 6375b5c8-04d2-47ea-bfa6-2630f4c7e0d9 /mnt/mymount/
[root@apsoul2 user]# df -h | grep mymount
/dev/xvdf1 497M 26M 472M 6% /mnt/mymount
[root@apsoul2 user]# umount /mnt/mymount/
[root@apsoul2 user]# gdisk /dev/xvdf
Command (? for help): d
Command (? for help): w
Do you want to proceed? (Y/N): Y
[root@apsoul2 user]# ls /dev/xvd*
/dev/xvda /dev/xvda1 /dev/xvda2 /dev/xvdf
Create and Remove Physical Volumes, Assign Physical Volumes to Volume Groups and Create and Delete Logical Volumes
What is a physical volume in terms of LVM?
-
Each LVM has an underlying physical storage unit.
-
A physical volume can either be a partition or a device or the entire disk
-
In order to use a physical volume, the physical volume must be initialised as a physical volume for LVM
-
A label is placed on the first part of the volume to help identify and provide metadata about that physical volume to the LVM
- Label is placed in the second 512 byte sector on the physical volume
- You can either have 0,1 or 2 copies of this metadata stored on each physical volume
- By Default 1 copy is stored on the physical volume
- Once you configure this number of copies, you cannot change the number of copies available
- First copy is stored at the start of the device
- Second copy is stored at the end of the device
- This helps protects against accidental overwriting of that data.
-
LVM provides a great amount of flexibility for applications that require spanning disk space.
-
LVM works by creating a virtual layer of storage on top of physical devices.
-
The OS operates against the LVM disk as if it whas a regular disk, allowing us to expand the disk without having to reformat the filesystem or create new partitions.
-
What happens when your disk fills up?
- Use LVM to attach a new device. We can add it to the volume group and expand the virtual LVM disk.
There are other uses for LVM, such as the ability to take any combination of physical devices to create an abstract layer that allows your disk space to be combined together with LVM. We could attach 5 different disk devices to create a volume group and from there we could create disk space that pulls from multiple physical volumes. You can create 5-50, however many physical disk devices as you want. You can pull from that available physical disk storage as if it was a single disk device. That what the LVM does. It allows us to virtually create that disk space based on multiple physical volumes.
- Volume Groups are a combination of physical volumes that create a pool of space that the LVM can allocate
- VG's are made up of extents
- Extents are inside of a VG
- Smallest unit of space that can be assigned to a VG
- VG Extents are referred to as physical extents
- LV is allocated into set of logical extents that are the space size as the physical extents - They map to it
- LV Extents map to the physical extents
- Use entire device or partition it out
- Need to create a filesystem and label it
xfscan only be increased in size, not decreased. ext4 can be increaed and decrased in size
Create And Remove LVM Disks
ls -l /dev/xvd*
gdisk /dev/xvdf
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-41943006, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-41943006, default = 41943006) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 8e00
Command (? for help): w
Do you want to proceed? (Y/N): Y
ls -l /dev/xvd*
gdisk /dev/xvdg
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-41943006, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-41943006, default = 41943006) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 8e00
Command (? for help): w
Do you want to proceed? (Y/N): Y
ls -l /dev/xvd*
pvcreate /dev/xvdf1 /dev/xvdg1
pvdisplay
vgcreate battlestar /dev/xvdf1 /dev/xvdg1
vgdisplay
lvcreate -n galactica -L 10G battlestar
lvdisplay
lvcreate -n galactica1 -L 5G battlestar
vgdisplay
lvdisplay
ls /dev/battlestar/
mkfs -t xfs /dev/battlestar/galactica
mkdir /mnt/mymount
mount /dev/battlestar/galactica /mnt/mymount/
df -h
umount /mnt/mymount/
lvremove /dev/battlestar/galactica
lvdisplay
lvremove /dev/battlestar/galactica1
lvdisplay
vgdisplay
vgremove battlestar
pvdisplay
pvremove /dev/xvdg1 /dev/xvdf1
Extend LVS
Check Underlying Volume Group
[root@ospbastvm-001 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
vg00 1 9 0 wz--n- 98.80g <16.93g
vg_glance 1 1 0 wz--n- <250.00g 1020.00m
16.93GB Free
Extend Target Logical Volume
Check Logical Volume to Extend
[root@ospbastvm-001 ~]# df -h /home
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-lvol9 4.0G 4.0G 3.6M 100% /home
Extend Target LVM
[root@ospbastvm-001 ~]# lvextend -L +6G /dev/mapper/vg00-lvol9
Size of logical volume vg00/lvol9 changed from 4.00 GiB (1024 extents) to 10.00 GiB (2560 extents).
Logical volume vg00/lvol9 successfully resized.
Check Extended LVM
[root@ospbastvm-001 ~]# lvdisplay /dev/mapper/vg00-lvol9
--- Logical volume ---
LV Path /dev/vg00/lvol9
LV Name lvol9
VG Name vg00
LV UUID qLDl4G-blHD-0RL9-BEze-IF0p-3NBT-n4YAe3
LV Write Access read/write
LV Creation host, time localhost, 2019-03-13 15:20:43 +0000
LV Status available
# open 1
LV Size 10.00 GiB
Current LE 2560
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:7
Grow the Filesystem for the LVM
[root@ospbastvm-001 ~]# xfs_growfs /dev/mapper/vg00-lvol9
meta-data=/dev/mapper/vg00-lvol9 isize=512 agcount=4, agsize=262144 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=1048576, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 1048576 to 2621440
Check the grown filesystem is reporting correctly
[root@ospbastvm-001 ~]# df -hT /home
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg00-lvol9 xfs 10G 4.0G 6.1G 40% /home