Today we will practice some Unix commands which usually we use and are really helpful and required when your file size is more than 60K records, because those files you can not open at Excel.
- How you will find the row count of a file?
- How you will find total number of files inside any directory?
- How will you print first 2 lines of file?
- How to find number of records without displaying file name in output
- Suppose you have 100 records, out of which you want to read 10 to 20 records. You have a header in a file, how will you do that?
- I want distinct Department_Id from Employee File
- Check with Piyush what command he used to fire for record count without the header record.
- I want all Employees with department as 10 ?
- I've a flat file where 1st two bytes of each record is an identifier for that record. How do I find distinct count of these records?
- How will you compare 2 files
- Find out files starting with Emp in a particular directory
- How will you find particular name in a file ?
- How will you find files in a directory ?
- How will you search file in same / current directory?
- How will you change the file permissions of all files inside particular directory ?
- How will you search all files owned by a particular user
- How will you replace character / string in a particular file with another character / string ?
- How will you sort files by their size in a particular direcotry?
- How will you sort files by their timestamps
- To find out total size of a folder
- How will you delete particular content of a file
- How will you print last 2 lines of a file
- How to find folders in particular directory
Thanks to Nikhil Kulkarni for below Unix Part
- directory commands
- Process commands
- Sort files by date
- sort files by size
- grep
- find
- Creating Files
- Chmod command.
- File Operations
- Wc command
- Mv command.
- Rm command.
- Rmdir command
- Cut command
- Paste Command
- Cmp command.
- Sort command
- Head command
- Tail command
- Crontab command
1. How you will find the row count of a file?
Mandar@Mandar-PC /MyFiles
$ wc -l Employees.csv
108 Employees.csv
2. How you will find total number of files inside any directory?
Mandar@Mandar-PC /MyFiles
$ ls | wc -l
8
3. How will you print first 2 lines of file?
Mandar@Mandar-PC /MyFiles
$ head -2 Employees.csv
employee_id,first_name,last_name,email,phone_number,hire_date,job_id,salary,commission_pct,manager_id,department_id
100,Steven,King,SKING,515.123.4567,6/17/1987,AD_PRES,24000,null,null,90
4. How to find number of records without displaying file name in output
Mandar@Mandar-PC /MyFiles
$ awk '{n++} END {print n}' Employees.csv
108
Even this will work same way Mandar@Mandar-PC /MyFiles
$ wc -l < Employees.csv
108
5. Suppose you have 100 records, out of which you want to read 10 to 20 records. You have a header in a file, how will you do that?
Mandar@Mandar-PC /MyFiles
$ head -20 Employees.csv | tail -10
109,Daniel,Faviet,DFAVIET,515.124.4169,8/16/1994,FI_ACCOUNT,9000,null,108,100
110,John,Chen,JCHEN,515.124.4269,9/28/1997,FI_ACCOUNT,8200,null,108,100
111,Ismael,Sciarra,ISCIARRA,515.124.4369,9/30/1997,FI_ACCOUNT,7700,null,108,100
112,Jose Manuel,Urman,JMURMAN,515.124.4469,3/7/1998,FI_ACCOUNT,7800,null,108,100
113,Luis,Popp,LPOPP,515.124.4567,12/7/1999,FI_ACCOUNT,6900,null,108,100
114,Den,Raphaely,DRAPHEAL,515.127.4561,12/7/1994,PU_MAN,11000,null,100,30
115,Alexander,Khoo,AKHOO,515.127.4562,5/18/1995,PU_CLERK,3100,null,114,30
116,Shelli,Baida,SBAIDA,515.127.4563,12/24/1997,PU_CLERK,2900,null,114,30
117,Sigal,Tobias,STOBIAS,515.127.4564,7/24/1997,PU_CLERK,2800,null,114,30
118,Guy,Himuro,GHIMURO,515.127.4565,11/15/1998,PU_CLERK,2600,null,114,30
6. I want distinct Department_Id from Employee File
Mandar@Mandar-PC /MyFiles
$ cut -d"," -f11 Employees.csv | sort | uniq
10
100
110
20
30
40
50
60
70
80
90
department_id
null
Give man cut so that you will get all parameter informationuniq will output all lines exactly once:
uniq -d will output all lines that appear more than once, and it will print them once:
uniq -u will output all lines that appear exactly once, and it will print them once
7. Check with Piyush what command he used to fire for record count without the header record.
8. I want all Employees with department as 10 ?
Mandar@Mandar-PC /MyFiles/NewFolder
$ awk -F"," '{ if($11=10) print $0}' Employees1.csv > Dep10File.csv
9.I've a flat file where 1st two bytes of each record is an identifier for that record. How do I find distinct count of these records?
Flat file ex:
AB|123|hello moto|
AB|456|googly|
BC|sick day|booya|
ID|inter doing|hahahah
So I want to find distinct record count... it should give me
AB 2
BC 1
ID 1
Mandar@Mandar-PC /MyFiles
$ awk -F'|' '{++c[$1]}END{for (i in c)print i, c[i]}' SampleData.txt
AB 2
BC 1
ID 1
10. How will you compare 2 files
Mandar@Mandar-PC /MyFiles
$ comp mandar.txt myname.txt
Comparing mandar.txt and myname.txt...
Compare error at OFFSET 6
file1 = A
file2 = 20
Compare error at OFFSET 10
file1 = A
file2 = 20
Compare more files (Y/N) ? n
Mandar@Mandar-PC /MyFiles
$ diff mandar.txt myname.txt
1,3c1
< mandar
< raghunath
< gogate
---
> mandar raghunath gogate
Mandar@Mandar-PC /MyFiles
$ sdiff mandar.txt myname.txt
mandar | mandar raghunath gogate
raghunath <
gogate <
11. Find out files starting with Emp in a particular directory
Mandar@Mandar-PC /MyFiles
$ ls -l | grep Emp*
-rwxr-xr-x 1 Mandar None 8314 May 1 00:39 Employees.csv
12. How will you find particular name in a file ?
Mandar@Mandar-PC /MyFiles
$ grep Sundi* Employees.csv
166,Sundar,Ande,SANDE,011.44.1346.629268,3/24/2000,SA_REP,6400,0.1,147,80
173,Sundita,Kumar,SKUMAR,011.44.1343.329268,4/21/2000,SA_REP,6100,0.1,148,80
13. How will you find files in a directory ?
Mandar@Mandar-PC /MyFiles
$ find -name 'Emp*'
./Employees.csv
./Employees1.csv
./Employees10.csv
./Employees11.csv
./Employees12.csv
./Employees2.csv
./Employees3.csv
./Employees4.csv
./Employees5.csv
./Employees6.csv
./Employees7.csv
./Employees8.csv
./Employees9.csv
As files were inside MyFile folder itself, it gave the search result like this. but same command works for finding files in sub - direcotry also. Now I have moded all Empl* files to new directory called NewFolder which is inside MyFiles directory
Mandar@Mandar-PC /MyFiles
$ find -name 'Emp*'
./NewFolder/Employees.csv
./NewFolder/Employees1.csv
./NewFolder/Employees10.csv
./NewFolder/Employees11.csv
./NewFolder/Employees12.csv
./NewFolder/Employees2.csv
./NewFolder/Employees3.csv
./NewFolder/Employees4.csv
./NewFolder/Employees5.csv
./NewFolder/Employees6.csv
./NewFolder/Employees7.csv
./NewFolder/Employees8.csv
./NewFolder/Employees9.csv
This command searches even the directory name and not only the file name
Mandar@Mandar-PC /MyFiles
$ find . -name New*
./NewFolder
To find only the files and exclude directories Mandar@Mandar-PC /MyFiles
$ find . -type f -name New*
14. How will you search file in same / current directory?
To search in all directories Mandar@Mandar-PC /MyFiles
$ find / -type f -name Emp*
15. How will you change the file permissions of all files inside particular directory ?
Mandar@Mandar-PC /MyFiles
$ find ./NewFolder/ -exec chmod 777 {} \;
16. How will you search all files owned by a particular user
Mandar@Mandar-PC /MyFiles
$ find . -user mandar
17. How will you replace character / string in a particular file with another character / string ?
Mandar@Mandar-PC /MyFiles/NewFolder
$ less Employees1.csv | tr 'manager_id' 'MANAGER_ID' | less
Above command will give display character with MANAGER_ID at command line You can redirect this output to new file with below command
Mandar@Mandar-PC /MyFiles/NewFolder
$ less Employees1.csv | tr 'manager_id' 'MANAGER_ID' > MyNewFile.csv
18. How will you sort files by their size in a particular direcotry?
Mandar@Mandar-PC /MyFiles/NewFolder
$ du -s * | sort -n
0 Employees.csv
12 Employees1.csv
12 MyNewFile.csv
20 Employees2.csv
36 Employees3.csv
68 Employees4.csv
180 Employees5.csv
344 Employees6.csv
764 Employees7.csv
2292 Employees8.csv
9160 Employees9.csv
27476 Employees10.csv
137376 Employees11.csv
412128 Employees12.csv
Mandar@Mandar-PC /MyFiles/NewFolder
$ du -a | sort -n
0 ./Employees.csv
1 ./helloworld.txt
12 ./Employees1.csv
12 ./MyNewFile.csv
20 ./Employees2.csv
36 ./Employees3.csv
68 ./Employees4.csv
180 ./Employees5.csv
344 ./Employees6.csv
764 ./Employees7.csv
2292 ./Employees8.csv
9160 ./Employees9.csv
27476 ./Employees10.csv
137376 ./Employees11.csv
412128 ./Employees12.csv
589873 .
19. How will you sort files by their timestamps
Mandar@Mandar-PC /MyFiles/NewFolder
$ ls -ctl | sort -n
-rw-r--r-- 1 Mandar None 11 May 5 12:34 helloworld.txt
-rw-r--r-- 1 Mandar None 8314 May 5 12:20 MyNewFile.csv
-rwxrwxrwx 1 Mandar None 0 May 5 12:09 Employees.csv
-rwxrwxrwx 1 Mandar None 8314 May 5 12:03 Employees1.csv
-rwxrwxrwx 1 Mandar None 16628 May 5 12:03 Employees2.csv
-rwxrwxrwx 1 Mandar None 33256 May 5 12:03 Employees3.csv
-rwxrwxrwx 1 Mandar None 66512 May 5 12:03 Employees4.csv
-rwxrwxrwx 1 Mandar None 182908 May 5 12:03 Employees5.csv
-rwxrwxrwx 1 Mandar None 349188 May 5 12:03 Employees6.csv
-rwxrwxrwx 1 Mandar None 781516 May 5 12:03 Employees7.csv
-rwxrwxrwx 1 Mandar None 2344548 May 5 12:03 Employees8.csv
-rwxrwxrwx 1 Mandar None 9378192 May 5 12:03 Employees9.csv
-rwxrwxrwx 1 Mandar None 28134576 May 5 12:03 Employees10.csv
-rwxrwxrwx 1 Mandar None 140672880 May 5 12:03 Employees11.csv
-rwxrwxrwx 1 Mandar None 422018640 May 5 12:03 Employees12.csv
total 589869
20. To find out total size of a folder
Mandar@Mandar-PC /MyFiles/NewFolder
$ du -h
577M .
22. How will you print last 2 lines of a file
Mandar@Mandar-PC /MyFiles
$ tail -2 Second.sh
echo -e "\n\nFile $1, found and successfully echoed"
fi
23. How to find folders in particular directory
Mandar@Mandar-PC /MyFiles
$ find . -type d
.
./NewFolder
--------------------
mkdir directoreyname -> creates a new directory
cd directoryname -> Change directory
25.Process commands
-----------------
PS command
ps command is probably the most useful command for systems administrators. It reports information on active processes.
ps options
* options. -a Lists all processes in system except processes not attached to terminals.
* -e Lists all processes in system.
* -f Lists a full listing.
* -j print process group ID and session ID.
ps -e -- lists all proceses with information about process
kill pid -- kills the process
Mandar@Mandar-PC /MyFiles/NewFolder
$ ps -a
PID PPID PGID WINPID TTY UID STIME COMMAND
4104 1912 4104 7052 pty0 1000 11:17:00 /usr/bin/bash
1912 1 1912 1912 ? 1000 11:17:00 /usr/bin/mintty
4764 4104 4764 6688 pty0 1000 13:59:06 /usr/bin/bash
2496 4764 2496 2344 pty0 1000 19:55:13 /usr/bin/ps
Mandar@Mandar-PC /MyFiles/NewFolder
$ ps -e
PID PPID PGID WINPID TTY UID STIME COMMAND
4104 1912 4104 7052 pty0 1000 11:17:00 /usr/bin/bash
1912 1 1912 1912 ? 1000 11:17:00 /usr/bin/mintty
4764 4104 4764 6688 pty0 1000 13:59:06 /usr/bin/bash
5908 4764 5908 3040 pty0 1000 19:55:46 /usr/bin/ps
Mandar@Mandar-PC /MyFiles/NewFolder
$ ps -f
UID PID PPID TTY STIME COMMAND
Mandar 4104 1912 pty0 11:17:00 /usr/bin/bash
Mandar 1912 1 ? 11:17:00 /usr/bin/mintty
Mandar 4764 4104 pty0 13:59:06 /usr/bin/bash
Mandar 5496 4764 pty0 19:56:05 /usr/bin/ps
26.Sort files by date
-------------------
ls - list all files in a directory
-a displays all files
-l Displays the long format listing.
-t Displays newest files first. (based on timestamp)
-r Displays files in reverse order.
ls -ltr : displays files based on timestamp in ascending order.
Only files. Excluding directories
Mandar@Mandar-PC /MyFiles/NewFolder
$ ls -l | wc -l
35
Including default directories. i.e. 2 more
Mandar@Mandar-PC /MyFiles/NewFolder
$ ls -al | wc -l
37
r will sort files output depending upon timestamp
Mandar@Mandar-PC /MyFiles/NewFolder
$ ls -ltr
total 2017219
-rwxrwxrwx 1 Mandar None 8314 May 5 01:17 Employees1.csv
-rwxrwxrwx 1 Mandar None 16628 May 5 01:18 Employees2.csv
-rwxrwxrwx 1 Mandar None 33256 May 5 01:18 Employees3.csv
-rwxrwxrwx 1 Mandar None 66512 May 5 01:18 Employees4.csv
-rwxrwxrwx 1 Mandar None 182908 May 5 01:19 Employees5.csv
-rwxrwxrwx 1 Mandar None 349188 May 5 01:19 Employees6.csv
-rwxrwxrwx 1 Mandar None 781516 May 5 01:20 Employees7.csv
-rwxrwxrwx 1 Mandar None 2344548 May 5 01:20 Employees8.csv
-rwxrwxrwx 1 Mandar None 9378192 May 5 01:21 Employees9.csv
-rwxrwxrwx 1 Mandar None 28134576 May 5 01:21 Employees10.csv
-rwxrwxrwx 1 Mandar None 140672880 May 5 01:22 Employees11.csv
-rwxrwxrwx 1 Mandar None 422018640 May 5 01:22 Employees12.csv
-rw-r--r-- 1 Mandar None 8314 May 5 12:20 MyNewFile.csv
-rw-r--r-- 1 Mandar None 11 May 5 12:34 helloworld.txt
-rw-r--r-- 1 Mandar None 8185 May 5 17:30 Dep10File.csv
-rw-r--r-- 1 Mandar None 8197 May 13 20:43 1.csv
27.sort files by size
------------------------
ls -al | sort +4n
This command performs a numeric sort on the fifth column of the "ls -al" output.
This results in a file listing where the files are listed in ascending order, from smallest in size to largest in size.
28.grep
-----
grep [options] pattern [files]
-i Ignore case sensitivity.
-w Match whole word.
-n line number
-c count
commnad to find a word "Error" in file1 and print it to file2 : grep Error file1 >> File2.txt
ps -ef | grep sleep will display all the sleep processes running in the system as follows.
Mandar@Mandar-PC /MyFiles/NewFolder
$ cat > GrepTest.txt
Mandar R. Gogate
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep mandar
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep mandar GrepTest.txt
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep -i mandar GrepTest.txt
Mandar R. Gogate
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep -in mandar GrepTest.txt
1:Mandar R. Gogate
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep Mandar GrepTest.txt
Mandar R. Gogate
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep -w Man GrepTest.txt
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep Man GrepTest.txt
Mandar R. Gogate
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep -ci man GrepTest.txt
1
Mandar@Mandar-PC /MyFiles/NewFolder
$ grep -i man GrepTest.txt > Out.txt
29.find
----
find : finds a file in a directory
-mtime is the last modified time of a file
-atime is the last access time
1. To find files modified in the last 5 days:
find . -mtime -5
2. To find files modified before 5 days:
find . -mtime +5
->If you omit the '+', it has a different meaning. It means to find files modified exactly before 5 days.
File got modified in last day
Mandar@Mandar-PC /MyFiles/NewFolder
$ find -mtime -1
.
./GrepTest.txt
./Out.txt
./Way1.txt
./Way2.txt
./Way3.txt
Files got accessed in last day
Mandar@Mandar-PC /MyFiles/NewFolder
$ find . -atime -1
.
./GrepTest.txt
./Out.txt
./Way1.txt
./Way2.txt
./Way3.txt
File got accessed before 45 days
Mandar@Mandar-PC /MyFiles/NewFolder
$ find . -atime +45
./Employees.csv
30.Creating Files
===========================
CAT
TOUCH
CAT > Filename, CTRL+D
> Create
< View
>> appends
Touch: this command creates an empty file
31.Chmod command.
-------------
chmod command is used to change permissions on a file.
for example if I have a text file with calender in it called cal.txt.
initially when this file will be created the permissions for this file depends upon umask set in your profile files.
As you can see this file has 666 or -rw-rw-rw attributes.
ls -la cal.txt
-rw-rw-rw- 1 ssb dxidev 135 Dec 3 16:14 cal.txt
In this line above I have -rw-rw-rw- meaning respectively that owner can read and write file, member of the owner's group can read and write this file and anyone else connected to this system can read and write this file., next ssb is owner of this file dxidev is the group of this file, there are 135 bytes in this file, this file was created on December 3 at time16:14 and at the end there is name of this file. Learn to read these permissions in binary, like this for example Decimal 644 which is 110 100 100 in binary meand rw-r--r-- or user can read,write this file, group can read only, everyone else can read only. Similarly, if permissions are 755 or 111 101 101 that means rwxr-xr-x or user can read, write and execute, group can read and execute, everyone else can read and execute. All directories have d in front of permissions. So if you don't want anyone to see your files or to do anything with it use chmod command and make permissions so that only you can read and write to that file, i.e.
chmod 600 filename.
r -4
w -2
x -1
32.File Operations
--------------------------
33.Wc command
------------------
wc command counts the characters, words or lines in a file depending upon the option.
* Options wc -l filename will print total number of lines in a file.
* wc -w filename will print total number of words in a file.
* wc -c filename will print total number of characters in a file.
34.Mv command.
------------------
mv command is used to move a file from one directory to another directory or to rename a file.
* Some examples: mv oldfile newfile will rename oldfile to newfile.
* mv -i oldfile newfile for confirmation prompt.
* mv -f oldfile newfile will force the rename even if target file exists.
* mv * /usr/bajwa/ will move all the files in current directory to /usr/bajwa directory.
35.Rm command.
------------------
To delete files use rm command.
* Options: rm oldfile will delete file named oldfile.
* rm -f option will remove write-protected files without prompting.
* rm -r option will delete the entire directory as well as all the subdirectories, very dangerous command.
36.Rmdir command.
------------------
rmdir command will remove directory or directories if a directory is empty.
* Options: rm -r directory_name will remove all files even if directory is not empty.
* rmdir sandeep is how you use it to remove sandeep directory.
* rmdir -p will remove directories and any parent directories that are empty.
* rmdir -s will suppress standard error messages caused by -p.
37.Cut command.
------------------
cut command selects a list of columns or fields from one or more files.
Option -c is for columns and -f for fields. It is entered as
cut options [files]
for example if a file named testfile contains
this is firstline
this is secondline
this is thirdline
Examples:
cut -c1,4 testfile will print this to standard output (screen)
ts
ts
ts
It is printing columns 1 and 4 of this file which contains t and s (part of this).
* Options: -c list cut the column positions identified in list.
* -f list will cut the fields identified in list.
* -s could be used with -f to suppress lines without delimiters.
38.Paste Command.
------------------
paste command merge the lines of one or more files into vertical columns separated by a tab.
for example if a file named testfile contains
this is firstline
and a file named testfile2 contains
this is testfile2
then running this command
paste testfile testfile2 > outputfile
will put this into outputfile
this is firstline this is testfile2
it contains contents of both files in columns.
who | paste - - will list users in two columns.
* Options: -d'char' separate columns with char instead of a tab.
* -s merge subsequent lines from one file.
Mandar@Mandar-PC /MyFiles/NewFolder
$ paste GrepTest.txt Way2.txt > PasteTest.txt
Mandar@Mandar-PC /MyFiles/NewFolder
$ cat PasteTest.txt
Mandar R. Gogate Hello World
39.Cmp command.
-----------------
cmp command compares the two files.
40.Sort command.
------------------
sort command sort the lines of a file or files, in alphabetical order. for example if you have a file named testfile with these contents
zzz
aaa
1234
yuer
wer
qww
wwe
Then running
sort testfile
will give us output of
1234
aaa
qww
wer
wwe
yuer
zzz
* Options: -b ignores leading spaces and tabs.
* -c checks whether files are already sorted.
* -d ignores punctuation.
* -i ignores non-printing characters.
* -n sorts in arithmetic order.
* -ofile put output in a file.
* +m[-m] skips n fields before sorting, and sort upto field position m.
* -r reverse the order of sort.
* -u identical lines in input file apear only one time in output.
41.Head command.
------------------
head filename by default will display the first 10 lines of a file.
If you want first 50 lines you can use head -50 filename or for 37 lines head -37 filename and so forth.
42.Tail command.
------------------
tail filename by default will display the last 10 lines of a file.
If you want last 50 lines then you can use tail -50 filename.
43.Crontab command.
------------------
crontab command is used to schedule jobs. You must have permission to run this command by unix Administrator. Jobs are scheduled in five numbers, as follows.
Minutes 0-59
Hour 0-23
Day of month 1-31
month 1-12
Day of week 0-6 (0 is sunday)
so for example you want to schedule a job which runs from script named backup_jobs in /usr/local/bin directory on sunday (day 0) at 11.25 (22:25) on 15th of month. The entry in crontab file will be. * represents all values.
25 22 15 * 0 /usr/local/bin/backup_jobs
The * here tells system to run this each month.
Syntax is
crontab file So a create a file with the scheduled jobs as above and then type
crontab filename .This will scheduled the jobs
----------------------------------------------------------------------------
->Mail command is used to send mails.
