Why shell scripting?
Posted on Sep 2, 2008 01:05:24 AM
1) Shell script can take input from user or file and output them on screen
2) Whenever you find yourself doing same task again and again you should use scripting i.e. repetitive task automation
3) Time saving
4) Creating your own power tools/utilities
5) Customizing administrative task
6) Since scripts are well tested, chances of errors reduced while configuring services or system administration task such as adding new users
7) Practical examples where shell scripting actively used:
o Monitoring Linux system
o Data backup and creating snapshots
o Dumping Oracle or MySQL database for backup
o Creating email based alert system
o Find out process which eats up more system resources
o Find out available and free memory
o Find out all logged in users and what they are doing
o Find out if all necessary network services are running or not. For example if web server failed send an alert to system administrator via a pager or an email
o Find out all failed login attempt, if login attempt are continue repeatedly from same network IP automatically block all those IPs accessing our network/service via firewall
o Find out process which eats up more system resources
o User administration as per your own security policies
o Find out information of local or remote servers
o Configure server such as BIND (DNS server) to add zone entries
Shell scripting is fun, I enjoy it because not every handy little functionality available in built into command or program library. It is useful to create nice (perhaps ugly) things in shell scripting. Here are few script example I use everyday:
* Find out todays weather (useful when you are busy in chat room)
* Find out what’s that site running (just like netcraft)
* Download RSS feed and display them as long as you login or email them
* Find out name of MP3 file you are playing (useful when you are busy in chat room)
* Monitor your domain expiry day
Steps to write Shell Script?
Posted on Sep 2, 2008 12:46:05 AM
By following the below steps you can write shell script:-
1) You can use any editor like vi or mcedit to write shell script.
2) Once you have written shell script set execute permission for your script which are as follows
syntax:
chmod permission your-script-name
Examples:
$ chmod +x your-script-name
$ chmod 755 your-script-name
Note: This will set read write execute(7) permission for owner, for group and other permission is read and execute only(5).
3) Now execute your script as
syntax:
bash your-script-name
sh your-script-name
./your-script-name
Examples:
$ bash bar
$ sh bar
$ ./bar
NOTE In the last syntax ./ means current directory, But only . (dot) means execute given command file in current shell without starting the new copy of shell, The syntax for . (dot) command is as follows
Syntax:
. command-name
Example:
$ . foo
Now you are ready to write first shell script that will print “Knowledge is Power” on screen. See the common vi command list , if you are new to vi.
$ vi first
#
# My first shell script
#
clear
echo “Knowledge is Power”
After saving the above script, you can run the script as follows:
$ ./first
This will not run script since we have not set execute permission for our script first; to do this type command
$ chmod 755 first
$ ./first
First screen will be clear, then Knowledge is Power is printed on screen.
Linux echo Command
Posted on Sep 2, 2008 12:14:53 AM
Use echo command to display text or value of variable.
echo [options] [string, variables…]
Displays text or variables value on screen.
Options
-n Do not output the trailing new line.
-e Enable interpretation of the following backslash escaped characters in the strings:
\a alert (bell)
\b backspace
\c suppress trailing new line
\n new line
\r carriage return
\t horizontal tab
\\ backslash
For e.g. $ echo -e “An apple a day keeps away \a\t\tdoctor\n”
Arithmetic in Shell Script
Posted on Sep 2, 2008 12:13:56 AM
It use to perform arithmetic operations.
Syntax:
expr op1 math-operator op2
Examples:
$ expr 1 + 3
$ expr 2 - 1
$ expr 10 / 2
$ expr 20 % 3
$ expr 10 \* 3
$ echo `expr 6 + 3`
Note:
expr 20 %3 - Remainder read as 20 mod 3 and remainder is 2.
expr 10 \* 3 - Multiplication use \* and not * since its wild card.
For the last statement not the following points
(1) First, before expr keyword we used ` (back quote) sign not the (single quote i.e. ‘) sign. Back quote is generally found on the key under tilde (~) on PC keyboard OR to the above of TAB key.
(2) Second, expr is also end with ` i.e. back quote.
(3) Here expr 6 + 3 is evaluated to 9, then echo command prints 9 as sum
(4) Here if you use double quote or single quote, it will NOT work
For e.g.
$ echo “expr 6 + 3″ # It will print expr 6 + 3
$ echo ‘expr 6 + 3′ # It will print expr 6 + 3
Linux::Define Processes
Posted on Sep 2, 2008 12:13:16 AM
Process is kind of program or task carried out by your PC. For e.g.
$ ls -lR
ls command or a request to list files in a directory and all subdirectory in your current directory - It is a process.
Process defined as:
“A process is program (command given by user) to perform specific Job. In Linux when you start process, it gives a number to process (called PID or process-id), PID starts from 0 to 65535.”
Why Process are required in Linux
Posted on Sep 2, 2008 12:10:50 AM
As You know Linux is multi-user, multitasking operating system. You can run more than 2 process at the same time. For example if you want to find the number of files you have on your system you may give command like:
$ ls / -R | wc -l
This command will take lot of time to search all files on your system. So you can run such command in Background or simultaneously by giving command like
$ ls / -R | wc -l &
The ampersand (&) at the end of command tells shells start process (ls / -R | wc -l) and run it in background takes next command immediately.
Process & PID defined as:
“An instance of running command is called process and the number printed by shell is called process-id (PID), this PID can be use to refer specific running
How to de-bug the shell script?
Posted on Sep 2, 2008 12:09:51 AM
While programming shell sometimes you need to find the errors (bugs) in shell script and correct the errors (remove errors - debug). For this purpose you can use -v and -x option with sh or bash command to debug the shell script. General syntax is as follows:
Syntax:
sh option { shell-script-name }
OR
bash option { shell-script-name }
Option can be
-v Print shell input lines as they are read.
-x After expanding each simple-command, bash displays the expanded value of PS4 system variable, followed by the command and its expanded arguments.
Example:
$ cat > dsh1.sh
#
# Script to show debug of shell
#
tot=`expr $1 + $2`
echo $tot
Press ctrl + d to save, and run it as
$ chmod 755 dsh1.sh
$ ./dsh1.sh 4 5
9
$ sh -x dsh1.sh 4 5
#
# Script to show debug of shell
#
tot=`expr $1 + $2`
expr $1 + $2
++ expr 4 + 5
+ tot=9
echo $tot
+ echo 9
9
See the above output, -x shows the exact values of variables (or statements are shown on screen with values).
$ sh -v dsh1.sh 4 5
Use -v option to debug complex shell script.
Loops in Shell Scripts
Posted on Sep 2, 2008 12:09:07 AM
Loop defined as:
“Computer can repeat particular instruction again and again, until particular condition satisfies. A group of instruction that is executed repeatedly is called a loop.”
Bash supports:
* for loop
* while loop
Note that in each and every loop,
(a) First, the variable used in loop condition must be initialized, then execution of the loop begins.
(b) A test (condition) is made at the beginning of each iteration.
(c) The body of loop ends with a statement that modifies the value of the test (condition) variable.
AWK::Arithmetic
Posted on Sep 2, 2008 12:07:58 AM
You can easily, do the arithmetic with awk as follows:-
$ cat > math
{
print $1 ” + ” $2 ” = ” $1 + $2
print $1 ” - ” $2 ” = ” $1 - $2
print $1 ” / ” $2 ” = ” $1 / $2
print $1 ” x ” $2 ” = ” $1 * $2
print $1 ” mod ” $2 ” = ” $1 % $2
}
Run the awk program as follows:
$ awk -f math
20 3
20 + 3 = 23
20 - 3 = 17
20 / 3 = 6.66667
20 x 3 = 60
20 mod 3 = 2
(Press CTRL + D to terminate)
Note:In above program print $1 ” + ” $2 ” = ” $1 + $2, statement is used for addition purpose. Here $1 + $2, means add (+) first field with second field. Same way you can do - (subtraction ), * (Multiplication), / (Division), % (modular use to find remainder of division operation).
Clustered Server
Posted on Jul 28, 2008 06:43:49 PM
Why do we need clustering?
The use of the vast information available online cannot be maximized unless an effective means of organizing it can be provided. Clustering engines put search results together based on textual and linguistic similarity. This basic similarity is supported by heuristics which are coded by programmers using as basis the users’ preference on what they want to see on clustered documents. Clusters are presented using the style of folders and sub-folders.
When a search engine provides millions of results for a particular query, the searcher can either sift through the endless pages of results or depend on the search engine’s judgment as to the most relevant results. Neither can ensure that the targeted information can be accessed as it may remain buried under pages of results or it may not meet the search engine’s criteria. In the same way that all other things are clustered or organized, the world of web searching would be more useful once given the benefit of organized search results.
Clustering engines automatically cluster results into categories that have been intelligently selected from words and phrases contained in search results. Categories are intended to reach human-level accuracy and to offer hierarchical drill doom capability in a familiar folder-style interface. Mind-numbing lists need not be scrolled through or ignored as the main themes are viewed in the first 300 – 500 results right on the first page. A quick overview of the types of information available on a particular topic is made available so that the area of interest can be immediately put into focus.
With the great improvement of search engines’ capability to return a large number of relevant results, it became more difficult to navigate meaningfully through all the results. A typical searcher does not take the time to view results beyond the first page which makes it very probable to miss results that would have been relevant and useful to his/her search or query. Clusters make it possible for results found on the tenth page to be just a click away. Related items can also be viewed together without much effort. It even reveals unexpected relationships between words, ideas and concepts.
A good cluster is considered such if it possesses a readable description. It should be able to assist in narrowing down a search to find exact results. A clustering engine queries multiple search engines and combines the results to be clustered and displayed on one screen. Each result list comes with information regarding the total number of results clustered and retrieved. The clustering engine’s own heuristics shall determine the pages to be favored. Search engines sometimes return multiple copies of the same page with slightly different URLs but this is minimized in search result clustering. This is because clustering engines does not reproduce results with similar descriptions. Clusters are specific enough that repeated documents are very rare. Some are able to offer advanced search features which allows searchers to specify which sources should be searched, the number of results desired, allowable waiting time, the desired language to be used and the filtering out of offensive contents.
Search Engines that Clusters
Google Sets do not provide results but rather helps in finding similar terms to the ones entered. This allows the user to create more complex queries in one area and brainstorm on how to put a search together. Google Sets is Google Labs’ clustering agent.
Wisenut is a full-text search engine which provides for related topics aside from a number of results for any search item entered. This is called the WiseGuide. Some results would have subtopics which will show underneath the clustered results. A link can be found next to each of the clustered results whose keywords can be used to run another search. A different set of clustered results shall be produced in addition to the web page results. This search engine has been bought by LookSmart.
Teoma has been dubbed as the “Google Killer” due to its very interesting clustering technology. A single search run will produce four sets of results. Those found at the top left are sponsored results, those found at the bottom are website non-sponsored results, those at the top right are the suggestions for refining the result and those at the bottom right are link calculations from experts and enthusiasts. The link collections are suitable for general information needs while the suggestions are for more specific searches. A click on any would signal the search to run again where a different set of site results shall be provided. Teoma has been purchased by AskJeeves.
Infonetware.com is more of a demonstration of Infonetware’s Real Term Technology than a search engine. The results page is framed where the area on the left provides topics related to the search term while the web page search results are found on the right frame. It works with full searching.
Oingo uses the open Directory Project as its search source. The search results page gives a drop-down list of potential meanings. The list of categories in order of relevance to the search can be found beneath it as well as the site results from the directory itself. It is more useful for general term searches or search terms that are in a broad category.
Vivisimo is a meta-search engine that clusters its results. It provides a very simple front page with search results that are organized in groups. The page design makes it easy to explore several categories without having to “lose your place”. Clusty is the consumer search destination powered and owned by Vivisimo. It queries results from Ask, MSN, Open Directory, LookSmart, Gigablast and WiseNut. These sites were chosen because of their accurate results and quick return speeds.
Query Server offers several types of search on the left side of the front page. Each search has more or less the same interface and all cluster results. Search results are presented in a frame at the right side of the site.
Surfwax offers both subscription based and free services. A focus link can be seen in the upper left corner after a search is entered. These focus words can be used in addition to the search term. They are divided into narrower or broader categories and contain generic words and not links to specific people or places.
Northern Light News search requires a search to have a certain number of results in order to be clustered into folders. However, folder listing does not provide information about the contents of a particular folder although there are subfolders provided for broad topics. Search results are listed by order of date.
Clustering search engines break up several hundred results into manageable packages. Suggestions are provided so that the use of information is maximized and the search itself a lot easier. A search query cannot always be specific enough to target the right
information at once.