The tutorial below is designed to be completed in ~30 minutes by anyone with basic understanding of UNIX. To begin please download the three example files to your working directory: chr7.bed, chr11.bed and chroms.csv.
What is awk?
Awk is a data manipulation programming language that exists within UNIX, similar to grep or cut. It is useful for extracting text and quick processing of tabular data. Awk works by processing data one record at a time. Each line in a text file is a record and each record is composed of fields. So if you imagine a table full of data, each row is a record and each column within that row is a field.
Awk commands follow a basic structure of pattern {action}. The user gives the program a regular expression to be evaluated pattern. If the regular expression turns out to be true, then a command is executed {action}. If the user just provides the {action} with no pattern to be matched, then awk will run the command on all records. As a quick reminder, a regular expression is just a sequence of characters that define a pattern.
Important notes
Cisco Webex is the leading enterprise solution for video conferencing, webinars, and screen sharing. Web conferencing, online meeting, cloud calling and equipment. Mac OS X and macOS both come with a built-in compression system that can zip and unzip files. This integrated system is relatively basic, which is why many third-party apps are also available. A quick look at the Mac App Store reveals more than 50 apps for zipping and unzipping files.
A few important notes before we begin, most of the commands below are designed to print to screen so that it will be easy to visualize what is happening to our files as we move along, but you can also print all results to a file by adding this sign > and a new output file name at the end of each command pattern {action} > newfile.
Commands to be executed in the terminal will look like this:
Comments will be noted by this sign #. You do not need to copy these into your terminal, they are just here to give additional information on what the commands are doing.
We will be working with example files in .bed format downloaded and modified from the UCSC genome browser (https://genome.ucsc.edu/). We are using the .bed format because the data istab-delimited, which is exactly the type of data that awk excels at working with. For more on the .bed format see: https://genome.ucsc.edu/FAQ/FAQformat.html#format1. I also want to mention that the content and structure of this tutorial is largely based on the great chapter Unix Data Tools in Buffalo V (2015). Bioinformatics Data Skills. California: O'Reilly Media. You can learn more about the book here: http://shop.oreilly.com/product/0636920030157.do.
So let's get started!
To start, let's take a look at our first example file using cat:
This file is in .bed tabular format. There are columns for chromosome name, chromosome start position, chromosome end, position , score, strand, etc.. So, how can we replicate the cat function using awk?
The command above prints all the contents of a file to the screen. {print} is the action, but by using $0 we have omitted a pattern. Instead we are telling awk that we want all fields for each record to be printed. What if we just wanted to print the first field (column) in our file instead?
Now if we want to save the output of our awk commands to file instead of printing it to the screen we need to use > and provide the program with a new file name. We can view our new file using cat. Let us leave that file in our directory for now, but can you think of how we could erase it using the command line?
We can also use awk to select more than one field in our data file.
Wait, what happened here? Awk assumes we are working with tab-delimited input data, but it does not assume our output is in this format. We have to tell awk how to separate the data and we can do that by specifying it in our command.
We can even print out non-contiguous fields!
What if we wanted to combine the 1st field with the chromosome name, the 2nd field with the position and field 6 which tells us if the variant is on the positive or negative strand in one tab-delimited file? How would you do this with what you have learned so far?
That works! But if we had a much larger file structuring our command like this would involve a lot of typing. Awk has some built in variables that help modify the output so its not necessary to type in the tab-delimiter symbol t every time. There are many different types of these built in variables. We will use a type called field separators. Specifically, we will use the output field separator OFS which specifies what the output should be delimited by. To use the field separator we need to insert the -v flag in our command.
You can use the output field separator to specify any symbol. Like for example, if we wanted to separate our data fields with colons we would do this:
This capability is extremely useful for bioinformatics because sometimes it is necessary to transform data into different formats. Awk can help you do this quite easily.
As we have learned so far, awk assumes all your input data is tab-delimited. If we want to work with something else, like say a .csv file (comma separated values), we have to tell awk that the input field separator will be different. For that we can use the built in variable FS. Let's try it by importing our second example file: chroms.csv
And of course you can combine input and output separator statements.
There are many more flags you can use to modify input or output with awk. You can see them all if you type man awk into the command line. (Note: To exit the man page, type :q).
Another helpful built in variable is NR. NR stands for range of lines, or current record and it allows you to select specific lines, so for example lets take a look at our third example file: chr11.bed
This file has a header, now what if we want to exclude that header and just work with the data columns? We can use NR to do that. Note that in this case I do not need the -v flag
Note here a few new things, mainly the use of this symbol: !=. This is a negation statement that means NOT TRUE. In this case we are using it to tell awk to print all fields except those in the first record (or line) which is NR=1. The above command is a good demonstration of how you can use awk with conditional statements. Another example of that is using NR to extract ranges of lines.
Here we are telling awk to select any record (lines) equal or above record 3 (>=) AND (represented by &&) lower or equal to record 5. This is almost like an if condition. In fact, you can use if statements and loops with awk, but I will let you guys explore that on your own later.
Another thing you can do with awk is perform arithmetic operations. This could be useful for example if you aretrying to filter data. Say that for our chr7.bed file we only want to select genes for analysis that are longer than 1200 base pairs (we are going to go back to chr7.bed file because there is no header in that file, although you can probably imagine how we could do all of the subsequent operations with the chr11.bed file by excluding the header using NR). Let's remind ourselves what this file looks like using cat:
So as a reminder, the 2nd field in this file provides the chromosome start position of our gene of interest. The 3rd field provides the end position. So if we wanted to see what is the length of each of the genes in chromosome 7 we can use this command to print a list of base pair lengths to the screen:
Ok, so there are a few genes that are longer than 1200 bp. Let's filter out the information for the other genes that we are not interested in by using a regular expression.
You can also chain statements, like say for example we only wanted genes on the positive DNA strand (field 4 provides this information) and that are over 1100 base pairs long.
So here we have done few things, the first is we use a pattern $4 ~/Pos*/. This pattern is a regular expression, its matching field 4 to the statement Pos and then using * which is a wildcard. In other words its saying, in field 4 ($4), any time you see the pattern 'Pos' (~/Pos), regardless of what follows (*), select that line. The pattern to be matched is in slashes. Then we have && for AND. So the command is saying, after you match this pattern (i.e. find elements on the positive strand) do something else, which in this case is perform the arithmetic.
We can even use awk to create new fields in a file based on previously inputed information. For example if we wanted to make a new file that only contains the chromosome name, positions and then the length of each gene we could do:
Can you explain what we did here?
This is just skimming the surface of all that awk can do, you can also use conditional statements, calculate morecomplex things like means, or apply user created variables in awk. But I think that with this short tutorial you willhave enough knowledge to be able to search for those commands and apply them.
##References and sources for more information#####Main source:Buffalo V. 2015. Bioinformatics Data Skills. California: O'Reilly Media.#####Secondary sources:
Malare Mawnama Malarin Mp3son Dawnlod; Melbourne Cup Challenge Download Full Version Free. Malare mawnama malarin mp3son dawnlod. Malare Mawnama Malarin Mp3son Dawnlod; Hp Compaq Nc8000 Base System Device Driver; Tweakbit Fix My Pc 2015 Serial Key; MagicDraw UML Enterprise V16.5 SP4 MacOSX-ZWT; Automation Studio 5 Library Download; Komik Eyeshield 21 Bahasa Indonesia 301.
- AWK Tutorial
- AWK Useful Resources
- Selected Reading
This chapter describes several useful AWK commands and their appropriate examples. Consider a text file marks.txt to be processed with the following content −
Printing Column or Field
You can instruct AWK to print only certain columns from the input field. The following example demonstrates this −
Example
On executing this code, you get the following result −
Output
In the file marks.txt, the third column contains the subject name and the fourth column contains the marks obtained in a particular subject. Let us print these two columns using AWK print command. In the above example, $3 and $4 represent the third and the fourth fields respectively from the input record.
Printing All Lines
By default, AWK prints all the lines that match pattern.
Example
On executing this code, you get the following result −
Output
In the above example, we are searching form pattern a. When a pattern match succeeds, it executes a command from the body block. In the absence of a body block − default action is taken which is print the record. Hence, the following command produces the same result −
Example
Printing Columns by Pattern
When a pattern match succeeds, AWK prints the entire record by default. But you can instruct AWK to print only certain fields. For instance, the following example prints the third and fourth field when a pattern match succeeds.
Example
On executing this code, you get the following result − I3 2120 specs.
Output
Printing Column in Any Order
You can print columns in any order. For instance, the following example prints the fourth column followed by the third column.
Example
On executing the above code, you get the following result −
Output
Counting and Printing Matched Pattern
Let us see an example where you can count and print the number of lines for which a pattern match succeeded.
Example
On executing this code, you get the following result −
Output
In this example, we increment the value of counter when a pattern match succeeds and we print this value in the END block. Note that unlike other programming languages, there is no need to declare a variable before using it.
Printing Lines with More than 18 Characters
Let us print only those lines that contain more than 18 characters.
Example
On executing this code, you get the following result −
Output
AWK provides a built-in length function that returns the length of the string. $0 variable stores the entire line and in the absence of a body block, default action is taken, i.e., the print action. Hence, if a line has more than 18 characters, then the comparison results true and the line gets printed.