Rectangle 27 200

The following will print the line matching TERMINATE till the end of the file:

sed -n -e '/TERMINATE/,$p'

Explained: -n disables default behavior of sed of printing each line after executing its script on it, -e indicated a script to sed, /TERMINATE/,$ is an address (line) range selection meaning the first line matching the TERMINATE regular expression (like grep) to the end of the file ($), and p is the print command which prints the current line.

This will print from the line that follows the line matching TERMINATE till the end of the file: (from AFTER the matching line to EOF, NOT including the matching line)

sed -e '1,/TERMINATE/d'

Explained: 1,/TERMINATE/ is an address (line) range selection meaning the first line for the input to the 1st line matching the TERMINATE regular expression, and d is the delete command which delete the current line and skip to the next line. As sed default behavior is to print the lines, it will print the lines after TERMINATE to the end of input.

If you want the lines before TERMINATE:

sed -e '/TERMINATE/,$d'

And if you want both lines before and after TERMINATE in 2 different files in a single pass:

sed -e '1,/TERMINATE/w before
/TERMINATE/,$w after' file

The before and after files will contain the line with terminate, so to process each you need to use:

head -n -1 before
tail -n +2 after

IF you do not want to hard-code the filenames in the sed script, you can:

before=before.txt
after=after.txt
sed -e "1,/TERMINATE/w $before
/TERMINATE/,\$w $after" file

But then you have to escape the $ meaning the last line so the shell will not try to expand the $w variable (note that we now use double quotes around the script instead of single quotes).

I forgot to tell that the new line is important after the filenames in the script so that sed knows that the filenames end.

Sbastien Clment asked: "How would you replace the hardcoded TERMINATE by a variable?"

You would make a variable for the matching text and then do it the same way as the previous example:

matchtext=TERMINATE
before=before.txt
after=after.txt
sed -e "1,/$matchtext/w $before
/$matchtext/,\$w $after" file

to use a variable for the matching text with the previous examples:

## Print the line containing the matching text, till the end of the file:
## (from the matching line to EOF, including the matching line)
matchtext=TERMINATE
sed -n -e "/$matchtext/,\$p"
## Print from the line that follows the line containing the 
## matching text, till the end of the file:
## (from AFTER the matching line to EOF, NOT including the matching line)
matchtext=TERMINATE
sed -e "1,/$matchtext/d"
## Print all the lines before the line containing the matching text:
## (from line-1 to BEFORE the matching line, NOT including the matching line)
matchtext=TERMINATE
sed -e "/$matchtext/,\$d"

The important points about replacing text with variables in these cases are:

  • Variables ($variablename) enclosed in single quotes ['] won't "expand" but variables inside double quotes ["] will. So, you have to change all the single quotes to double quotes if they contain text you want to replace with a variable.
  • The sed ranges also contain a $ and are immediately followed by a letter like: $p, $d, $w. They will also look like variables to be expanded, so you have to escape those $ characters with a backslash [\] like: \$p, \$d, \$w.

How can we get the lines before TERMINATE and delete all that follows ?

How would your replace the hardcoded TERMINAL by a variable?

One use case that's missing here is how to print lines after the last marker (if there can be multiple of them in the file .. think log files etc).

sed -e "1,/$matchtext/d"
$matchtext
sed -e "0,/$matchtext/d"

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 198

The following will print the line matching TERMINATE till the end of the file:

sed -n -e '/TERMINATE/,$p'

Explained: -n disables default behavior of sed of printing each line after executing its script on it, -e indicated a script to sed, /TERMINATE/,$ is an address (line) range selection meaning the first line matching the TERMINATE regular expression (like grep) to the end of the file ($), and p is the print command which prints the current line.

This will print from the line that follows the line matching TERMINATE till the end of the file: (from AFTER the matching line to EOF, NOT including the matching line)

sed -e '1,/TERMINATE/d'

Explained: 1,/TERMINATE/ is an address (line) range selection meaning the first line for the input to the 1st line matching the TERMINATE regular expression, and d is the delete command which delete the current line and skip to the next line. As sed default behavior is to print the lines, it will print the lines after TERMINATE to the end of input.

If you want the lines before TERMINATE:

sed -e '/TERMINATE/,$d'

And if you want both lines before and after TERMINATE in 2 different files in a single pass:

sed -e '1,/TERMINATE/w before
/TERMINATE/,$w after' file

The before and after files will contain the line with terminate, so to process each you need to use:

head -n -1 before
tail -n +2 after

IF you do not want to hard-code the filenames in the sed script, you can:

before=before.txt
after=after.txt
sed -e "1,/TERMINATE/w $before
/TERMINATE/,\$w $after" file

But then you have to escape the $ meaning the last line so the shell will not try to expand the $w variable (note that we now use double quotes around the script instead of single quotes).

I forgot to tell that the new line is important after the filenames in the script so that sed knows that the filenames end.

Sbastien Clment asked: "How would you replace the hardcoded TERMINATE by a variable?"

You would make a variable for the matching text and then do it the same way as the previous example:

matchtext=TERMINATE
before=before.txt
after=after.txt
sed -e "1,/$matchtext/w $before
/$matchtext/,\$w $after" file

to use a variable for the matching text with the previous examples:

## Print the line containing the matching text, till the end of the file:
## (from the matching line to EOF, including the matching line)
matchtext=TERMINATE
sed -n -e "/$matchtext/,\$p"
## Print from the line that follows the line containing the 
## matching text, till the end of the file:
## (from AFTER the matching line to EOF, NOT including the matching line)
matchtext=TERMINATE
sed -e "1,/$matchtext/d"
## Print all the lines before the line containing the matching text:
## (from line-1 to BEFORE the matching line, NOT including the matching line)
matchtext=TERMINATE
sed -e "/$matchtext/,\$d"

The important points about replacing text with variables in these cases are:

  • Variables ($variablename) enclosed in single quotes ['] won't "expand" but variables inside double quotes ["] will. So, you have to change all the single quotes to double quotes if they contain text you want to replace with a variable.
  • The sed ranges also contain a $ and are immediately followed by a letter like: $p, $d, $w. They will also look like variables to be expanded, so you have to escape those $ characters with a backslash [\] like: \$p, \$d, \$w.

How can we get the lines before TERMINATE and delete all that follows ?

How would your replace the hardcoded TERMINAL by a variable?

One use case that's missing here is how to print lines after the last marker (if there can be multiple of them in the file .. think log files etc).

sed -e "1,/$matchtext/d"
$matchtext
sed -e "0,/$matchtext/d"

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 43

As a simple approximation you could use

grep -A100000 TERMINATE file

which greps for TERMINATE and outputs up to 100000 lines following that line.

-A NUM, --after-context=NUM

Print NUM lines of trailing context after matching lines. Places a line containing a group separator (--) between contiguous groups of matches. With the -o or --only-matching option, this has no effect and a warning is given.

similarly -B NUM, --before-context=NUM Print NUM lines of leading context before matching lines. Places a line containing a group separator (--) between contiguous groups of matches. With the -o or --only-matching option, this has no effect and a warning is given.

this solution worked for me because i can easily use variables as my string to check for.

Nice idea! If you are uncertain about the size of the context you may count the lines of file instead: grep -A$(cat file | wc -l) TERMINATE file

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 43

As a simple approximation you could use

grep -A100000 TERMINATE file

which greps for TERMINATE and outputs up to 100000 lines following that line.

-A NUM, --after-context=NUM

Print NUM lines of trailing context after matching lines. Places a line containing a group separator (--) between contiguous groups of matches. With the -o or --only-matching option, this has no effect and a warning is given.

similarly -B NUM, --before-context=NUM Print NUM lines of leading context before matching lines. Places a line containing a group separator (--) between contiguous groups of matches. With the -o or --only-matching option, this has no effect and a warning is given.

this solution worked for me because i can easily use variables as my string to check for.

Nice idea! If you are uncertain about the size of the context you may count the lines of file instead: grep -A$(cat file | wc -l) TERMINATE file

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 22

  • if a match for 'TERMINATE' is found with the regular expression, we set it to one.

The other solutions might consume a lot of memory if you use them on very large files.

Thanks for a good alternative.

cat file | awk 'BEGIN{ found=0} /###/{found=found+1}  {if (found<2) print }'

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 22

  • if a match for 'TERMINATE' is found with the regular expression, we set it to one.

The other solutions might consume a lot of memory if you use them on very large files.

Thanks for a good alternative.

cat file | awk 'BEGIN{ found=0} /###/{found=found+1}  {if (found<2) print }'

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 8

Use bash parameter expansion like the following:

content=$(cat file)
echo "${content#*TERMINATE}"

Can you explain what are you doing ?

I copied the content of "file" into the $content variable. Then I removed all the characters until "TERMINATE" was seen. It didn't use greedy matching, but you can use greedy matching by ${content##*TERMINATE}.

Downvote: This is horrible (reading the file into a variable) and wrong (using the variable without quoting it; and you should properly use printf or make sure you know exactly what you are passing to echo.).

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 8

Use bash parameter expansion like the following:

content=$(cat file)
echo "${content#*TERMINATE}"

Can you explain what are you doing ?

I copied the content of "file" into the $content variable. Then I removed all the characters until "TERMINATE" was seen. It didn't use greedy matching, but you can use greedy matching by ${content##*TERMINATE}.

Downvote: This is horrible (reading the file into a variable) and wrong (using the variable without quoting it; and you should properly use printf or make sure you know exactly what you are passing to echo.).

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 6

If I understand your question correctly you do want the lines after TERMINATE, not including the TERMINATE-line. awk can do this in a simple way:

awk '{if(found) print} /TERMINATE/{found=1}' your_file
  • Although not best practice you could rely on the fact that all vars defaults to 0 or the empty string if not defined. So the first expression (if(found) print) will not print anything to start off with.
  • After the printing is done we check if the this is the starter-line (that should not be included).
TERMINATE
  • You have a file with start- and end-lines and you want the lines between those lines excluding the start- and end-lines.
  • start- and end-lines could be defined by a regular expression matching the line.
$ cat ex_file.txt 
not this line
second line
START
A good line to include
And this line
Yep
END
Nope more
...
never ever
$ awk '/END/{found=0} {if(found) print} /START/{found=1}' ex_file.txt 
A good line to include
And this line
Yep
$
  • If the end-line is found no printing should be done. Note that this check is done before the actual printing to exclude the end-line from the result.
  • Print the current line if found is set.
  • If the start-line is found then set found=1 so that the following lines are printed. Note that this check is done after the actual printing to exclude the start-line from the result.
  • The code rely on the fact that all awk-vars defaults to 0 or the empty string if not defined. This is valid but may not be best practice so you could add a BEGIN{found=0} to the start of the awk-expression.
  • If multiple start-end-blocks is found they are all printed.

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 6

If I understand your question correctly you do want the lines after TERMINATE, not including the TERMINATE-line. awk can do this in a simple way:

awk '{if(found) print} /TERMINATE/{found=1}' your_file
  • Although not best practice you could rely on the fact that all vars defaults to 0 or the empty string if not defined. So the first expression (if(found) print) will not print anything to start off with.
  • After the printing is done we check if the this is the starter-line (that should not be included).
TERMINATE
  • You have a file with start- and end-lines and you want the lines between those lines excluding the start- and end-lines.
  • start- and end-lines could be defined by a regular expression matching the line.
$ cat ex_file.txt 
not this line
second line
START
A good line to include
And this line
Yep
END
Nope more
...
never ever
$ awk '/END/{found=0} {if(found) print} /START/{found=1}' ex_file.txt 
A good line to include
And this line
Yep
$
  • If the end-line is found no printing should be done. Note that this check is done before the actual printing to exclude the end-line from the result.
  • Print the current line if found is set.
  • If the start-line is found then set found=1 so that the following lines are printed. Note that this check is done after the actual printing to exclude the start-line from the result.
  • The code rely on the fact that all awk-vars defaults to 0 or the empty string if not defined. This is valid but may not be best practice so you could add a BEGIN{found=0} to the start of the awk-expression.
  • If multiple start-end-blocks is found they are all printed.

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 4

If for any reason, you want to avoid using sed, the following will print the line matching TERMINATE till the end of the file:

tail -n "+$(grep -n 'TERMINATE' file | head -n 1 | cut -d ":" -f 1)" file

and the following will print from the following line matching TERMINATE till the end of the file:

tail -n "+$(($(grep -n 'TERMINATE' file | head -n 1 | cut -d ":" -f 1)+1))" file

It takes 2 processes to do what sed can do in one process, and if the file changes between the execution of grep and tail, the result can be incoherent, so I recommend using sed. Moreover, if the file dones not contain TERMINATE, the 1st command fails.

file is scanned twice. what if it is 100GB size?

Downvoted because this is a crappy solution, but then upvoted because 90% of the answer is caveats.

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 4

If for any reason, you want to avoid using sed, the following will print the line matching TERMINATE till the end of the file:

tail -n "+$(grep -n 'TERMINATE' file | head -n 1 | cut -d ":" -f 1)" file

and the following will print from the following line matching TERMINATE till the end of the file:

tail -n "+$(($(grep -n 'TERMINATE' file | head -n 1 | cut -d ":" -f 1)+1))" file

It takes 2 processes to do what sed can do in one process, and if the file changes between the execution of grep and tail, the result can be incoherent, so I recommend using sed. Moreover, if the file dones not contain TERMINATE, the 1st command fails.

file is scanned twice. what if it is 100GB size?

Downvoted because this is a crappy solution, but then upvoted because 90% of the answer is caveats.

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 1

First of all, there are a couple of problems in your grep regular expression:

  • The repetition count ({1,3}) applies to its preceding atom (i.e. '.') rather than the next one (i.e. '[0-9]').

An unescaped dot ('.') in the regex matches any character, which hides the previous error. Your regex (192.168.0.{1,3}[0-9]) matches 192.168.0.123 as follows:

192.168.0 matches 192.168.0
.{1,3}    matches .12
[0-9]     matches 3

But it would also match the following strings:

192116810abc1
192.681.012.9
192\.168\.0\.[0-9]{1,3}
grep -Eo '192\.168\.0\.[0-9]{1,3}'

Yet, the wrong regex can hardly explain the problem your are seeing with cron.

One problem may be that you are using a fixed name new.txt for your temporary file. If you do the same in your other scripts, or if you set up this cron job to run every minute while it takes nmap more than a minute to complete scanning the network, then new.txt may be overwritten at the wrong time.

Please fix your script as follows and check if the problem disappears:

#!/bin/bash

tmpfile="$(mktemp)"
trap "rm $tmpfile" EXIT
nmap -sn 192.168.0.1-255 | grep -Eo '192\.168\.0\.[0-9]{1,3}' > "$tmpfile"
date >> network_log
echo ---------------------------- >> network_log
cat "$tmpfile" >> network_log

Thank-you for all these suggestions, I would have never realized the tmpfile thing is alot better, especially since nmap does take a relatively long time to run. & thankyou for regex corrections

bash - different results between crontab and running script manually -...

bash cron network-programming nmap
Rectangle 27 2

There are many ways to do it with sed or awk:

sed -n '/TERMINATE/,$p' file

This looks for TERMINATE in your file and prints from that line up to the end of the file.

awk '/TERMINATE/,0' file

This is exactly the same behaviour as sed.

In case you know the number of the line from which you want to start printing, you can specify it together with NR (number of record, which eventually indicates the number of the line):

awk 'NR>=535' file
$ seq 10 > a        #generate a file with one number per line, from 1 to 10
$ sed -n '/7/,$p' a
7
8
9
10
$ awk '/7/,0' a
7
8
9
10
$ awk 'NR>=7' a
7
8
9
10
more +7 file

This includes the matching line, which is not what is wanted in this question.

@mivk well, this is also the case of the accepted answer and the 2nd most upvoted, so the problem may be with a misleading title.

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 2

There are many ways to do it with sed or awk:

sed -n '/TERMINATE/,$p' file

This looks for TERMINATE in your file and prints from that line up to the end of the file.

awk '/TERMINATE/,0' file

This is exactly the same behaviour as sed.

In case you know the number of the line from which you want to start printing, you can specify it together with NR (number of record, which eventually indicates the number of the line):

awk 'NR>=535' file
$ seq 10 > a        #generate a file with one number per line, from 1 to 10
$ sed -n '/7/,$p' a
7
8
9
10
$ awk '/7/,0' a
7
8
9
10
$ awk 'NR>=7' a
7
8
9
10
more +7 file

This includes the matching line, which is not what is wanted in this question.

@mivk well, this is also the case of the accepted answer and the 2nd most upvoted, so the problem may be with a misleading title.

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 1

  • is much, much faster than sed especially working on really big file. It works up to 10M lines (or whatever you put in) so no harm in making this big enough to handle about anything you hit.

bash - How to get the part of file after the line that matches grep ex...

bash shell scripting grep
Rectangle 27 139

Piping find into grep is often more convenient; it gives you the full power of regular expressions for arbitrary wildcard matching.

For example, to find all files with case insensitive string "foo" in the filename:

~$ find . -print | grep -i foo

find also has the -iname, -regex, and -iregex flags for case-insensitive wildcard, regex, and case-insensitive regex matching, so piping to grep is unnecessary.

I don't think it is about being unnecessary, but being more convenient.

However, piping to grep -v can allow you to use simple strings or regexes to remove entries you don't want.

linux - How can I recursively find all files in current and subfolders...

linux shell
Rectangle 27 132

Piping find into grep is often more convenient; it gives you the full power of regular expressions for arbitrary wildcard matching.

For example, to find all files with case insensitive string "foo" in the filename:

~$ find . -print | grep -i foo

find also has the -iname, -regex, and -iregex flags for case-insensitive wildcard, regex, and case-insensitive regex matching, so piping to grep is unnecessary.

I don't think it is about being unnecessary, but being more convenient.

However, piping to grep -v can allow you to use simple strings or regexes to remove entries you don't want.

linux - How can I recursively find all files in current and subfolders...

linux shell
Rectangle 27 22

grep is a regular expression matcher, not a syntax highlighter :). You'll have to use multiple invocations of grep, using a different value of GREP_COLOR for each.

GREP_COLOR="1;32" grep foo file.txt | GREP_COLOR="1;36" grep bar

That would highlight "foo" and "bar" in different colors in lines that match both. I don't think there is a (simple) way to handle all occurrences of either pattern, short of merging the output stream of two independent calls:

{ GREP_COLOR="1;32" grep foo file.txt
  GREP_COLOR="1;36" grep bar file.txt
} | ...

which will obviously look different than if there were a way to assign a separate color to each regular expression.

You can use awk to substitute each match with itself wrapped in the correct control code.

echo "foo bar" | awk '{ gsub("bar", "\033[1;33m&\033[0m");
                         gsub("foo", "\033[1;36m&\033[0m"); print }'

In each line, you globally replace anything matching the given regular expression with itself (&) wrapped in the ANSI escape sequences for desired color (which grep --color does for you). After processing all of the possible matches, you need to explicitly print the line.

bash - Grep output with multiple Colors? - Stack Overflow

bash colors sed awk grep
Rectangle 27 22

grep is a regular expression matcher, not a syntax highlighter :). You'll have to use multiple invocations of grep, using a different value of GREP_COLOR for each.

GREP_COLOR="1;32" grep foo file.txt | GREP_COLOR="1;36" grep bar

That would highlight "foo" and "bar" in different colors in lines that match both. I don't think there is a (simple) way to handle all occurrences of either pattern, short of merging the output stream of two independent calls:

{ GREP_COLOR="1;32" grep foo file.txt
  GREP_COLOR="1;36" grep bar file.txt
} | ...

which will obviously look different than if there were a way to assign a separate color to each regular expression.

You can use awk to substitute each match with itself wrapped in the correct control code.

echo "foo bar" | awk '{ gsub("bar", "\033[1;33m&\033[0m");
                         gsub("foo", "\033[1;36m&\033[0m"); print }'

In each line, you globally replace anything matching the given regular expression with itself (&) wrapped in the ANSI escape sequences for desired color (which grep --color does for you). After processing all of the possible matches, you need to explicitly print the line.

bash - Grep output with multiple Colors? - Stack Overflow

bash colors sed awk grep