Top 25 SED Commands

Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of editors.

Here are the top 25 SED commands voted by everyone

1) Stream YouTube URL directly to mplayer.

mplayer -fs $(echo “http://youtube.com/get_video.php?$(curl -s $youtube_url | sed -n “/watch_fullscreen/s;.*\(video_id.\+\)&title.*;\1;p”)”)

This is the result of a several week venture without X. I found myself totally happy without X (and by extension without flash) and was able to do just about anything but watch YouTube videos… so this a the solution I came up with for that. I am sure this can be done better but this does indeed work… and tends to work far better than YouTube’s ghetto proprietary flash player ;-)

Replace $youtube_url with any youtube URL you want and this will scrape the site for the _real_ URL to the full quality .FLV file on Youtube’s server and will then will hand that over to mplayer (or vlc or whatever you want) to be streamed.

In some browsers you can replace $youtube_url with just a % or put this in a shell script so all YouTube URLs can be handed directly off to your media player of choice for true streaming without the need for Flash or a downloader like clive. (I do however fully recommend clive if you wish to archive videos instead of streaming them)

If any interest is shown I would be more than happy to provide similar commands for other sites. Most streaming flash players use similar logic to YouTube.

Also, if you want to download videos, just add the -dumpstream option to mplayer.

2) Google Translate

translate(){ wget -qO- “http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=$1&langpair=$2|${3:-en}” | sed ‘s/.*”translatedText”:”\([^”]*\)”.*}/\1\n/’; }

Usage:

translate <phrase> <source-language> <output-language>Example:

translate hello en esSee this for a list of language codes:

http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes

3) Print all the lines between 10 and 20 of a file

sed -n ‘10,20p’ <filename>

Similarly, if you want to print from 10 to the end of line you can use: sed -n ’10,$p’ filename
This is especially useful if you are dealing with a large file. Sometimes you just want to extract a sample without opening the entire file.

4) Graphical tree of sub-directories

ls -R | grep “:$” | sed -e ‘s/:$//’ -e ‘s/[^-][^\/]*\//–/g’ -e ‘s/^/   /’ -e ‘s/-/|/’

Prints a graphical directory tree from your current directory

5) Check your unread Gmail from the command line

curl -u username –silent “https://mail.google.com/mail/feed/atom” | perl -ne ‘print “\t” if /<name>/; print “$2\n” if /<(title|name)>(.*)<\/\1>/;’

Checks the Gmail ATOM feed for your account, parses it and outputs a list of unread messages.

6) To print a specific line from a file

sed -n 5p <file>

You can get one specific line during any procedure. Very interesting to be used when you know what line you want.

7) Find geographical location of an ip address

lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|grep address|egrep ‘city|state|country’|awk ‘{print $3,$4,$5,$6,$7,$8}’|sed ‘s\ip address flag \\’|sed ‘s\My\\’

I save this to bin/iptrace and run “iptrace ipaddress” to get the Country, City and State of an ip address using the http://ipadress.com service.

I add the following to my script to get a tinyurl of the map as well:

URL=`lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|grep details|awk ‘{print $2}’`

lynx -dump http://tinyurl.com/create.php?url=$URL|grep tinyurl|grep “19. http”|awk ‘{print $2}’

8) Convert PDF to JPG

for file in `ls *.pdf`; do convert -verbose -colorspace RGB -resize 800 -interlace none -density 300 -quality 80 $file `echo $file | sed ‘s/\.pdf$/\.jpg/’`; done

(relies on ‘imagemagick’)

This command will convert all .pdf files in a directory into a 800px (wide or height, whichever is smaller) image (with the aspect ratio kept) .jpg.

If the file is named ‘example1.pdf’ it will be named ‘example1.jpg’ when it is complete.

This is a VERY worthwhile command! People pay hundreds of dollars for this in the Windows world.

My .jpg files average between 150kB to 300kB, but your’s may differ.

9) Remove a line in a text file. Useful to fix “ssh host key change” warnings

sed -i 8d ~/.ssh/known_hosts

10) Recursive search and replace old with new string, inside files

grep -rl oldstring . |xargs sed -i -e ‘s/oldstring/newstring/’

recursively traverse the directory structure from . down, look for string “oldstring” in all files, and replace it with “newstring”, wherever found

also:

grep -rl oldstring . |xargs perl -pi~ -e 's/oldstring/newstring'

11) Efficiently print a line deep in a huge log file

sed ‘1000000!d;q’ < massive-log-file.log

Sed stops parsing at the match and so is much more effecient than piping head into tail or similar. Grab a line range using

sed '999995,1000005!d' < my_massive_file

12) View all date formats, Quick Reference Help Alias

alias dateh=’date –help|sed “/^ *%a/,/^ *%Z/!d;y/_/!/;s/^ *%\([:a-z]\+\) \+/\1_/gI;s/%/#/g;s/^\([a-y]\|[z:]\+\)_/%%\1_%\1_/I”|while read L;do date “+${L}”|sed y/!#/%%/;done|column -ts_’

If you have used bash for any scripting, you’ve used the date command alot. It’s perfect for using as a way to create filename’s dynamically within aliases,functions, and commands like below.. This is actually an update to my first alias, since a few commenters (below) had good observations on what was wrong with my first command.

# creating a date-based ssh-key for askapache.github.com

ssh-keygen -f ~/.ssh/`date +git-$USER@$HOSTNAME-%m-%d-%g` -C 'webmaster@askapache.com' # /home/gpl/.ssh/git-gplnet@askapache.github.com-04-22-10# create a tar+gzip backup of the current directory

tar -czf $(date +$HOME/.backups/%m-%d-%g-%R-`sed -u 's/\//#/g' <<< $PWD`.tgz) . # tar -czf /home/gpl/.backups/04-22-10-01:13-#home#gpl#.rr#src.tgz .I personally find myself having to reference

date --help quite a bit as a result. So this nice alias saves me a lot of time. This is one bdash mofo. Works in sh and bash (posix), but will likely need to be changed for other shells due to the parameter substitution going on.. Just extend the sed command, I prefer sed to pretty much everything anyways.. but it’s always preferable to put in the extra effort to go for as much builtin use as you can. Otherwise it’s not a top one-liner, it’s a lazyboy recliner.

Here’s the old version:

alias dateh='date --help|sed "/^ *%%/,/^ *%Z/!d;s/ \+/ /g"|while read l;do date "+ %${l/% */}_${l/% */}_${l#* }";done|column -s_ -t'This trick from  [ http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html bash_profile ]

13) Generate a Random MAC address

MAC=`(date; cat /proc/interrupts) | md5sum | sed -r ‘s/^(.{10}).*$/\1/; s/([0-9a-f]{2})/\1:/g; s/:$//;’`

Original author unknown (I believe off of a wifi hacking forum).

Used in conjuction with ifconfig and cron.. can be handy (especially spoofing AP’s)

14) Change prompt to MS-DOS one (joke)

export PS1=”C:\$( pwd | sed ‘s:/:\\\\\\:g’ )\\> “

15) Delete the specified line

sed -i 8d ~/.ssh/known_hosts

16) geoip information

curl -s “http://www.geody.com/geoip.php?ip=$(curl -s icanhazip.com)” | sed ‘/^IP:/!d;s/<[^>][^>]*>//g’

This script gives a single line as shown in the sample output.

17) Stream YouTube URL directly to mplayer

id=”dMH0bHeiRNg”;mplayer -fs http://youtube.com/get_video.php?video_id=$id\&t=$(curl -s http://www.youtube.com/watch?v=$id | sed -n ‘s/.*, “t”: “\([^”]*\)”, .*/\1/p’)

The original doesn’t work for me – but this does. I’m guessing that Youtube updated the video page so the original doesn’t work.

18) Another Curl your IP command

curl -s http://checkip.dyndns.org | sed ‘s/[a-zA-Z<>/ :]//g’

Just another curl command to get your public facing IP

19) Print just line 4 from a textfile

sed -n ‘4{p;q}’

Prints the 4th line and then quits. (Credit goes to flatcap in comments: http://www.commandlinefu.com/commands/view/6031/print-just-line-4-from-a-textfile#comment.)

20) Working random fact generator

wget randomfunfacts.com -O – 2>/dev/null | grep \<strong\> | sed “s;^.*<i>\(.*\)</i>.*$;\1;”

Though without infinite time and knowledge of how the site will be designed in the future this may stop working, it still will serve as a simple straight forward starting point.

This uses the observation that the only item marked as strong on the page is the single logical line that includes the italicized fact.

If future revisions of the page show failure, or intermittent failure, one may simply alter the above to read.

wget randomfunfacts.com -O - 2>/dev/null | tee lastfact | grep \<strong\> | sed "s;^.*<i>\(.*\)</i>.*$;\1;"

The file lastfact, can then be examined whenever the command fails.

21) Given process ID print its environment variables

sed ‘s/\o0/\n/g’ /proc/INSERT_PID_HERE/environ

22) Pronounce an English word using Dictionary.com

pronounce(){ wget -qO- $(wget -qO- “http://dictionary.reference.com/browse/$@” | grep ‘soundUrl’ | head -n 1 | sed ‘s|.*soundUrl=\([^&]*\)&.*|\1|’ | sed ‘s/%3A/:/g;s/%2F/\//g’) | mpg123 -; }

23) find and replace tabs for spaces within files recursively

find ./ -type f -exec sed -i ‘s/\t/  /g’ {} \;

24) count IPv4 connections per IP

netstat -anp |grep ‘tcp\|udp’ | awk ‘{print $5}’ | sed s/::ffff:// | cut -d: -f1 | sort | uniq -c | sort -n

usefull in case of abuser/DoS attacks.

25) display an embeded help message from bash script header

[ “$1” == “–help” ] && { sed -n -e ‘/^# Usage:/,/^$/ s/^# \?//p’ < $0; exit; }

With this one liner you can easily output a standard help message using the following convention:

Usage: is the start marker

Stop at the last #

wget randomfunfacts.com -O – 2>/dev/null | grep \<strong\> | sed “s;^.*<i>\(.*\)</i>.*$;\1;”Working random fact generator

And there you have it the top 25 Voted SED commands
These where brought to you by commandlinefu
Check them out and if possible add your own command line gem
and remember when in doubt check the man pages.

4 Comments

  1. Pingback: Tweets that mention Top 25 SED Commands -- Topsy.com

  2. juan manuel fangio

    Reply

    Heres one to cache some random.org quotes for display at login, try running it with or without $1==”gimmemore” ^_^

    #!/bin/bash

    quotefile=”${0%/*}/Quotes”
    fun=$[$RANDOM%15+1]

    [ “$1” == “gimmemore” ] &’>$quotefile
    exit 0
    }

    quote=$[$RANDOM%$[$(sed -n ‘/^.*\]#.*$/p’ “$quotefile” | wc -l)-2]+1]
    quote=$(bash -c “cat -n \”$quotefile\” | sed -n ‘/^.*\]#.*$/p’ | sed -n \”${quote}p\” | sed ‘s/^ *\([0-9]\+\).*$/\1/g'”)
    sed -n “$quote,/^.*\]#.*$/{$quoted;/^.*\]#.*$/d;p}” $quotefile

Leave Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.