Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of editors.
Here are the top 25 SED commands voted by everyone
1) Stream YouTube URL directly to mplayer.
mplayer -fs $(echo “http://youtube.com/get_video.php?$(curl -s $youtube_url | sed -n “/watch_fullscreen/s;.*\(video_id.\+\)&title.*;\1;p”)”)
This is the result of a several week venture without X. I found myself totally happy without X (and by extension without flash) and was able to do just about anything but watch YouTube videos… so this a the solution I came up with for that. I am sure this can be done better but this does indeed work… and tends to work far better than YouTube’s ghetto proprietary flash player ;-)
Replace $youtube_url with any youtube URL you want and this will scrape the site for the _real_ URL to the full quality .FLV file on Youtube’s server and will then will hand that over to mplayer (or vlc or whatever you want) to be streamed.
In some browsers you can replace $youtube_url with just a % or put this in a shell script so all YouTube URLs can be handed directly off to your media player of choice for true streaming without the need for Flash or a downloader like clive. (I do however fully recommend clive if you wish to archive videos instead of streaming them)
If any interest is shown I would be more than happy to provide similar commands for other sites. Most streaming flash players use similar logic to YouTube.
Also, if you want to download videos, just add the -dumpstream option to mplayer.
2) Google Translate
translate(){ wget -qO- “http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=$1&langpair=$2|${3:-en}” | sed ‘s/.*”translatedText”:”\([^”]*\)”.*}/\1\n/’; }
Usage:
translate <phrase> <source-language> <output-language>Example:
translate hello en esSee this for a list of language codes:
http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
3) Print all the lines between 10 and 20 of a file
sed -n ‘10,20p’ <filename>
Similarly, if you want to print from 10 to the end of line you can use: sed -n ’10,$p’ filename
This is especially useful if you are dealing with a large file. Sometimes you just want to extract a sample without opening the entire file.
4) Graphical tree of sub-directories
ls -R | grep “:$” | sed -e ‘s/:$//’ -e ‘s/[^-][^\/]*\//–/g’ -e ‘s/^/ /’ -e ‘s/-/|/’
Prints a graphical directory tree from your current directory
5) Check your unread Gmail from the command line
curl -u username –silent “https://mail.google.com/mail/feed/atom” | perl -ne ‘print “\t” if /<name>/; print “$2\n” if /<(title|name)>(.*)<\/\1>/;’
6) To print a specific line from a file
sed -n 5p <file>
7) Find geographical location of an ip address
lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|grep address|egrep ‘city|state|country’|awk ‘{print $3,$4,$5,$6,$7,$8}’|sed ‘s\ip address flag \\’|sed ‘s\My\\’
I save this to bin/iptrace and run “iptrace ipaddress” to get the Country, City and State of an ip address using the http://ipadress.com service.
I add the following to my script to get a tinyurl of the map as well:
URL=`lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|grep details|awk ‘{print $2}’`
lynx -dump http://tinyurl.com/create.php?url=$URL|grep tinyurl|grep “19. http”|awk ‘{print $2}’
8) Convert PDF to JPG
for file in `ls *.pdf`; do convert -verbose -colorspace RGB -resize 800 -interlace none -density 300 -quality 80 $file `echo $file | sed ‘s/\.pdf$/\.jpg/’`; done
(relies on ‘imagemagick’)
This command will convert all .pdf files in a directory into a 800px (wide or height, whichever is smaller) image (with the aspect ratio kept) .jpg.
If the file is named ‘example1.pdf’ it will be named ‘example1.jpg’ when it is complete.
This is a VERY worthwhile command! People pay hundreds of dollars for this in the Windows world.
My .jpg files average between 150kB to 300kB, but your’s may differ.
9) Remove a line in a text file. Useful to fix “ssh host key change” warnings
sed -i 8d ~/.ssh/known_hosts
10) Recursive search and replace old with new string, inside files
grep -rl oldstring . |xargs sed -i -e ‘s/oldstring/newstring/’
recursively traverse the directory structure from . down, look for string “oldstring” in all files, and replace it with “newstring”, wherever found
also:
grep -rl oldstring . |xargs perl -pi~ -e 's/oldstring/newstring'11) Efficiently print a line deep in a huge log file
sed ‘1000000!d;q’ < massive-log-file.log
Sed stops parsing at the match and so is much more effecient than piping head into tail or similar. Grab a line range using
sed '999995,1000005!d' < my_massive_file12) View all date formats, Quick Reference Help Alias
alias dateh=’date –help|sed “/^ *%a/,/^ *%Z/!d;y/_/!/;s/^ *%\([:a-z]\+\) \+/\1_/gI;s/%/#/g;s/^\([a-y]\|[z:]\+\)_/%%\1_%\1_/I”|while read L;do date “+${L}”|sed y/!#/%%/;done|column -ts_’
If you have used bash for any scripting, you’ve used the date command alot. It’s perfect for using as a way to create filename’s dynamically within aliases,functions, and commands like below.. This is actually an update to my first alias, since a few commenters (below) had good observations on what was wrong with my first command.
# creating a date-based ssh-key for askapache.github.com
ssh-keygen -f ~/.ssh/`date +git-$USER@$HOSTNAME-%m-%d-%g` -C 'webmaster@askapache.com' # /home/gpl/.ssh/git-gplnet@askapache.github.com-04-22-10# create a tar+gzip backup of the current directory
tar -czf $(date +$HOME/.backups/%m-%d-%g-%R-`sed -u 's/\//#/g' <<< $PWD`.tgz) . # tar -czf /home/gpl/.backups/04-22-10-01:13-#home#gpl#.rr#src.tgz .I personally find myself having to reference
date --help quite a bit as a result. So this nice alias saves me a lot of time. This is one bdash mofo. Works in sh and bash (posix), but will likely need to be changed for other shells due to the parameter substitution going on.. Just extend the sed command, I prefer sed to pretty much everything anyways.. but it’s always preferable to put in the extra effort to go for as much builtin use as you can. Otherwise it’s not a top one-liner, it’s a lazyboy recliner.
Here’s the old version:
alias dateh='date --help|sed "/^ *%%/,/^ *%Z/!d;s/ \+/ /g"|while read l;do date "+ %${l/% */}_${l/% */}_${l#* }";done|column -s_ -t'This trick from [ http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html bash_profile ]
13) Generate a Random MAC address
MAC=`(date; cat /proc/interrupts) | md5sum | sed -r ‘s/^(.{10}).*$/\1/; s/([0-9a-f]{2})/\1:/g; s/:$//;’`
Original author unknown (I believe off of a wifi hacking forum).
Used in conjuction with ifconfig and cron.. can be handy (especially spoofing AP’s)
14) Change prompt to MS-DOS one (joke)
export PS1=”C:\$( pwd | sed ‘s:/:\\\\\\:g’ )\\> “
15) Delete the specified line
sed -i 8d ~/.ssh/known_hosts
16) geoip information
curl -s “http://www.geody.com/geoip.php?ip=$(curl -s icanhazip.com)” | sed ‘/^IP:/!d;s/<[^>][^>]*>//g’
This script gives a single line as shown in the sample output.
17) Stream YouTube URL directly to mplayer
id=”dMH0bHeiRNg”;mplayer -fs http://youtube.com/get_video.php?video_id=$id\&t=$(curl -s http://www.youtube.com/watch?v=$id | sed -n ‘s/.*, “t”: “\([^”]*\)”, .*/\1/p’)
The original doesn’t work for me – but this does. I’m guessing that Youtube updated the video page so the original doesn’t work.
18) Another Curl your IP command
curl -s http://checkip.dyndns.org | sed ‘s/[a-zA-Z<>/ :]//g’
Just another curl command to get your public facing IP
19) Print just line 4 from a textfile
sed -n ‘4{p;q}’
Prints the 4th line and then quits. (Credit goes to flatcap in comments: http://www.commandlinefu.com/commands/view/6031/print-just-line-4-from-a-textfile#comment.)
20) Working random fact generator
wget randomfunfacts.com -O – 2>/dev/null | grep \<strong\> | sed “s;^.*<i>\(.*\)</i>.*$;\1;”
Though without infinite time and knowledge of how the site will be designed in the future this may stop working, it still will serve as a simple straight forward starting point.
This uses the observation that the only item marked as strong on the page is the single logical line that includes the italicized fact.
If future revisions of the page show failure, or intermittent failure, one may simply alter the above to read.
wget randomfunfacts.com -O - 2>/dev/null | tee lastfact | grep \<strong\> | sed "s;^.*<i>\(.*\)</i>.*$;\1;"The file lastfact, can then be examined whenever the command fails.
21) Given process ID print its environment variables
sed ‘s/\o0/\n/g’ /proc/INSERT_PID_HERE/environ
22) Pronounce an English word using Dictionary.com
pronounce(){ wget -qO- $(wget -qO- “http://dictionary.reference.com/browse/$@” | grep ‘soundUrl’ | head -n 1 | sed ‘s|.*soundUrl=\([^&]*\)&.*|\1|’ | sed ‘s/%3A/:/g;s/%2F/\//g’) | mpg123 -; }
23) find and replace tabs for spaces within files recursively
find ./ -type f -exec sed -i ‘s/\t/ /g’ {} \;
24) count IPv4 connections per IP
netstat -anp |grep ‘tcp\|udp’ | awk ‘{print $5}’ | sed s/::ffff:// | cut -d: -f1 | sort | uniq -c | sort -n
usefull in case of abuser/DoS attacks.
25) display an embeded help message from bash script header
[ “$1” == “–help” ] && { sed -n -e ‘/^# Usage:/,/^$/ s/^# \?//p’ < $0; exit; }
With this one liner you can easily output a standard help message using the following convention:
Usage: is the start marker
Stop at the last #
And there you have it the top 25 Voted SED commands
These where brought to you by commandlinefu
Check them out and if possible add your own command line gem
and remember when in doubt check the man pages.
