16 Cool Ass Random Linux Stuff

$ echo “Hello world”
Hello world

Urfix has compiled a sweet ass list of random stuff you can do from the command line. Hope you enjoy.

Fry: Make up some feelings and tell her you have them.

1 Multiple variable assignments from command output in BASH

read day month year <<< $(date +'%d %m %y')

This version uses read instead of eval.

2 Generate an XKCD #936 style 4 word password

shuf -n4 /usr/share/dict/words | tr -d '\n'

4 random words are better than one obfuscated word
http://xkcd.com/936/

3 All IP connected to my host

netstat -lantp | grep ESTABLISHED |awk '{print $5}' | awk -F: '{print $1}' | sort -u

find all computer connected to my host through TCP connection.

4 Ask for a password, the passwd-style

read -s -p"Password: " USER_PASSWORD_VARIABLE; echo

You can ask repeatedly for a non-blank password using this function:
function read_password() {
while [ ! -n “$USER_PASSWORD” ]; do
read -s -p”Password: ” USER_PASSWORD
if [ ! -n “$USER_PASSWORD” ]; then
echo “ERROR: You must specify a valid password, please try again”
fi
echo
done
}
Also you can set a time out (in seconds) to write the password
read -t 10 -s -p”Password: ” USER_PASSWORD_VARIABLE
if [ ! $? -eq 0 ]; then
echo “Time out!”
fi

5 Check syntax for all PHP files in the current directory and all subdirectories

find . -name \*.php -exec php -l "{}" \;

6mkdir & cd into it as single command

mkdir /home/foo/doc/bar && cd $_

The biggest advantage of this over the functions is that it is portable.

7 Merge PDFs into single file

gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=output.pdf input1.pdf input2.pdf ...

8 Create a QR code image in MECARD format

qrencode -o myqr.png 'MECARD:N:Daprophet,Isaiah;TEL:8881234567;EMAIL:isaiah.daprophet@urfix.com;;'

Add the QR code image on your webpage, business card ., etc, so people can scan it and quick add to their Contact Address Book. Tested on iPhone with QRreader.

9 Chmod all directories (excluding files)

find public_html/ -type d -exec chmod 755 {} +

+ at the end means that many filenames will be passed to every chmod call, thus making it faster. And find own {} makes sure that it will work with spaces and other characters in filenames.

10 List only the directories

find . -maxdepth 1 -type d | sort

11 copy with progress bar – rsync

rsync -rv   --progress

-r for recursive (if you want to copy entire directories)
src for the source file (or wildcards)
dst for the destination
–progress to show a progress bar

12 Find all files of a type and copy them elsewhere while keeping intact their full directory structure using find and cpio

find . -iname "*.flac" | cpio -pdm /Volumes/Music/FLAC

.flac is the filetype.
/Volumes/Music/FLAC is the destination.

13 Share a screen session

screen -x 

14 Backup entire system through SSH

ssh -C USER@HOST tar -c --exclude /proc --exclude /sys / | tar -x

15 Find the package a command belongs to on debian-based distros

apt-file search iostat

16 View files opened by a program on startup and shutdown

sudo lsof -rc command >> /tmp/command.txt

Run this before you run a command in order to see what the command does as it starts.
The -c flag is useful here as the PID is unknown before startup.
All config files, libraries, logs, ports, etc used by the command as it starts up, (and shuts down) will be captured at 1s intervals and written to a file.
Useful for debugging etc.

NOT – The Best 25 Linux Commands

Here is another installment of Command Line Gems, I present you with the “Not” The best Linux Commands.

1) Save an HTML page, and covert it to a .pdf file

wget $URL | htmldoc --webpage -f "$URL".pdf - ; xpdf "$URL".pdf &

Uses htmldoc to perform the conversion

2) backup and synchronize entire remote folder locally (curlftpfs and rsync over FTP using FUSE FS)

curlftpfs ftp://YourUsername:YourPassword@YourFTPServerURL /tmp/remote-website/ && rsync -av /tmp/remote-website/* /usr/local/data_latest && umount /tmp/remote-website

connect to a remote server using ftp protocol over FUSE file system, then rsync the remote folder to a local one and then unmount the remote ftp server (FUSE FS)

it can be divided to 3 different commands and you should have curlftpfs and rsync installed

3)  Do some Perl learning…

podwebserver& sleep 2; elinks 'http://127.0.0.1:8020'


Prerequisites: module Pod::Webserver installed. You can install it typing:
sudo perl -MCPAN -e 'install Pod::Webserver'
You can replace elinks with your fav browser. For FF:
podwebserver& sleep 2; firefox -remote 'openurl( http://127.0.0.1:8020/, new-tab )'
If you have Firefox open, this will pop-up the index web in a new tab.

4) Serve current directory tree at http://$HOSTNAME:8080/

twistd -n web –path .

5) Convert (almost) any video file into webm format for online html5 streaming

ffmpeg -i input_file.avi output_file.webm

6) Compare an archive with filesystem

tar dfz horde-webmail-1.2.3.tar.gz

and you quickly know the files you changed

7) tar via network

tar cfX - exclude_opt_weblogic . | ssh tmp-esxsb044 "cd /opt/weblogic ; tar xf -"

8) download and install the software package in one step


rpm -ivh 'http://www.website.com/path/to/desired_software_package.rpm'

9) Download all images from a site

wget -r -l1 --no-parent -nH -nd -P/tmp -A".gif,.jpg" http://example.com/images

This recursively downloads all images from a given website to your /tmp directory. The -nH and -nd switches disable downloading of the directory structure.

10) check site ssl certificate dates

echo | openssl s_client -connect www.google.com:443 2>/dev/null |openssl x509 -dates -noout
remotely connects to an https site, fetches the ssl certificate and displays the valid dates for the cert

11) get all pdf and zips from a website using wget

wget –reject html,htm –accept pdf,zip -rl1 url

If the site uses https, use:

wget --reject html,htm --accept pdf,zip -rl1 --no-check-certificate https-url

12) Download an entire static website to your local machine

wget –recursive  –page-requisites –convert-links www.moyagraphix.co.za

Very nice command when you want to download a site locally to your machine, including images, css and javascript

13) Quickly share code or text from vim to others.

:w !curl -F "sprunge=<-" http://sprunge.us | xclip

Sprunge.us is a code/text sharing site like pastebin, but it is easy to post stuff from the command line.

How it works:

:w !commandIn vim, w writes the current tab to a file when a filename is given afterwards, but if !command is given, the output is piped to the stdin of command.

curl -F "sprunge=<-" http://sprunge.uscurl is an HTTP client. The -F option does an HTTP post to the given address. The data in the quotes is passed in the post. The “sprunge=” part sets up a fieldname – the part that follows is what is associated with the name. The “<” tells curl to send data from the file descriptor that follows it. The “-” in bash is a file descriptor that points to stdin instead of an actual file; in this case, stdin is being piped in from vim. After we send the HTTP post to sprunge.us, it will give back a url that points to the data you just sent.

| xclipxclip is a utility that lets you put stuff in your clipboard or selection buffer. This part uses a bash pipe ( | ) to redirect the stdout of the previous command to the stdin of the next command. So, we’re capturing the URL that curl gave us and putting it into the selection buffer, ready to paste into IRC or a forum.

Notes:

Of course, for this to work, you must have curl (which comes by default on most distroes), and xclip installed.

When you share the url, you can append “?lang” to highlight and have line numbers. Check out http://sprunge.us/BZXV?log for line numbers and http://sprunge.us/BZXV?ruby for highlighting.

If you prefer to use ctrl-v (paste from clipboard) instead of middle-click (paste from selection buffer), look up options on xclip – you can do that.

14) Create an SSH tunnel for accessing your remote MySQL database with a local port

ssh -CNL 3306:localhost:3306 user@site.com


15) Extract dd-image from VirtualBox VDI container and mount it

vditool COPYDD my.vdi my.dd ; sudo mount -t ntfs -o ro,noatime,noexex,loop,offset=32256 my.dd ./my_dir

Tested with NTFS and found on this site:

http://forensicir.blogspot.com/2008/01/virtualbox-and-forensics-tools.html

The first 32256 bytes is the MBR

16) Forward port 8888 to remote machine for SOCKS Proxy

ssh -D 8888 user@site.com

Simply change your web browser’s proxy settings to point to a SOCKS proxy at port 8888 and you’re good to go.

17)  Dump a web page

curl -s http://google.com | hexdump -C|less

Useful to browse dangerous web sites.

18) Edit a google doc with vim

google docs edit –title “To-Do List” –editor vim

Google just released a new commend line tool offering all sorts of new services from the commend line. One of them is uploading a youtube video but there are plenty more google services to interact with.

Download it here: http://code.google.com/p/googlecl/

Manual: http://code.google.com/p/googlecl/wiki/Manual

This specific command courtesy of lifehacker:http://lifehacker.com/5568817/

Though all can be found in manual page linked above.

19) Colorful man

apt-get install most && update-alternatives –set pager /usr/bin/most

That command installs “most” and make this command as the default man reader. The “most” works like “less” (the current man reader), but it render colors for manpages and may do more things. Read “man most”.

You can see a preview here: http://www.dicas-l.com.br/dicas-l/20090718.php

20) Mirror a directory structure from websites with an Apache-generated file indexes


lftp -e "mirror -c" http://example.com/foobar/

wget/curl/friends are not good with mirroring files off websites, especially those with Apache-generated directory listings. These tools endlessly waste time downloading useless index HTML pages. lftp’s mirror command does a better job without the mess.

21) Convert images to a multi-page pdf

convert -adjoin -page A4 *.jpeg multipage.pdf

The linux package imagmagick is required for this command

22) printing barcodes

ls /home | head -64 | barcode -t 4×16 | lpr

64 elements max on 16 rows, 4 cols.

GNU Barcode will adapt automagically the width and the eight of your elements to fill the page.

Standard output format is PostScript.

23) convert a web page into a png

touch $2;firefox -print $1 -printmode PNG -printfile $2

This requires the command-line print extension (see #2861 for more details). I use it to make up complex images with formatted text using CSS and whatnot. It’s a lot slicker than imagemagick for certain things. Now imagine using a local webserver with PHP and a database to generate the images. Oh, the possibilities…

24) Watch the progress of ‘dd’

pkill -USR1 ^dd$

The ‘dd’ command doesn’t provide a progress when writing data. So, sending the “USR1” signal to the process will spit out its progress as it writes data. This command is superior to others on the site, as it doesn’t require you to previously know the PID of the dd command.

25) Hex math with bc

echo 'obase=16; C+F' | bc

To do hex to binary: echo ‘ibase=16; obase=2; 16*16’ | bc # prints: 111100100
To do 16*16 from decimal to hex: echo ‘ibase=10; obase=16; 16*16’ | bc # prints: 100

You get the idea… Alternatively, run bc in interactive mode (see man page)

11 Awesome DD Commands

dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data. dd is an application that will “convert and copy a file”according to the referenced manual page for Version 7 Unix and is most likely inspired from DD found in IBM JCL, and the command’s syntax is meant to be reminiscent of this.
Learn how to use DD by visiting this site Here

Hope you enjoy

1) Duplicate several drives concurrently

dd if=/dev/sda | tee >(dd of=/dev/sdb) | dd of=/dev/sdc

If you have some drive imaging to do, you can boot into any liveCD and use a commodity machine. The drives will be written in parallel.

To improve efficiency, specify a larger block size in dd:

dd if=/dev/sda bs=64k | tee >(dd of=/dev/sdb bs=64k) | dd of=/dev/sdc bs=64kTo image more drives , insert them as additional arguments to tee:

dd if=/dev/sda | tee >(dd of=/dev/sdb) >(dd of=/dev/sdc) >(dd of=/dev/sdd) | dd of=/dev/sde

2) create an emergency swapfile when the existing swap space is getting tight

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024000;sudo mkswap /swapfile; sudo swapon /swapfile

Create a temporary file that acts as swap space. In this example it’s a 1GB file at the root of the file system. This additional capacity is added to the existing swap space.

3) Backup your hard drive with dd

sudo dd if=/dev/sda of=/media/disk/backup/sda.backup

This will create an exact duplicate image of your hard drive that you can then restore by simply reversing the “if” & “of” locations.

sudo dd if=/media/disk/backup/sda.backup of=/dev/sdaAlternatively, you can use an SSH connection to do your backups:

dd if=/dev/sda | ssh user@ssh.server.com dd of=~/backup/sda.backup

4) Convert a Nero Image File to ISO

dd bs=1k if=image.nrg of=image.iso skip=300

This line removes the 300k header from a Nero image file converting it to ISO format

5) send DD a signal to print its progress

while :;do killall -USR1 dd;sleep 1;done

every 1sec sends DD the USR1 signal which causes DD to print its progress.

6) show dd progress part 2

killall -USR1 dd

if you need see progress of long dd command, enter subj on other console

7) How to copy CD/DVD into hard disk (.iso)

dd if=/dev/cdrom of=whatever.iso

A dear friend of mine asked me how do I copy a DVD to your hard drive? If you want to make a copy of the ISO image that was burned to a CD or DVD, insert that medium into your CD/DVD drive and (assuming /dev/cdrom is associated with your computer?s CD drive) type the following command

8) Watch the progress of ‘dd’

dd if=/dev/zero | pv | dd of=/dev/null

need pv (pipe view) :

http://www.ivarch.com/programs/pv.shtml

9) Clone IDE Hard Disk

sudo dd if=/dev/hda1 of=/dev/hdb2

This command clone the first partition of the primary master IDE drive to the second partition

of the primary slave IDE drive (!!! back up all data before trying anything like this !!!)

10) Test network speed without wasting disk

dd if=/dev/zero bs=4096 count=1048576 | ssh user@host.tld 'cat > /dev/null'

The above command will send 4GB of data from one host to the next over the network, without consuming any unnecessary disk on either the client nor the host. This is a quick and dirty way to benchmark network speed without wasting any time or disk space.

Of course, change the byte size and count as necessary.

This command also doesn’t rely on any extra 3rd party utilities, as dd, ssh, cat, /dev/zero and /dev/null are installed on all major Unix-like operating systems.

11) clone a hard drive to a remote directory via ssh tunnel, and compressing the image

dd if=/dev/sda | gzip -c | ssh user@ip 'dd of=/mnt/backups/sda.dd'