bash survival kit

• Useful commands 
  - disk activity per process and thread, IO monitor, disk access including swapping
  - iotop -o
  - lsof (List files/network connections open by user)
  --lsof -i:80 (processes listening on port 80) - see also lsof description 
  - dmesg |less
  - uname -a
  - du -sh * (Folders size)
  - du -sh * | sort -nr | head -10 (biggest subfolders)
  - ls -l | grep '^d' (list only directories)
  - watch 'ps aux | grep someprocess' (execute periodically a command)
  - ssh -X toto@machine (enables X11)
  - ajust time and start ntp date tracking:
  - - /etc/init.d/ntp stop
  - - ntpdate 0.debian.pool.ntp.org (synchronize computers time)
  - - /etc/init.d/ntp start
  - - see /etc/ntp.conf - to kill all XServer sessions: "sudo killall Xorg"
  - tunnel: ssh -f -L 10.193.129.2:9090:192.168.0.202:8080 192.168.50.99
  - sudo fuser -v 12001/tcp    (check which process/user is using a port)
  - nslookup: get the name from a remote machine from its IP
  - check if  a remote machine port is open:
  -- wget -qS -O- http://csimg.toto.me:80
  -- curl http://csimg.toto.me:80 

• route (to see the gateway)
• open ports
• netstat -ltu   (local)
• nmap (not showing all open ports, can be remote)
• sudo nmap -sU  -p 500 192.168.1.254   (check internet key exchange port open, used for vpn)
• most recent java files in the tree

• find . |grep "\.java" |xargs ls -l|awk -F$' ' '{print $6 " " $7 " " $9}' |sort
• swap per process
• for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done

• find
• find . -name pom.xml | xargs grep 3.5 --color      (emacs/dired to replace)
• find . -name '*.log.*' -delete 
• find . | xargs wc (count the number of files in a folder and sub-directories)

• grep
• grep -o OTHERS tb-201610.sql |wc  (new line)
• grep -P "\t8" file1.txt |awk -F$'\t' '{print $1}' |while read -r x; do grep "$x" file2.txt; done
• look for multiple values
• egrep "126|127|128|129|130|131|132|133" toto.txt
• System 
• sudo bash (remain as root)
• uname -a
• less /proc/meminfo
• less /proc/cpuinfo
• lspci -v | less (list all PCI devices connected to the PCI bus)
• dmesg | less (kernel boot messages)
• ls -1R | wc -l (number of files in a folder)
• fdisk -l (disks on a machine)
• dmidecode (physical processor info)
• ps auxf  (process tree), 
• ps -efj   (ppid)
• ppid:  use htop F2
• uptime   (when did the machine restarted)
• DNS
• dig cnn.com

• load CPU:
for i in {1..1000000}; do gzip speetest;gunzip speetest.gz ;done
.
for f in $(find . -name "*.db" | grep 'rep_13' | grep run | xargs ls -1); do echo $f;done .

.

awk survival kit

look for a string in the 28th colon in a tsv:

awk -F$'\t' '{if($28 == "popcorn") print $0;}' toto.tsv
awk -F$'\t' '{print $18}' wm_marchand_Data.txt | sort -u
awk -F$'\t' '{print $2 "\t" $1}' sb.txt

for special caracters:
iconv -f ISO-8859-1 -t utf8 ri_thesaurus_v2_Data_LG161___.txt > ri_thesaurus_v2_Data_LG161____.txt
iconv -f utf8 -t ISO-8859-1//TRANSLIT lbm-ko-full-201806.csv > lbm-ko-full-201806-ISO8859.csv

• keep numbers after = in the output

awk -F$' ' '{print $11}' noc.log  | sed -r 's/.*=([0-9]+)/\1/g'

• keep only numbers

awk -F':' '{print $3}'  1300-siren.txt | sed -r 's/[^0-9]//g'

• sort by full number

awk -F$' ' '{print $11}' noc.log  |sed -r 's/.*=([0-9]+)/\1/g' | sort -k1,1n > noc-out.log

• grep -P "\t8" file1.txt |awk -F$'\t' '{print $1}' |while read -r x; do grep "$x" file2.txt; done

• to find out swap

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done| awk -F$' ' '{print $2}'  |sort  -k1,1n

.
.