Python

>>> import form_utils
>>> form_utils.__file__
'/data1/django/lib/python2.6/site-packages/form_utils/__init__.pyc'

>>> globals()

{'__builtins__': , 'functools': , '__package__': None, '__name__': '__main__', 'os': , '__doc__': None}

print(globals())

find . -name \*.pyc -delete

------  python3
apt-get install python3

warning: don't change symbolic links in /usr/bin (/usr/bin/python and /usr/bin/python-config). Otherwise apt-get and other scripts don't work anymore.
Rather, add an alias in your .bashrc

alias python=python3

Django

• version
• python 

Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2Type "help", "copyright", "credits" or "license" for more information.

> import django

> django.VERSION

(1, 8, 0, 'final', 0)

• create a new site
• “django-admin.py startproject mysite”   creates a mysite directory in your current directory.
• templates
• Django template language uses  {% %} in the html
• {% if results %}
• logs
• 
  
•     logger = logging.getLogger('my-django-logs')
•     logger.info('________________________ (3)')
• LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'filters': { 'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse' } }, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler' }, 'applogfile': { 'level':'DEBUG', 'class':'logging.handlers.RotatingFileHandler', 'filename': os.path.join('/tmp', 'my-django-logs.log'), 'maxBytes': 1024*1024*15, # 15MB 'backupCount': 10, } }, 'loggers': { 'django.request': { 'handlers': ['mail_admins'], 'level': 'ERROR', 'propagate': True, }, 'my-django-logs': { 'handlers': ['applogfile'], 'level': 'DEBUG', } } }
• clearsessions
• python ./manage.py clearsessions (cleans up django_session in MySQL)

Some Web Servers and Python Web Frameworks

• Web Servers (not Python) used to serve Python web frameworks data
• Apache HTTP server with mod_wsgi
• Nginx
• WSGI HTTP Server
• http://gunicorn.org/ WSGI HTTP Server
• http://serverfault.com/questions/331256/why-do-i-need-nginx-and-something-like-gunicorn

• Python Web Frameworks
• django  ("manage.py runserver" runs a WSGI application for development purposes)
• flask    http://flask.pocoo.org/     (Restfull microframework)
• Zope
• wsgi
• Client connections that Apache receives will be translated into the WSGI format that the Django application expects using the mod_wsgi

commoncrawl

• looking for a domain:
• http://index.commoncrawl.org/
• enter:   http://www.cnn.com/*
• download from:
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-36/segments/1471982292181.27/warc/CC-MAIN-20160823195812-00279-ip-10-153-172-175.ec2.internal.warc.gz
• list of segments
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-40/warc.paths.gz

gatling stress tests

http://gatling.io/

Download Url  bundle Gatling :
https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.2.2/gatling-charts-highcharts-bundle-2.2.2-bundle.zip

Scala IDE for Eclipse:
http://scala-ide.org/download/sdk.html


Typical class (where title column in the csv is "Keyword"):

package com.toto.backend.scenarios.ww

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
import io.gatling.http.protocol.HttpProtocolBuilder.toHttpProtocol
import io.gatling.http.request.builder.HttpRequestBuilder.toActionBuilder

class WWBasicScenario extends Simulation {
  val ts = csv("ww/kw_utf_1M.csv").random

  val httpConf = http
    .baseURL("http://bench.preprod.toto.com:8080") // Here is the root for all relative URLs
    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") // Here are the common headers
    .doNotTrackHeader("1")
    .acceptLanguageHeader("en-US,en;q=0.5")
    .acceptEncodingHeader("gzip, deflate")
    .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0")

  val headers_10 = Map("Content-Type" -> "application/x-www-form-urlencoded") // Note the headers specific to a given request

    val scn = scenario("Serve random kw")
    .feed(ts)
    .exec(
      http("...")
        .get("/ww/api/1.0/offer/search/lang/fr/site-id/1?words=${Keyword}&rows=5&fac_cat=true")
        .check(status.is(200)))

     setUp(scn.inject(constantUsersPerSec(50) during(6 hours)).protocols(httpConf))}

yum

• if you generate 1.0.0.rc1, the version 1.0.0 is considered as older. You can still list all versions:

yum info  --showduplicates  datamining-something

ssh

• avoid automatic time outs on sessions
• GSSAPIAuthentication no
• GSSAPIDelegateCredentials no
• ServerAliveInterval 60

• to run something on different machines:
• for i in `seq 17 19`; do ssh xml$i sed -i '"s/boost_clicks=0.0/boost_clicks=10/"' my-search/depot/search/Whatever/etc/*/rank.conf ; done
• 
  
for i in 2 3 4 7 8 9 10 11 12 13 14 15 16
do
echo xml$i
ssh toto@xml$i cat /etc/debian_version
done

• ssh users are Ubuntu users. There are no specific ssh users

• to do ssh without entering a pw

• ssh-copy-id user@server

Python data types

------------------------ built in types

-- tuples (immutables)
> empty = ()
> t = 12345, 54321, 'hello!'
or
> t = (12345, 54321, 'hello!')
> t
(12345, 54321, 'hello!')

-- set

> a=set()

> a.add("orange")

> a

{'orange'}

> basket = {'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}      # creates a set without duplicates
> basket.add("'orange")
> basket
{'orange', 'pear', 'apple', 'banana'}

>

> my_data = {0,0}

> my_data

{0}

-- list  (might contain different types, duplicates. It is mutable)

z_list = []
my_list = ["apple", "banana", "cherry"]
my_list.append("pear")
a_list=[0]
a_list = [66.25, 333, 333, 1, 1234.5]
a_list = ['foo', 'bar', 'bar']
a_list = [[ 4.9,  3. ,  1.4,  0.2], [ 4.9,  3. ,  1.4,  0.2]]

-- dictionaries (have keys)

> my_dict = {}

> d={0:0}

> d

{0: 0}

tel = {'jack': 4098, 'sape': 4139}
tel['john'] = 4444.  # add
print (tel.keys())         # returns a list

  dict_keys(['jack', 'sape', 'john'])

print (tel.values())      # returns a list

  dict_values([4098, 4139, 4444])

https://docs.python.org/3.6/library/datatypes.html

Use
>>> globals()

to see all variables.

Joel Test score

The Joel Test is a twelve-question measure of the quality of a software team.

• Do you use source control?
• Can you make a build in one step?
• Do you make daily builds?
• Do you have a bug database?
• Do you fix bugs before writing new code?
• Do you have an up-to-date schedule?
• Do you have a spec?
• Do programmers have quiet working conditions?
• Do you use the best tools money can buy?
• Do you have testers?
• Do new candidates write code during their interview?
• Do you do hallway usability testing?

regex

^ Matches the starting position within the string
$ Matches the ending position within the string


^10[1-9]{6}$    ("10" followed by 6 digits)
^10.{6}$           ("10" followed by any 6 characters)

• mystring/   followed by two or three digits
• sed -r 's/.*mystring\/([0-9]{2,3}).*/\1/g'

.

apache nutch

apache-nutch-1.11
--------------------------- one shot
./bin/nutch parsechecker -dumpText http://www.deco.com > out.txt
./bin/nutch parsechecker -dumpText http://www.deco.com/cuisine/vos-visites |grep tc

wget -O - 'http://www.deco.com/cuisine/vos-visites' |grep tc_vars.http

--------------------------- crawl
./bin/crawl urls/seeds.txt out-crawl 5
(look at what the crawl script does, it is basically a loop calling bin/nutch)

./bin/nutch fetch 5-out-crawl/segments/20160717060128/ -noParsing  (the one did in crawl)

--------------------------- read /crawl db
./bin/nutch readdb out-crawl/crawldb -stats
./bin/nutch readdb 4-out-crawl/crawldb -dump 4-out-crawl-db-dump-20160512 -format csv -topN 1000

--------------------------- read /segments
./bin/nutch readseg -dump 4-out-crawl/segments/20160716234706 4-out-crawl-readseg-20160716234706
grep -ir outlink 3-out-crawl-readseg  --color  |wc

--------------------------- read one key from  a segment
./bin/nutch readseg -get 4-out-crawl/segments/20160717025923  "http://www.moulinex.fr/Cuisson/Croques-%26-Gaufres/c/waffle%2Bmakers"  -nocontent -noparse -noparsetext

--------------------------- read HTML from one key from  a segment
./bin/nutch readseg -get 2-out-crawl-wiki-nikon-num-round-2/segments/20160823085007/  "https://en.wikipedia.org/wiki/Nikon" -nofetch -nogenerate -noparse -noparsedata -noparsetext > Nikon.html

--------------------------- doc
https://wiki.apache.org/nutch/CommandLineOptions

--------------------------- plugin
bin/nutch plugin   parse-html   org.apache.nutch.parse.html.HtmlParser   index2.htm
https://wiki.apache.org/nutch/bin/nutch%20plugin

.
.

docker survival kit

on Ubuntu 14.04:

-------------- docker service itself

service docker start

-------------- install

sudo bash
vi /etc/apt/sources.list.d/docker.list
deb https://apt.dockerproject.org/repo ubuntu-trusty main

apt-get update
you may have to do this with the associated keys if the former doesn't work
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F76221572C52609D

apt-get -y install docker.io
apt-get install docker-engine
(don't do    "apt-get install docker")

ln -sf /usr/bin/docker /usr/local/bin/docker

sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker.io
or
sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker
(depends which works)

• start docker daemon
• $ sudo service docker start

docker -v    (should be at least 1.12)
docker version
docker info
docker pull hello-world
docker run -i -t  hello-world

docker pull centos:6.6
docker run -i -t centos:6.6 /bin/bash

sudo docker images -a

------------- Main

• Run but exits right away with an error. Still, creates a container.
• docker run ubuntu
• Run and keeps an interactive session
• docker run -it ubuntu /bin/bash
• docker run -it alpine /bin/sh
• sudo docker run -i -t {repository_name}:{tag} /bin/bash
• Execute in a running container:
• “docker exec -i -t 7b33f4ed03a7 /bin/bash”
• sudo docker exec -i -t $(sudo docker ps -l -q) env
• Restart an existing container to continue working in it
• docker start 912ccdad8380
• then docker exec -it 912ccdad8380 /bin/bash

• To build a container from a Dockerfile:
• sudo docker build {Dockerfile_path}

------------- 

docker rename 7d8b244480a2 my_django____4         (rename a container)
docker commit 7d8b244480a2  bob/hello_django:version2  (commit a new image from a running container)

docker save -o /home/bob/bob-hello-django-version4.tar  bob/hello_django:version2

docker load -i bob-hello-django-version2.tar

-------------

- "ls /etc/*release"  to know the container OS.
- remove all containers and clear cache (do not remove images...):
-- docker rm $(docker ps -a -q) && docker ps -a | cut -c-12 | xargs docker rm

• remove a specific image 
• sudo docker rmi -f 4f8eceb5cb78
• ignore container entrypoint:
• docker run -it --entrypoint=/bin/bash michal/hello_django_srp:version5

------------- problem 1

Docker version 1.6.2, build 7c8fca2

> docker save -o /tmp/docker1.tar  michal/hello_django_srp:version6

FATA[0000] Error response from daemon: could not find image: no such id:  michal/hello_django_srp:version6 

--> there are two spaces before "michal" and docker is not robust to this

------------- static IP address

docker network create --subnet=172.18.0.0/16 mynet18 docker run --net mynet18 --ip 172.18.0.22 -it ubuntu:14.04 bash

------------- bind volume mount (you need data from the host machine)

docker run -it -v /home/lee/tmp/docker-bind:/opt/container-path bob/solr_6_6:v7

otherwise, you can simply copy from the host to the container:

docker cp ./syn_acc.txt dfccf5a493c3:/usr/share/elasticsearch/config/resources/v1_02/

------------- other commands

docker inspect 7832d56ddf56

docker run -it -p 8580:8080 bob/my-jetty:v1 (port matching)

.
.

strange colors on vlc / Windows 7 / need nvidia driver

http://fr.download.nvidia.com/Windows/341.92/341.92-notebook-win8-win7-64bit-international.exe

Debugging Tomcat remotely using JMX

Start the tomcat as follows
./start.sh -d -p 8001 -j -m 9001 -h 10.2.14.2

From Eclipse:
  - debug configurations
  - "Remote Java application"
  - "connect" tab
    . Host running the tomcat
    . Port: 8001

Java Web Services

• Specifications
• JAX-RS    (Java API for RESTful Web Services)
• implementations:
• Jersey
• Apache CXF
• RESTEasy (JBoss implementation)
• JAX-WS     (Java API for XML Web Services)
• implementations:
• part of the J2EE platform
• Apache CXF
• Axis2
• based on SOAP
• Implementations
• Jersey
• https://jersey.java.net/
• only JAX-RS  support (RESTful)
• WS framework
• ServletContainer
• Apache CXF
• https://cxf.apache.org/
• WS framework
• JAX-WS and  JAX-RS  front end APIs support
• mainly SOAP: good for client-side WSDL services
• can speak with SOAP, XML/HTTP, RESTful HTTP

• Java  application servers and application frameworks
• http://swiss-knife.blogspot.fr/search?q=application+framework

PostgreSQL / Postgres / Postgre

lg@lgstage11:~$ psql -U user -p 5432 mydb

mydb=> \dt

 Liste des relations
 Schéma |       Nom       | Type  | Propriétaire
--------+-----------------+-------+--------------
 public | shop_c           | table | user
 public | shop_ean       | table | user
(2 lignes)

mydb=> \x      -> formatting

select * from shop_e limit 10;

service postgresql stop
service postgresql start

psql -U user --password
if this complains that the user doesn't have a database, you can do it another way:
postgre@machine:  createuser search
postgre@machine:  createdb search

\copy (select lt.image.lt_kimage_id from lt.image LIMIT 100000) To '/tmp/test.csv';

DELETE FROM image WHERE image_id IN (select image.image_id from image INNER JOIN keyword ON image_id = keyword.image_id WHERE keyword.keyword LIKE '%_dead' ORDER BY image_id LIMIT 200000);

-- if "psql" doesn't work, do this first:

    su - postgres

 https://gist.github.com/Kartones/dd3ff5ec5ea238d4c546
.
.