USB external drive

If you have an USB external disk:

• if it is FAT: 2GB max file transfer
• NTFS: don"t support unix privileges
• EXT4: cannot see it from Windows

Python

>>> import form_utils
>>> form_utils.__file__
'/data1/django/lib/python2.6/site-packages/form_utils/__init__.pyc'

>>> globals()

{'__builtins__': , 'functools': , '__package__': None, '__name__': '__main__', 'os': , '__doc__': None}

print(globals())

- commands
    find . -name \*.pyc -delete

- run simple code from the command line

   python3 -c 'import apache_beam; print(apache_beam.__version__)'

------  python3
apt-get install python3

warning: don't change symbolic links in /usr/bin (/usr/bin/python and /usr/bin/python-config). Otherwise apt-get and other scripts don't work anymore.
Rather, add an alias in your .bashrc

alias python=python3

Django

• version
• python 

Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2Type "help", "copyright", "credits" or "license" for more information.

> import django

> django.VERSION

(1, 8, 0, 'final', 0)

• create a new site
• “django-admin.py startproject mysite”   creates a mysite directory in your current directory.
• templates
• Django template language uses  {% %} in the html
• {% if results %}
• logs
• 
  
•     logger = logging.getLogger('my-django-logs')
•     logger.info('________________________ (3)')
• LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'filters': { 'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse' } }, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler' }, 'applogfile': { 'level':'DEBUG', 'class':'logging.handlers.RotatingFileHandler', 'filename': os.path.join('/tmp', 'my-django-logs.log'), 'maxBytes': 1024*1024*15, # 15MB 'backupCount': 10, } }, 'loggers': { 'django.request': { 'handlers': ['mail_admins'], 'level': 'ERROR', 'propagate': True, }, 'my-django-logs': { 'handlers': ['applogfile'], 'level': 'DEBUG', } } }
• clearsessions
• python ./manage.py clearsessions (cleans up django_session in MySQL)

Some Web Servers and Python Web Frameworks

• Web Servers (not Python) used to serve Python web frameworks data
• Apache HTTP server with mod_wsgi
• Nginx
• WSGI HTTP Server
• http://gunicorn.org/ WSGI HTTP Server
• http://serverfault.com/questions/331256/why-do-i-need-nginx-and-something-like-gunicorn

• Python Web Frameworks
• django  ("manage.py runserver" runs a WSGI application for development purposes)
• flask    http://flask.pocoo.org/     (Restfull microframework)
• Zope
• wsgi
• Client connections that Apache receives will be translated into the WSGI format that the Django application expects using the mod_wsgi

commoncrawl

• looking for a domain:
• http://index.commoncrawl.org/
• enter:   http://www.cnn.com/*
• download from:
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-36/segments/1471982292181.27/warc/CC-MAIN-20160823195812-00279-ip-10-153-172-175.ec2.internal.warc.gz
• list of segments
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-40/warc.paths.gz

gatling stress tests

http://gatling.io/

Download Url  bundle Gatling :
https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.2.2/gatling-charts-highcharts-bundle-2.2.2-bundle.zip

Scala IDE for Eclipse:
http://scala-ide.org/download/sdk.html


Typical class (where title column in the csv is "Keyword"):

package com.toto.backend.scenarios.ww

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
import io.gatling.http.protocol.HttpProtocolBuilder.toHttpProtocol
import io.gatling.http.request.builder.HttpRequestBuilder.toActionBuilder

class WWBasicScenario extends Simulation {
  val ts = csv("ww/kw_utf_1M.csv").random

  val httpConf = http
    .baseURL("http://bench.preprod.toto.com:8080") // Here is the root for all relative URLs
    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") // Here are the common headers
    .doNotTrackHeader("1")
    .acceptLanguageHeader("en-US,en;q=0.5")
    .acceptEncodingHeader("gzip, deflate")
    .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0")

  val headers_10 = Map("Content-Type" -> "application/x-www-form-urlencoded") // Note the headers specific to a given request

    val scn = scenario("Serve random kw")
    .feed(ts)
    .exec(
      http("...")
        .get("/ww/api/1.0/offer/search/lang/fr/site-id/1?words=${Keyword}&rows=5&fac_cat=true")
        .check(status.is(200)))

     setUp(scn.inject(constantUsersPerSec(50) during(6 hours)).protocols(httpConf))}

yum

• if you generate 1.0.0.rc1, the version 1.0.0 is considered as older. You can still list all versions:

yum info  --showduplicates  datamining-something