tools

• https://builtwith.com/cnd.fr     (what is this site built with ?)
• 
  

dataiku

-- docker
docker pull jbfavre/dataiku-dss:latest
docker run -p 10000:10000 -i -v /home/dev/docker-dataiku-dss-data:/var/lib/dataiku -t jbfavre/dataiku-dss:latest

-- doc
https://www.dataiku.com/learn/guide/tutorials/101.html
https://doc.dataiku.com/dss/latest/preparation/processors/index.html
https://doc.dataiku.com/dss/2.3/preparation/geo_processors.html#resolve-geoip

-- get 10K row limit
http://answers.dataiku.com/663/getting-10-000-row-limit

http://127.0.0.1:10000/projects/DKU_TUTORIAL101STARTINGPROJECT/datasets/out-few-col-1M_prepared/explore/
-- maps
http://127.0.0.1:10000/projects/DKU_TUTORIAL101STARTINGPROJECT/analysis/wZlvCIKS/charts/
-- stat
http://127.0.0.1:10000/projects/DKU_TUTORIAL101STARTINGPROJECT/datasets/out_simplified/visualize/

-- used by Dataiku for geoloc
https://www.maxmind.com/en/home

USB external drive

If you have an USB external disk:

• if it is FAT: 2GB max file transfer
• NTFS: don"t support unix privileges
• EXT4: cannot see it from Windows

Python

>>> import form_utils
>>> form_utils.__file__
'/data1/django/lib/python2.6/site-packages/form_utils/__init__.pyc'

>>> globals()

{'__builtins__': , 'functools': , '__package__': None, '__name__': '__main__', 'os': , '__doc__': None}

print(globals())

- commands
    find . -name \*.pyc -delete

- run simple code from the command line

   python3 -c 'import apache_beam; print(apache_beam.__version__)'

------  python3
apt-get install python3

warning: don't change symbolic links in /usr/bin (/usr/bin/python and /usr/bin/python-config). Otherwise apt-get and other scripts don't work anymore.
Rather, add an alias in your .bashrc

alias python=python3

Django

• version
• python 

Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2Type "help", "copyright", "credits" or "license" for more information.

> import django

> django.VERSION

(1, 8, 0, 'final', 0)

• create a new site
• “django-admin.py startproject mysite”   creates a mysite directory in your current directory.
• templates
• Django template language uses  {% %} in the html
• {% if results %}
• logs
• 
  
•     logger = logging.getLogger('my-django-logs')
•     logger.info('________________________ (3)')
• LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'filters': { 'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse' } }, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler' }, 'applogfile': { 'level':'DEBUG', 'class':'logging.handlers.RotatingFileHandler', 'filename': os.path.join('/tmp', 'my-django-logs.log'), 'maxBytes': 1024*1024*15, # 15MB 'backupCount': 10, } }, 'loggers': { 'django.request': { 'handlers': ['mail_admins'], 'level': 'ERROR', 'propagate': True, }, 'my-django-logs': { 'handlers': ['applogfile'], 'level': 'DEBUG', } } }
• clearsessions
• python ./manage.py clearsessions (cleans up django_session in MySQL)

Some Web Servers and Python Web Frameworks

• Web Servers (not Python) used to serve Python web frameworks data
• Apache HTTP server with mod_wsgi
• Nginx
• WSGI HTTP Server
• http://gunicorn.org/ WSGI HTTP Server
• http://serverfault.com/questions/331256/why-do-i-need-nginx-and-something-like-gunicorn

• Python Web Frameworks
• django  ("manage.py runserver" runs a WSGI application for development purposes)
• flask    http://flask.pocoo.org/     (Restfull microframework)
• Zope
• wsgi
• Client connections that Apache receives will be translated into the WSGI format that the Django application expects using the mod_wsgi

commoncrawl

• looking for a domain:
• http://index.commoncrawl.org/
• enter:   http://www.cnn.com/*
• download from:
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-36/segments/1471982292181.27/warc/CC-MAIN-20160823195812-00279-ip-10-153-172-175.ec2.internal.warc.gz
• list of segments
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-40/warc.paths.gz