commoncrawl

• looking for a domain:
• http://index.commoncrawl.org/
• enter:   http://www.cnn.com/*
• download from:
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-36/segments/1471982292181.27/warc/CC-MAIN-20160823195812-00279-ip-10-153-172-175.ec2.internal.warc.gz
• list of segments
• https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-40/warc.paths.gz

gatling stress tests

http://gatling.io/

Download Url  bundle Gatling :
https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/2.2.2/gatling-charts-highcharts-bundle-2.2.2-bundle.zip

Scala IDE for Eclipse:
http://scala-ide.org/download/sdk.html


Typical class (where title column in the csv is "Keyword"):

package com.toto.backend.scenarios.ww

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
import io.gatling.http.protocol.HttpProtocolBuilder.toHttpProtocol
import io.gatling.http.request.builder.HttpRequestBuilder.toActionBuilder

class WWBasicScenario extends Simulation {
  val ts = csv("ww/kw_utf_1M.csv").random

  val httpConf = http
    .baseURL("http://bench.preprod.toto.com:8080") // Here is the root for all relative URLs
    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") // Here are the common headers
    .doNotTrackHeader("1")
    .acceptLanguageHeader("en-US,en;q=0.5")
    .acceptEncodingHeader("gzip, deflate")
    .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0")

  val headers_10 = Map("Content-Type" -> "application/x-www-form-urlencoded") // Note the headers specific to a given request

    val scn = scenario("Serve random kw")
    .feed(ts)
    .exec(
      http("...")
        .get("/ww/api/1.0/offer/search/lang/fr/site-id/1?words=${Keyword}&rows=5&fac_cat=true")
        .check(status.is(200)))

     setUp(scn.inject(constantUsersPerSec(50) during(6 hours)).protocols(httpConf))}

yum

• if you generate 1.0.0.rc1, the version 1.0.0 is considered as older. You can still list all versions:

yum info  --showduplicates  datamining-something

ssh

• avoid automatic time outs on sessions
• GSSAPIAuthentication no
• GSSAPIDelegateCredentials no
• ServerAliveInterval 60

• to run something on different machines:
• for i in `seq 17 19`; do ssh xml$i sed -i '"s/boost_clicks=0.0/boost_clicks=10/"' my-search/depot/search/Whatever/etc/*/rank.conf ; done
• 
  
for i in 2 3 4 7 8 9 10 11 12 13 14 15 16
do
echo xml$i
ssh toto@xml$i cat /etc/debian_version
done

• ssh users are Ubuntu users. There are no specific ssh users

• to do ssh without entering a pw

• ssh-copy-id user@server

Python data types

------------------------ built in types

-- tuples (immutables)
> empty = ()
> t = 12345, 54321, 'hello!'
or
> t = (12345, 54321, 'hello!')
> t
(12345, 54321, 'hello!')

-- set

> a=set()

> a.add("orange")

> a

{'orange'}

> basket = {'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}      # creates a set without duplicates
> basket.add("'orange")
> basket
{'orange', 'pear', 'apple', 'banana'}

>

> my_data = {0,0}

> my_data

{0}

-- list  (might contain different types, duplicates. It is mutable)

z_list = []
my_list = ["apple", "banana", "cherry"]
my_list.append("pear")
a_list=[0]
a_list = [66.25, 333, 333, 1, 1234.5]
a_list = ['foo', 'bar', 'bar']
a_list = [[ 4.9,  3. ,  1.4,  0.2], [ 4.9,  3. ,  1.4,  0.2]]

-- dictionaries (have keys)

> my_dict = {}

> d={0:0}

> d

{0: 0}

tel = {'jack': 4098, 'sape': 4139}
tel['john'] = 4444.  # add
print (tel.keys())         # returns a list

  dict_keys(['jack', 'sape', 'john'])

print (tel.values())      # returns a list

  dict_values([4098, 4139, 4444])

https://docs.python.org/3.6/library/datatypes.html

Use
>>> globals()

to see all variables.

Joel Test score

The Joel Test is a twelve-question measure of the quality of a software team.

• Do you use source control?
• Can you make a build in one step?
• Do you make daily builds?
• Do you have a bug database?
• Do you fix bugs before writing new code?
• Do you have an up-to-date schedule?
• Do you have a spec?
• Do programmers have quiet working conditions?
• Do you use the best tools money can buy?
• Do you have testers?
• Do new candidates write code during their interview?
• Do you do hallway usability testing?

regex

^ Matches the starting position within the string
$ Matches the ending position within the string


^10[1-9]{6}$    ("10" followed by 6 digits)
^10.{6}$           ("10" followed by any 6 characters)

• mystring/   followed by two or three digits
• sed -r 's/.*mystring\/([0-9]{2,3}).*/\1/g'

.