Thursday, July 7, 2011

Scalability concepts

  • Google BigTable
    http://labs.google.com/papers/bigtable.html

  • BigData : are datasets that grow so large that they become awkward to work with
    http://en.wikipedia.org/wiki/Big_data

  • Hadoop

  • MapReduce : Google framework

  • Dynamo: Amazon's Highly Available Key-value Store
    http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf


  • - Master / slave
    - peer to peer
    - cluster: similar components
    - grid: different components
  • Wednesday, July 6, 2011

    aptitude search result

    p, meaning that no trace of the package exists on the system,
    c, meaning that the package was deleted but its configuration files remain on the system,
    i, meaning that the package is installed,
    v, meaning that the package is virtual.

    The second character indicates the stored action (if any; otherwise a blank space is displayed) to be performed on the package
    i, meaning that the package will be installed,
    d, meaning that the package will be deleted,
    p, meaning that the package and its configuration files will be removed.

    If the third character is A, the package was automatically installed

    Monday, May 23, 2011

    JNI

    I had to do:

    cd /usr/lib
    ln -s /usr/local/jdk1.6.0_22/jre/lib/amd64/libjava.so .

    to use -ljava

    See also   swig

    Thursday, April 28, 2011

    SEO

    SEO: Search Engine Optimization
    SMO: Social Media Optimization: Methodization of social media activity with the intent of attracting unique visitors to website content (Facebook button "I like")
    SEM: Search Engine Marketting: through the use of, paid placement, contextual advertising, and paid inclusion

    Monday, March 28, 2011

    gnuplot

    run gnuplot:

    gnuplot> set timefmt "%H:%M:%S"
    gnuplot> set xdata time
    gnuplot> set logscale y
    gnuplot> plot "day_poll.txt" using 2:1 title ' my title '

    gnuplot> plot "<awk '{i=i+$2; print $1,i}' clicks_.txt" title ' clicks'
    gnuplot> plot "<awk '{i=i+$2; print $1,i}' out-log-final.txt" title "LG", "<awk '{i=i+$2; print $1,i}' clicks_.txt" title "ozen"
    .

    Saturday, March 19, 2011

    freenx / qtnx

  • on the server:
      Follow these instructions: FreeNX
    • To see if nxserver processes were started after starting a client
    • watch 'ps aux |grep nx'
    • To kill all server nx sessions (they will be restarted automatically at the next connection)
    • sudo nxserver --help
      sudo nxserver --history
      sudo nxserver --terminate username
    • Other things:
    • sudo su - nx
    • keys problems
      when setting up a new server machine. The following worked for me in the case where you have "Authenticating client" forever on your qtnx client
      root@aaa:/var/log# /usr/lib/nx/nxsetup --install --clean --purge
  • on the client:
      install QTNX

      If the graphical Qtnx GUI client does not show up, kill all qtnx process running on the client machine and run the qtnx client again

      Make sure you set "gnome" (if you use gnome) and "adsl" in the QTNX config
      .
  • Friday, February 4, 2011

    gcc / g++

  • to see disassembled code in an object / library:
    objdump --source libSearch.a |grep ';'

    .
  • Thursday, January 13, 2011

    Solr


  • index csv
    curl "http://localhost:8080/solr/fnac/update/csv?commit=true&separator=%7c&header=false&fieldnames=id,reference,ean13,title,desc,mark,cat,,,,,,,,,,,,,,,,,,,,,," --data-binary @marchand.csv -H 'Content-type:text/plain; charset=iso-8859-1'

  • to delete all
    http://localhost:8080/solr/fnac/update?commit=true&stream.body=%3Cdelete%3E%3Cquery%3E*%3A*%3C%2Fquery%3E%3C%2Fdelete%3E

  • remove items where price=299.00
    http://192.168.0.70:8080/solr/lg_fr_small/update?commit=true&stream.body=%3Cdelete%3E%3Cquery%3Eprice%3A299.0%3C%2Fquery%3E%3C%2Fdelete%3E

  • usefull paths
    /var/lib/tomcat6/conf
    /etc/tomcat6/solr/fnac/conf
    /var/data/solr

  • logging
    /etc/tomcat6/logging.properties
    set the following
    org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO

  • compoundFiles: when set to false, will not authorize to have a lot of file descritpors open, so indexation is slower (you have to do it when you start to have too many open files problems)

  • query syntax
    "jouet cubes"~2 (ecarts)
    "(desc:bois)^10000 jouets" (boost)
    mobile && _val_:"popularity"^0.7
    q.op=AND df=text
    Using quoted means "these tokens in exactly this order", not "use this as a literal" (stemming implicitly disable exact search)
    &debugQuery=true
    &fl=offer_event,score: "field list" to be displayed in the output
  • query field mapping with edismax




  • /solr/nl_00/select?q=cat2:5010000&wt=xml&f.cat2.qf=cat&defType=edismax



  • Filter queries
    - fields used by the filter need to be indexed (everything used by Solr need to be indexed, expept what is used only for result feed presentation)
    - q=mobile&fq=popularity:[10 TO *] (filter query result set)
    - fq={!cache=false}year:[2005 TO *] (disable filter cache)

  • Function queries
    - boost scores: either
    . . _val_ in the query
    . . bf (boost function) parameter, or extended dismax boost parameter (multiplicative boost). 





  • stemming and exact search 
  • from nabble A phrase means to solr (or rather to the lucene and dismax query parsers, which are what understand double-quoted phrases) "these tokens in exactly this order" So a phrase of one token "manager", is exactly the same as if you didn't use the double quotes. It's only one token, so "all the tokens in this phrase in exactly the order specified" is, well, just the same as one token without phrase quotes.
    If you've set up a stemmed field at indexing time, then "manager" and "management" are stemmed IN THE INDEX, probably to something like "manag". There is no longer any information in the index (at least in that field) on what the original literal was, it's been stemmed in the index. So there's no way possible for it to only match certain un-stemmed versions -- at least using that field. And when you enter either 'manager' or 'management' at query time, it is analyzed and stemmed to match that stemmed something-like "manag" in the index either way. If it didn't analyze and stem at query time, then instead the query would just match NOTHING, because neither 'manager' nor 'management' are in the index at all, only the stemmed versions. 




  • Span queries and Payloads




  • Span queries are like a "near" query: tokens emitted during the analysis include the position of  the previous token



  • Payloads are a byte array that allow to tag infos at index time.




  •  Local params
    q={!q.op=AND df=title}solr rocks
    q={!type=dismax qf='myfield yourfield'}solr rocks
    q={!dismax qf=myfield}solr rocks ---(implies: type:dismax)
    q={!dismax}solr rocks --- is equivalent to --- q={!type=dismax qf=myfield v=$qq}&qq=solr rocks --- (parameter dereferencing)
    q={!geodist}
    q={!cache}
    q={!tag=foo} --- tag is a local param to arbitrarily label a parameter
    q={!lucene q.op=AND df=text}myfield:foo +bar -baz --- specifies a lucene/solr query

  • facets
    • number of results by type for a specific field
    • field faceted must be indexed
    • basic facet
      http://localhost:8080/solr/append/select?q=mobile&facet=true&facet.limit=10&facet.field=brand
    • simple facet query: new query on the result set (not really a facet) : how many from shop 164
      http://localhost:8080/solr/append/select?q=*:*&start=0&rows=10&facet=true&facet.query=shop:164
    • facets for automcompletion with "mo"
      http://192.168.0.1:8080/solr/append/select?q=mobile&facet=on&facet.limit=10&facet.mincount=1&facet.field=title&facet.prefix=mo
    • facets by fixed interval method 1 (facet.range)
      http://192.168.0.1:8080/solr/append/select?indent=on&version=2.2&q=mobile%0D%0A&fq=&start=0&rows=10&fl=*%2Cscore&facet=true&facet.range=price&facet.range.start=0&facet.range.end=1000&facet.range.gap=100
    • facets by fixed interval method 2 (facet.query)
      http://localhost:8080/solr/select?q=video&rows=0&facet=true&facet.query=price:[*+TO+500]&facet.query=price:[500+TO+*]
    • facets by dynamically computed intervals is not supported as of Solr 3.3
    • facet.pivot (Solr4)
      blog post on facet.pivot
    • post group faceting group.truncate=true JIRA SOLR-2665 (solved 3.4)
    • exlude filter when faceting
      http://localhost:8080/solr/select/?q=*%3A*&version=2.2&start=0&rows=0&indent=on&facet=on&facet.field={!ex=fcounty}fcounty&fq={!tag=fcounty}fcounty:Kent
    • using patch SOLR 2242 (order seems important)
      &facet.numTerms=true&facet.limit=-1&facet.mincount=1
  • Result Grouping
    • former name: Field collapsing
    • from Solr 3.3 on
    • best result for each type of a specific field
    • http://192.168.0.1:8080/solr/append/select?q=mobile&group=true&group.field=brand
    • ngroup
      • group=true&group.field=product&rows=0&group.ngroups=true
    • &group.main=true&group.format=simple: to get the same XML feed format as no grouping
    • group.truncate compute facet counts for only the highest ranking documents per-group
      q=chanel&start=0&rows=10&fl=*%2Cscore&group=true&group.field=shop_name&group.ngroups=true&group.truncate=true&facet=true&facet.field=type_on_sale
  • sort
    &sort=shop_state+desc,score+desc (put shop_state grouping first, then score in each grouping)






  • optimize
    Reorganize segments, merge all segments into one
    Remove any deleted docs
    stack overflox post about otimize






  • shards / sharding
    http://localhost:8080/solr/fr_big/select?shards=localhost:8080/solr/fr_big,localhost:8080/solr/fr_big2&indent=on&version=2.2&q=mobile&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=

  • StatsComponent to get stats on all data of a field in an index:
    http://192.168.0.1:8080/solr/append/select?q=mobile&stats=true&stats.field=price


  • Other notes:
    - solr.StrField does not support specifying an analyzer. Have to use solr.TextField with solr.KeywordTokenizerFactory for ex - 
    -When changing data model, remove the associated
    /var/data/solr
    - Solr4 atomic updates: curl '192.168.0.20:8080/solr/lg_fr_00/update?commit=true' -H 'Content-type:application/json' -d '[{"external_id":"28027304llant","offer_event":{"set":3}}]'
    - to specify a query encoding   ?ie=ISO-8859-1

    simple install  - Download solr from http://mir2.ovh.net/ftp.apache.org/dist//lucene/solr/1.4.1/apache-solr-1.4.1.zip
    - Copy apache-solr-1.4.1/dist/apache-solr-1.4.1.war in $TOMCAT_HOME/webapps/solr.war (for ex /var/lib/tomcat6/webapps/ )
    - From the solr distribution, copy example/solr in your solr home /etc/tomcat6/solr
    - in catalina.sh (/usr/share/tomcat/bin/catalina.sh):
    JAVA_OPTS="-Dsolr.solr.home=/etc/tomcat6/solr -Djava.awt.headless=true -server -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+DisableExplicitGC"
    - http://localhost:8080/solr/admin

    Full export
    wget --output-document=/tmp/offers_202.txt "http://solrhost:8080/solr/lg_fr_00/select?q=*:*&fl=external_id&rows=100000000&wt=csv&csv.header=false"





  • Zookeeper
    Zookeeper and Solr

  • Get Solr 3.5 source code
    svn co http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_3_5_0
    cd lucene_solr_3_5_0/
    ant get-maven-poms
    ant generate-maven-artifacts (may have mem exception but artifacts already generated)
    cd solr
    ant dist

  • Get Solr4    (doesn't compile with Java 8)





  • works also with Solr 6 / Java 8 ( https://issues.apache.org/jira/browse/SOLR-11014

  • tar zxvf ../Downloads/solr-4.1.0-src.tgz
    cd solr-4.1.0/

    cp /tmp/apache-ivy-2.4.0/ivy-2.4.0.jar /usr/share/ant/lib/
    ---------------- using Maven
    ant get-maven-poms
        Copying 49 files to /home/lee/tools/solr-4.1.0/maven-build
    ant ivy-bootstrap
    ant generate-maven-artifacts  
        actually builds everything: ./solr/build/solr-core/classes
    cd maven-build
    mvn -DskipTests source:jar-no-fork install
        To compile, package, and install all binary and source artifacts to your
        local repository, without running any tests:
    ------------- using Eclipse (importing the mvn project in Eclipse doesn't work)
    ant eclipse
    from Eclipse, create a new Java project with the "solr-4.1.0" folder. A new folder "eclipse-build" is then automatically created
    Build only classes. Useful for dev.
    ------------- using ant
    cd solr
    ant dist
    • Solr application servers
      • Following Solr5, Solr should be used as a blackbox.
    https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5
    http://grokbase.com/t/lucene/solr-user/15772hc1jd/jetty-in-solr-5-2-0
      •  As of Solr6, it is still using Jetty but Solr developers may make their own application server mechanism in the future. Solr should be used as a service, and not as a war anymore.



  • Solr versions



  • September 2008, Solr 1.3



  • November 2009 Solr 1.4



  • In March 2010, the Lucene and Solr projects merged.



  • 24-Jun-2010 1.4.1



  • 30-Mar-2011 3.1.0



  • 03-Jun-2011 3.2.0



  • 01-Jul-2011 3.3.0



  • 14-Sep-2011 3.4.0 



  • 6-jun-2017   6.6.0





  • main JIRAs
    --- Post group faceting:
    . - JIRA Lucene-3097: Post group faceting presents results in global facets and not in "per group" results. (open v3.4, 4.0)
    --- join:
    . - JIRA SOLR-2272: join: Map results onto other docs within the same shard fq={!join from=blog_id to=id}append=nikon
    --- Field collapsing
    . - JIRA SOLR-236: Field collapsing (closed v3.3)
    . - JIRA SOLR-1682: following up (open 3.4, 4.0)
    . - JIRA SOLR-1683: distributed field collapsing (open 3.4, 4.0)
    --- price min & max:
    . - JIRA SOLR-1581: Facet by Function, quantize buckets (open v3.4, 4.0): need SOLR-2251 that is duplicate of SOLR-1351
    --- distinct:
    . - JIRA SOLR-1814: select count(distinct fieldname) in SOLR (patch provided)
    . - JIRA SOLR-2242: Get distinct count names for a facet field (open 4.0): "shop count"
  • Sunday, January 2, 2011

    gccsense

    gccsense is used to do automatic completion when using emacs / C++. At first, did not install using Ubuntu 9.04, but eventually worked with Ubuntu 9.10.

    It worked OK using the following:

    http://cx4a.org/software/gccsense/

    I had to add the following two lines to my .emacs:

    (add-to-list 'load-path "~/.emacs.d/")
    (require 'gccsense)

    Got the completion when doing M-x gccsense-complete after . or ->.
    Have to record include paths with gccrec if you want completion with all your files (A little bit complicated)
    .