swiss knife blog

Thursday, January 17, 2013

Elastic Search Survival Kit

Each Lucene segment has its own cache. So indexing is not affecting too much search performances
every node is a "master", everybody indexes and everybody searches
To use kibana (http://127.0.0.1:5601/app/kibana#/dev_tools)
to be able to access kibana from remote, change kibana.yml:

server.host: "0.0.0.0"

PUT /fb
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"type": "custom",
"tokenizer" : "whitespace",
"filter": ["lowercase", "stop"]
}
}
}
},
"mappings": {
"pages": {
"properties": {
"about": {
"type": "text",
"fielddata": true,
"analyzer": "my_analyzer"
}
}
}
}
}

----- term facets

GET /fb/pages/_search
{
"size": 1,
"aggs" : {
"group_by_text_term" : {
"terms" : {
"field" : "about",
"size":30
}
}
}
}

----------- facets on a category (after v5, text columns are also keyword columns)
GET /fb2/pages/_search
{
"size": 1,
"query": {
"match_all": {}
},
"aggs": {
"group_by_cat": {
"terms": {
"field": "category.keyword",
"size": 10
}
}
}
}

------------- delete all docs
POST /fb2/_delete_by_query
{
"query": {
"match_all": {}
}
}

------------- to get more details of what is happening in a query

POST antoine-try-iso2/_search

{

"profile": true,

"_source": ["s_alpha.denom"],

"query": {

[...]

------------- http Elastic queries

http://127.0.0.1:9200/_cat/indices

http://localhost:9200/_search?q=lastname:Bond

http://127.0.0.1:9200/pj-search-index-1-6-test/_search

------------- search templates

GET _cluster/state/metadata?pretty&filter_path=**.stored_scripts

------------- versions

Elasticsearch 2.4.0 based on Lucene 5.5.2 (08/2016)
(ES version hop 2.4 to 5.0)

Elasticsearch 5.0 (11/2016)
Elasticsearch 5.5.1, based on Lucene 6.5.1 (07/2017)

Elasticsearch 5.6.3, based on Lucene 6.6.3 (07/2017)

Elasticsearch 6.0.0 beta (05/2017)   based on Lucene 7.0.0
Elasticsearch 6.7.1 beta (04/2019)   based on Lucene 7.7.1
Elasticsearch 7.0.0 beta (04/2019)   based on Lucene 8.0.0

In git sources:

vi buildSrc/version.properties

elasticsearch = 5.6.3
lucene = 6.6.1
.

Tuesday, January 8, 2013

Hadoop / Cloudera survival kit

----- Debian
add this in /etc/apt/sources.list:

deb http://archive.cloudera.com/cdh4/debian/squeeze/amd64/cdh/ squeeze-cdh4.1.2 contrib

then you can do:

apt-get update
apt-get install hadoop

and then, things like:

hadoop fs -ls hdfs://192.168.0.135:8020/

----- Ubuntu:
add to /etc/apt/sources.list

deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/ precise-cdh4 contrib
deb-src http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh precise-cdh4 contrib

curl -s http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -
apt-get update
sudo apt-get install hbase-master

[toto@vv182 ~]$ echo "scan 'offers', {LIMIT => 10, STARTROW => 'se|000029098138', ENDROW => 'se|0000291'}" |hbase shell

[toto@vv182 ~]$ echo "get 'offers','fr|000000002138|0000016418701245'" |hbase shell

[toto@vv182 ~]$ hadoop fs -cat /user/nomad/pipeline/delta_offers/my-file.txt

Friday, December 28, 2012

GIT survival kit

ssh -T git@192.168.0.444 (list remote git projects)
  git clone git@192.168.0.444:leg/testgitlab.git
(sort of check-out, but copy all the history of all the files)
  cd testgitlab/
  vi 2.txt
  git add 2.txt ("stage" the file)
  git commit -m " new file "
(local commit, commit everything that is "staged" into the local git repository)

show deleted files for each commit

git log --diff-filter=D --summary

branches

to add a branch "prod" locally and in the remote repository

git branch your-branch
git push origin your-branch

git branch -av

list all remote branches wit their pointer value

"checkout" a remote branch

git switch MEDNEM-whatever

to add a branch "prod" in .git/config

git branch --set-upstream prod origin/prod

to have master point to a former commit

git reset --hard 112ab334
git push -f

rename a branch locally and remotely

git branch -n new-name. (locally)
git push origin --delete old-name
git push --set-upstream origin new-name

delete remote branch

git push -d origin MEDNEM-47

diff between the local repo and the central repo

git diff master origin/master

ignore last commit not yet pushed (move the reference pointer, doesn't delete the commit)

git reset HEAD~1

remove last commit locally and remotely (if branch not protected)

```
git reset --hard HEAD^ 
```
```
git push -f 
```

"undo" a commit (if branch protected)

git revert  e52b3378f42c82720c41e7dc08ae211c56ef164d
(or "git revert HEAD" if only last commit)
git push

if I want to revert to a commit that was a merge
git revert -m 1 6b90123523c78ecc5a63e88b4a8605a530e80e56

You can recover an individual file with:

git checkout -- JavaTVServiceXlet.java...

or to restore all the files that are have been deleted, you could do:
git ls-files -z --deleted | xargs -0 -n 1 git checkout --

get local repo at a specific commit id

git reset --hard e52b3378f42c82720c41e7dc08ae211c56ef164d
see also at the end

git checkout -b tmp 03d0e...26a

restart from the remote branch (after non wanted local commits)

git reset --hard origin/mybranch

Dev scenario 1

2572 git clone git@192.168.0.44:/leg/leg/search-l
2577 git branch
2548 git branch dev_local
2549 git branch
2550 git checkout dev_local
----------- I do my dev and others commit too on the central repo
2556 git status
2557 git commit -a -m"." (you have to commit everything here, otherwise see Dev scenario 2)
2558 git status
2559 git checkout master
2560 git pull
2561 git checkout dev_local
2562 git merge master dev_local
2563 git checkout master
2564 git merge dev_local master
2565 git push

Dev Scenario 2 (cherry-pick scenario)

2776 git commit your stuff

2776 git stash

2776 git branch tmp2
2777 git checkout tmp2
2781 git cherry-pick 96c53838ba55b72ef16d31f1076393de03c2e0e1 (possibly with option -m 1)
2789 git checkout master
2790 git merge tmp2
2793 git pull

2794 git push
2795 git branch -D tmp2

Dev Scenario 3 (add a branch out of release)

git checkout release
git branch my_branch release
git push origin my_branch (my_branch points to release)
git checkout my_branch
git add myfile.txt
git commit ...
git push origin my_branch (the commits is shown as head, ahead of master)

Dev Scenario 4 (create a branch from tags)

git submodule foreach git checkout tags/3.3.11
git submodule foreach git checkout -b prod
git submodule foreach git push origin prod

Dev scenario 5: I mistakenly commit on branch and want to restart

git stash

if you haven't commit anything and want to update the branch
git stash pop (get back what was stashed, and remove it from the stash list)

To remove local tag

git tag -d 3.3.03.rc2

To remove tags on central repository

git push origin :3.1.2.b1
git submodule foreach git push origin :3.1.2.b1

tagging

git checkout tad_id (get a specific tag)

git merge conflict resolution

git merge one_branch
git checkout --ours pom.xml (keep master version numbers), or edit files
git add pom.xml
git commit
git push
Note that during git rebase and git pull --rebase, 'ours' and 'theirs' may appear swapped; --ours gives the version from the branch the changes are rebased onto, while --theirs gives the version from the branch that holds your work that is being rebased.

git rebase

instead of

git merge one_branch

git rebase one_branch
git push --force

git rebase -i develop-1

allow to chose or get rid off some commits
allow to squash by replacing "pick" by "squash" all line from the second one

https://www.git-tower.com/learn/git/faq/git-squash

squashing

git rebase -i HEAD~3 (no. of commits you want to squash including the current one)
You will get one interactive prompt where you need to Pick the top commit and insert squash or s in front of those which you want to combine/squash.
you will get another interactive prompt, put # in front of commits message that you don't want, and or add your own message.
git push --force

Mistakenly committed to the master branch (but didn't push)

git branch -m one_branch (rename master to one_branch)
more .git/config
git branch --unset-upstream
more .git/config (should be no more remote)
git push --set-upstream origin one_branch. (pushes to remote if you want to ?)
git checkout master

Reverting Working Copy to Most Recent Commit

http://stackoverflow.com/questions/4114095/revert-git-repo-to-a-previous-commit/4114122#4114122

Show added files since a commit

git diff b28d54ac880c2381d700001aa9f50dcbb5f73617 --diff-filter=A --numstat

A branch from which I created my branch is removed

way #1

git fetch
git branch -f MEDNEM-3665 origin/MEDNEM-3665
git rebase --onto origin/MEDNEM-3665 7319...34a MEDNEM-3716
git push --force

(while you are on MEDNEM-3716)

way #2

git fetch
git rebase --onto origin/main 6a4...e9 MEDNEM-3716
git push --force
(while you are on MEDNEM-3716)

I want to create a branch from a commit ID to test an older version

git checkout -b tmp 03d0e...26a

Config

https://swiss-knife.blogspot.com/2017/04/github.html

command line commit tree visualization

git log --graph --oneline --all
there is another tool "gitk" but it adds limited value)

difftool

git difftool commit_id_1 commit_id_2

difftool using "beyond compare"

lee@MacBook-Pro:~/dev$ git config --global diff.tool bc
lee@MacBook-Pro:~/dev$ git config --global difftool.bc.cmd 'bcomp "$LOCAL" "$REMOTE"'
lee@MacBook-Pro:~/dev$ git difftool master mybranch

Sunday, September 9, 2012

"screen" command

- list all screens: CTRL A "
- create one screen: CTRL A c
- close one screen: CTRL D
- help: CTRL A ?
- escape: space
- leave the screen session: CTRL A d
- rename session: (C-a :) and then enter the command sessionname SESSIONNAME
-rename window: (C-a A)

-renumber window: (C-a :) and then "number XXX" (warning: not bigger than the existing max)

- re-attach a "Detached" screen session

- re-attach an "Attached"
. - if the screen is attached: screen -D -r

Friday, April 20, 2012

boost and cpp-netlib

-- Boost --
./bootstrap.sh --prefix=/home/lll/tmp/tmp-cpp-netlib/boost_1_49_0/
./b2 --layout=tagged install

(need layout to generate multithread boost libs)

-- cpp-netlib --
cmake -DBOOST_ROOT=/home/lll/tmp/tmp-cpp-netlib/boost_1_49_0/ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ .
make

Thursday, July 7, 2011

Scalability concepts

Google BigTable
http://labs.google.com/papers/bigtable.html

BigData : are datasets that grow so large that they become awkward to work with
http://en.wikipedia.org/wiki/Big_data

Hadoop

MapReduce : Google framework

Dynamo: Amazon's Highly Available Key-value Store
http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

- Master / slave
- peer to peer
- cluster: similar components
- grid: different components

Wednesday, July 6, 2011

aptitude search result

p, meaning that no trace of the package exists on the system,
c, meaning that the package was deleted but its configuration files remain on the system,
i, meaning that the package is installed,
v, meaning that the package is virtual.

The second character indicates the stored action (if any; otherwise a blank space is displayed) to be performed on the package
i, meaning that the package will be installed,
d, meaning that the package will be deleted,
p, meaning that the package and its configuration files will be removed.

If the third character is A, the package was automatically installed