diff --git a/CHANGELOG.md b/CHANGELOG.md index 7edfe80e..57215d1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,7 @@ ### Minor changes +* A scaling of [large lists](doc/scaling) report is added * A new "hello world" example is added * Optimized validation of large lists * New xmldb_get1() returning actual cache - not a copy. This has lead to some householding instead of just deleting the copy diff --git a/README.md b/README.md index 38385af8..b26a32d2 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,7 @@ support. * [Background](#background) * [Frequently asked questions (FAQ)](doc/FAQ.md) + * [Hello world](example/hello/README.md) * [Changelog](CHANGELOG.md) * [Installation](#installation) * [Licenses](#licenses) @@ -26,7 +27,7 @@ support. * [Runtime](#runtime) * [Clixon project page](http://www.clicon.org) * [Tests and CI](test/README.md) - * [Scaling: large lists](doc/large-lists.md) + * [Scaling: large lists](doc/scaling/large-lists.md) * [Containers](docker/README.md) * [Roadmap](doc/ROADMAP.md) * [Reference manual](#reference) diff --git a/doc/large-lists.md b/doc/large-lists.md deleted file mode 100644 index 1b1b15a4..00000000 --- a/doc/large-lists.md +++ /dev/null @@ -1,134 +0,0 @@ -# Large lists in Clixon - - * [Background](#background) - * [Overview](#overview) - * [Test descriptions]#test-descriptions) - -## Background - -Clixon is a configuration management tool. In this paper the case of -a large number of "flat" list and leaf-list entries are investigated. -There may be other scaling usecases, such as large configuratin -"depth", large number of requesting clients, etc. However, these are -not investigated here. - -## Overview -The basic case is a large list, according to the following Yang specification: -``` - list y { - key "a"; - leaf a { - type int32; - } - leaf b { - type string; - } - } -``` -where `a` is a unique key and `b` is a payload, useful in replace operations. - -There is also a leaf-list as follows: -``` - leaf-list c { - type string; - } -``` - -XML lists with `N` elements are generated based on -this configuration, eg for `N=10`: -``` - 00 - 11 - 22 - 33 - 44 - 55 - 66 - 77 - 88 - 99 -``` - -Requests are made using a random function, a request on the list above will on the form: -``` - curl -G http://localhost/restconf/data/y=(rnd%$N) -``` - -## Test descriptions - -### Limitations - -Test were not made using CLI interaction. - -### Setup - -The setup consisted of the following components running on the same machine: -* A clixon backend daemon -* A clixon restconf daemon -* An nginx daemon daemon -* A netconf client program -* curl client -* A bash terminal and test script [plot_perf.sh](../test/plot_perf.sh) -* Gnuplot for generating plots - -### Config file -The following Clixon config file was used: -``` - - $cfg - $dir - /usr/local/share/clixon - scaling - /usr/local/var/example/example.sock - /usr/local/var/example/example.pidfile - false - $dir - false - -``` -where `$dir` and `$cfg`are local files. For more info see [plot_perf.sh]. - -### Testcases - -All tests measure the "real" time of a command on a lightly loaded -machine using the Linux command `time(1)`. - -The following tests were made (for each architecture and protocol): -* Write `N` entries in one single operation. (With an empty datastore) -* Read `N` entries in one single operation. (With a datastore of `N` entries) -* Commit `N` entries (With a candidate of `N` entries and empty running) -* Read 1 entry (In a datastore of `N` entries) -* Write/Replace 1 entry (In a datastore of `N` entries) -* Delete 1 entry (In a datastore of `N` entries) - -### Protocols - -The tests are made using: -* Netconf[RFC6241] and -* Restconf[RFC8040]. -Notably, CLI tests are for future study. - -### Architectures - -The tests were made on the following hardware, all running Ubuntu Linux: -* [i686] dual Intel Core Duo processor (IBM Thinkpad X60), 3GB memory -* arm 32-bit (Raspberry PI 3) -* x86 64-bit (Intel NUC) - -### Operating systems - -On i686: -``` -Linux version 4.4.0-143-generic (buildd@lgw01-amd64-037) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #169-Ubuntu SMP Thu Feb 7 07:56:51 UTC 2019 -``` - -## Results - -## References - -[RFC6241](https://tools.ietf.org/html/rfc6241) "Network Configuration Protocol (NETCONF)" -[RFC8040](https://tools.ietf.org/html/rfc8040) "RESTCONF Protocol" -[i686](https://ark.intel.com/content/www/us/en/ark/products/27235/intel-core-duo-processor-t2400-2m-cache-1-83-ghz-667-mhz-fsb.html) -[plot_perf.sh](../test/plot_perf.sh) Test script - - diff --git a/doc/scaling/clixon-commit-0.png b/doc/scaling/clixon-commit-0.png new file mode 100644 index 00000000..cbab820a Binary files /dev/null and b/doc/scaling/clixon-commit-0.png differ diff --git a/doc/scaling/clixon-delete-100.png b/doc/scaling/clixon-delete-100.png new file mode 100644 index 00000000..9396c483 Binary files /dev/null and b/doc/scaling/clixon-delete-100.png differ diff --git a/doc/scaling/clixon-get-0.png b/doc/scaling/clixon-get-0.png new file mode 100644 index 00000000..2879c21e Binary files /dev/null and b/doc/scaling/clixon-get-0.png differ diff --git a/doc/scaling/clixon-get-100.png b/doc/scaling/clixon-get-100.png new file mode 100644 index 00000000..246768d4 Binary files /dev/null and b/doc/scaling/clixon-get-100.png differ diff --git a/doc/scaling/clixon-put-0.png b/doc/scaling/clixon-put-0.png new file mode 100644 index 00000000..2e9acc15 Binary files /dev/null and b/doc/scaling/clixon-put-0.png differ diff --git a/doc/scaling/clixon-put-100.png b/doc/scaling/clixon-put-100.png new file mode 100644 index 00000000..3876d231 Binary files /dev/null and b/doc/scaling/clixon-put-100.png differ diff --git a/doc/scaling/large-lists.md b/doc/scaling/large-lists.md new file mode 100644 index 00000000..b36f9f2c --- /dev/null +++ b/doc/scaling/large-lists.md @@ -0,0 +1,131 @@ +# Large lists in Clixon + + * [Background](#background) + * [Overview](#overview) + * [Test descriptions](#test-descriptions) + * [Results](#results) + * [References](#references) + +## Background + +CIixon can handle large configurations. Here, large number of elements +in a "flat" list is presented. There are other scaling usecases, +such as large configuratin "depth", large number of requesting +clients, etc. + +## Overview + +The basic case is a large list, according to the following Yang specification: +``` + container x { + description "top-level container"; + list y { + description "List with potential large number of elements"; + key "a"; + leaf a { + description "key in list"; + type int32; + } + leaf b { + description "payload data"; + type string; + } + } + } +``` +where `a` is a unique key and `b` is a payload, useful in replace operations. + +XML lists with `N` elements are generated based on +this configuration, eg for `N=10`: +``` + 00 + 11 + 22 + 33 + 44 + 55 + 66 + 77 + 88 + 99 +``` + +Requests are either made over the _whole_ dataset, or for one specific element. The following example shows a Restconf GET operation of a single element: +``` + curl -X GET http://localhost/restconf/data/scaling:x/y=3 + {"scaling:y": [{"a": 3,"b": "3"}]} + +``` + +Operations of single elements (transactions) are made in a burst of +random elements, typically 100. + +## Tests + +All details of the setup are in the [test script](../../test/plot_perf.sh). + +### Testcases + +All tests measure the "real" time of a command on a lightly loaded +machine using the Linux command `time(1)`. + +The following tests were made (for each architecture and protocol): +* Write `N` entries in one single operation. (With an empty datastore) +* Read `N` entries in one single operation. (With a datastore of `N` entries) +* Commit `N` entries (With a candidate of `N` entries and empty running) +* Read 1 entry (In a datastore of `N` entries) +* Write/Replace 1 entry (In a datastore of `N` entries) +* Delete 1 entry (In a datastore of `N` entries) + +The tests are made using Netconf and Restconf, except commit which is made only for Netconf. + +### Architecture and OS + +The tests were made on the following hardware, all running Ubuntu Linux: +* i686: dual Intel Core Duo processor (IBM Thinkpad X60) +* arm: ARMv7 Processor rev 5 (v7l) (Raspberry PI 2 Model B) +* x86-64: Intel Quad-core I5-8259U (Intel NUC Coffee Lake) + +i686: Ubuntu 16.04.6 LTS +``` +Linux version 4.4.0-143-generic (buildd@lgw01-amd64-037) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #169-Ubuntu SMP Thu Feb 7 07:56:51 UTC 2019 +``` + +Arm : Raspbian GNU/Linux 9 +``` + +Linux version 4.14.79-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1159 SMP Sun Nov 4 17:50:20 GMT 2018 +``` + +x86_64: Ubuntu 18.04.1 LTS +``` +inux version 4.15.0-47-generic (buildd@lgw01-amd64-001) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 +``` + +## Results + +![Get config](clixon-get-0.png "Get config") + +![Put config](clixon-put-0.png "Put config") + +![Commit config](clixon-commit-0.png "Commit config") + +![Get single entry](clixon-get-100.png "Get single entry") + +![Put single entry](clixon-put-100.png "Put single entry") + +![Delete single entry](clixon-delete-100.png "Delete single entry") + +## Discussion + + + + +## References + +[RFC6241](https://tools.ietf.org/html/rfc6241) "Network Configuration Protocol (NETCONF)" +[RFC8040](https://tools.ietf.org/html/rfc8040) "RESTCONF Protocol" +[i686](https://ark.intel.com/content/www/us/en/ark/products/27235/intel-core-duo-processor-t2400-2m-cache-1-83-ghz-667-mhz-fsb.html) +[plot_perf.sh](../test/plot_perf.sh) Test script + + diff --git a/example/README.md b/example/README.md index 58cdc401..66149d8b 100644 --- a/example/README.md +++ b/example/README.md @@ -1,6 +1,5 @@ # Clixon examples Clixon have the following examples: - * [Main example](main/README.md) * [Hello world](hello/README.md) - \ No newline at end of file + * [Main example](main/README.md) diff --git a/test/plot_perf.sh b/test/plot_perf.sh index 5cbc3cc7..e4a95acf 100755 --- a/test/plot_perf.sh +++ b/test/plot_perf.sh @@ -1,39 +1,34 @@ #!/bin/bash -# Transactions per second for large lists read/write plotter using gnuplot -# What do I want to plot? -# First: on i32, i64, arm32 -# PART 1: Basic load -# 1. How long to write 100K entries? -# - netconf / restconf -# - list / leaf-list -# 2. How long to read 100K entries? -# - netconf/ restconf -# - list / leaf-list -# 3. How long to commit 100K entries? (netconf) -# - list / leaf-list -# -# PART 2: Load 100K entries. Commit. -# 4. How many read operations per second? -# - netconf/ restconf -# - list / leaf-list -# 5. How many write operations per second? -# - netconf / restconf -# - list / leaf-list -# 6. How may delete operations per second? -# - netconf / restconf -# - list / leaf-list -# The script uses bash builtin "time" command which is somewhat difficult to -# understand. See: https://linux.die.net/man/1/bash # pipelines -# You essentially have to do: { time stuff; } 2>&1 -# See: https://stackoverflow.com/questions/26784870/parsing-the-output-of-bashs-time-builtin +# Performance of large lists. See large-lists.md +# The parameters are shown below (under Default values) +# Examples +# 1. run all measurements up to 10000 entris collect all results in /tmp/plots +# run=true plot=false to=10000 resdir=/tmp/plots ./plot_perf.sh +# 2. Use existing data plot and show on X11 +# run=false plot=true resdir=/tmp/plots term=x11 ./plot_perf.sh +# 3. Use existing data plot i686 and armv7l data as png +# archs="i686 armv7l" run=false plot=true resdir=/tmp/plots term=png ./plot_perf.sh # Magic line must be first in script (see README.md) s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi -# op from step to reqs -to=1000 -step=100 -reqs=100 +arch=$(arch) +# Default values +: ${to:=5000} # Max N +: ${step=1000} # Iterate in steps (also starting point) +: ${reqs=100} # Number of requests in each burst +: ${run:=true} # run tests (or skip them). If false just plot +: ${term:=x11} # x11 interactive, alt: png +: ${resdir=$dir} # Result dir (both data and gnuplot) +: ${plot=false} # Result dir (both data and gnuplot) +: ${archs=$arch} # Plotting can be made for many architectures (not run) + +# 0 prefix to protect against shell dynamic binding) +to0=$to +step0=$step +reqs0=$reqs + +ext=$term # gnuplot output file extenstion # Global variables APPNAME=example @@ -42,6 +37,13 @@ fyang=$dir/plot.yang fxml=$dir/data.xml fjson=$dir/data.json +# Resultdir - if different from $dir that gets erased +#resdir=$dir + +if [ ! -d $resdir ]; then + mkdir $resdir +fi + # For memcheck # clixon_netconf="valgrind --leak-check=full --show-leak-kinds=all clixon_netconf" # clixon_netconf="valgrind --tool=callgrind clixon_netconf @@ -53,19 +55,20 @@ module scaling{ namespace "urn:example:clixon"; prefix sc; container x { - list y { - key "a"; - leaf a { - type uint32; + description "top-level container"; + list y { + description "List with potential large number of elements"; + key "a"; + leaf a { + description "key in list"; + type int32; + } + leaf b { + description "payload data"; + type string; + } } - leaf b { - type string; - } - } - leaf-list c { - type string; - } - } + } } EOF @@ -109,48 +112,49 @@ genfile(){ # where proto is one of: # netconf, restconf # where op is one of: -# writeall readall commitall read write +# get put delete commit runnet(){ op=$1 - n=$2 # Number of entries in DB + nr=$2 # Number of entries in DB (keep diff from n due to shell dynamic binding) reqs=$3 - echo -n "$n " >> $dir/$op-netconf-$reqs + file=$resdir/$op-netconf-$reqs-$arch + echo -n "$nr " >> $file case $op in - write) + put) if [ $reqs = 0 ]; then # Write all in one go - genfile $n netconf; - { time -p cat $fxml | $clixon_netconf -qf $cfg -y $fyang ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs + genfile $nr netconf; + { time -p cat $fxml | $clixon_netconf -qf $cfg -y $fyang ; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file else # reqs != 0 { time -p for (( i=0; i<$reqs; i++ )); do - rnd=$(( ( RANDOM % $n ) )); + rnd=$(( ( RANDOM % $nr ) )); echo "$rnd$rnd]]>]]>"; - done | $clixon_netconf -qf $cfg -y $fyang ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs + done | $clixon_netconf -qf $cfg -y $fyang > /dev/null; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file fi ;; - read) + get) if [ $reqs = 0 ]; then # Read all in one go - { time -p echo "]]>]]>" | $clixon_netconf -qf $cfg -y $fyang > /dev/null ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs + { time -p echo "]]>]]>" | $clixon_netconf -qf $cfg -y $fyang > /dev/null ; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file else # reqs != 0 { time -p for (( i=0; i<$reqs; i++ )); do rnd=$(( ( RANDOM % $nr ) )) - echo "$rnd$rnd]]>]]>" -done | $clixon_netconf -qf $cfg -y $fyang; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs + echo "$rnd$rnd]]>]]>" + done | $clixon_netconf -qf $cfg -y $fyang > /dev/null; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file fi ;; delete) { time -p for (( i=0; i<$reqs; i++ )); do rnd=$(( ( RANDOM % $nr ) )) - echo "$rnd$rnd]]>]]>" -done | $clixon_netconf -qf $cfg -y $fyang; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs + echo "$rnd$rnd]]>]]>" +done | $clixon_netconf -qf $cfg -y $fyang; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file ;; commit) - { time -p echo "]]>]]>" | $clixon_netconf -qf $cfg -y $fyang > /dev/null ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs + { time -p echo "]]>]]>" | $clixon_netconf -qf $cfg -y $fyang > /dev/null ; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file ;; *) err "Operation not supported" "$op" exit - ;; + ;; esac } @@ -159,43 +163,43 @@ done | $clixon_netconf -qf $cfg -y $fyang; } 2>&1 | awk '/real/ {print $2}' >> $ # where proto is one of: # netconf, restconf # where op is one of: -# writeall readall commitall read write +# get put delete runrest(){ op=$1 - n=$2 # Number of entries in DB + nr=$2 # Number of entries in DB reqs=$3 - echo -n "$n " >> $dir/$op-restconf-$reqs + file=$resdir/$op-restconf-$reqs-$arch + echo -n "$nr " >> $file case $op in - write) + put) if [ $reqs = 0 ]; then # Write all in one go - genfile $n restconf + genfile $nr restconf # restconf @- means from stdin - { time -p curl -sS -X PUT -d @$fjson http://localhost/restconf/data/scaling:x ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs + { time -p curl -sS -X PUT -d @$fjson http://localhost/restconf/data/scaling:x ; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file else # Small requests { time -p for (( i=0; i<$reqs; i++ )); do - rnd=$(( ( RANDOM % $n ) )); + rnd=$(( ( RANDOM % $nr ) )); curl -sS -X PUT http://localhost/restconf/data/scaling:x/y=$rnd -d "{\"scaling:y\":{\"a\":$rnd,\"b\":\"$rnd\"}}" - done ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs + done ; } 2>&1 | awk '/real/ {print $2}' | tr , .>> $file # fi ;; - read) + get) if [ $reqs = 0 ]; then # Read all in one go - { time -p curl -sS -X GET http://localhost/restconf/data/scaling:x > /dev/null; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs + { time -p curl -sS -X GET http://localhost/restconf/data/scaling:x > /dev/null; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $file else # Small requests { time -p for (( i=0; i<$reqs; i++ )); do - rnd=$(( ( RANDOM % $n ) )); + rnd=$(( ( RANDOM % $nr ) )); curl -sS -X GET http://localhost/restconf/data/scaling:x/y=$rnd - done ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs + done ; } 2>&1 | awk '/real/ {print $2}' | tr , .>> $file fi ;; - delete) + delete) { time -p for (( i=0; i<$reqs; i++ )); do - rnd=$(( ( RANDOM % $n ) )); + rnd=$(( ( RANDOM % $nr ) )); curl -sS -X GET http://localhost/restconf/data/scaling:x/y=$rnd - done ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs - + done ; } 2>&1 | awk '/real/ {print $2}' | tr , .>> $file ;; *) err "Operation not supported" "$op" @@ -246,8 +250,8 @@ plot(){ fi # reset file - new "Create file $dir/$op-$proto-$reqs" - echo "" > $dir/$op-$proto-$reqs + new "Create file $resdir/$op-$proto-$reqs-$arch" + echo -n "" > $resdir/$op-$proto-$reqs-$arch for (( n=$from; n<=$to; n=$n+$step )); do reset if [ $can = n ]; then @@ -256,7 +260,7 @@ plot(){ commit fi fi - new "$op-$proto-$reqs $n" + new "$op-$proto-$reqs-$arch $n" if [ $proto = netconf ]; then runnet $op $n $reqs else @@ -266,6 +270,7 @@ plot(){ echo # newline } +if $run; then new "test params: -f $cfg -y $fyang" if [ $BE -ne 0 ]; then new "kill old backend" @@ -286,26 +291,35 @@ start_restconf -f $cfg -y $fyang new "waiting" sleep $RCWAIT + +to=$to0 +step=$step0 +reqs=$reqs0 + +# Put all tests for proto in netconf restconf; do - new "$proto write all entries to candidate (restconf:running)" - plot write $proto $step $step $to 0 0 0 # all candidate 0 running 0 + new "$proto put all entries to candidate (restconf:running)" + plot put $proto $step $step $to 0 0 0 # all candidate 0 running 0 done +# Get all tests for proto in netconf restconf; do - new "$proto read all entries from running" - plot read netconf $step $step $to 0 n n # start w full datastore + new "$proto get all entries from running" + plot get $proto $step $step $to 0 n n # start w full datastore done +# Netconf commit all new "Netconf commit all entries from candidate to running" plot commit netconf $step $step $to 0 n 0 # candidate full running empty -reqs=100 +# Transactions get/put/delete +reqs=$reqs0 for proto in netconf restconf; do - new "$proto read $reqs from full database" - plot read $proto $step $step $to $reqs n n + new "$proto get $reqs from full database" + plot get $proto $step $step $to $reqs n n - new "$proto Write $reqs to full database(replace / alter values)" - plot write $proto $step $step $to $reqs n n + new "$proto put $reqs to full database(replace / alter values)" + plot put $proto $step $step $to $reqs n n new "$proto delete $reqs from full database(replace / alter values)" plot delete $proto $step $step $to $reqs n n @@ -324,15 +338,127 @@ if [ $BE -ne 0 ]; then # kill backend stop_backend -f $cfg fi +fi # if run + +if $plot; then + +# 1. Get config +gplot="" +for a in $archs; do + gplot="$gplot \"$resdir/get-restconf-0-$a\" title \"rc-$a\", \"$resdir/get-netconf-0-$a\" title \"nc-$a\"," +done -arch=$(arch) gnuplot -persist <