startup measurements: test, plot and report

doc/scaling/large-lists.md

@@ -12,10 +12,17 @@ Olof Hagsand, 2019-04-17
 
 ## 1. Background
 
-Clixon can handle large configurations. Here, large number of elements
-in a "flat" list is presented.  There are other scaling usecases,
-such as large configuratin "depth", large number of requesting
-clients, etc.
+Clixon can handle large configurations. Here, measurements using a
+large number of elements in a simple "flat" list is analysed. This
+includes starting up with alarge existing database; initializing an
+empty database with a large number of entries, accessing single
+entries with a large database, etc.
+
+In short, the results show a linear dependency on the number of
+entries. This is OK for startup scenarions, but single-enrty (transactional) operations need improvement.
+
+There are other scaling usecases, such as large configuratin "depth",
+large number of requesting clients, etc.
 
 Thanks to [Netgate](www.netgate.com) for supporting this work.
 
@@ -41,7 +48,7 @@ The basic case is a large list, according to the following Yang specification:
 ```
 where `a` is a unique key and `b` is a payload, useful in replace operations.
 
-XML lists with `N` elements are generated based on
+With this XML lists with `N` elements are generated based on
 this configuration, eg for `N=10`:
 ```
    <y><a>0</a><b>0</b></y>
@@ -66,6 +73,7 @@ Requests are either made over the _whole_ dataset, or for one specific element.
 Operations of single elements (transactions) are made in a burst of
 random elements, typically 100. 
 
+
 ## 3. Tests
 
 All details of the setup are in the [test script](../../test/plot_perf.sh).
@@ -76,6 +84,7 @@ All tests measure the "real" time of a command on a lightly loaded
 machine using the Linux command `time(1)`.
 
 The following tests were made (for each architecture and protocol):
+* Write `N` entries into the startup configuration. The clixon_backend was started with options `-1s startup`.
 * Write `N` entries in one single operation. (With an empty datastore)
 * Read `N` entries in one single operation. (With a datastore of `N` entries)
 * Commit `N` entries (With a candidate of `N` entries and empty running)
@@ -83,7 +92,7 @@ The following tests were made (for each architecture and protocol):
 * Write/Replace 1 entry (In a datastore of `N` entries)
 * Delete 1 entry (In a datastore of `N` entries)
 
-The tests are made using Netconf and Restconf, except commit which is made only for Netconf.
+The tests are made using Netconf and Restconf, except commit which is made only for Netconf and startup where protocol is irrelevant.
 
 ### Architecture and OS
 
@@ -118,9 +127,12 @@ The tests were made on the following hardware, all running Ubuntu Linux:
 
 ## 4. Results
 
-### Access of the whole datastore
-
 This section shows the results of the measurements as defined in [Tests](#tests).
+### Startup
+
+![Startup](clixon-startup.png "Startup")
+
+### Access of the whole datastore
 
 ![Get config](clixon-get-0.png "Get config")
 
@@ -196,11 +208,14 @@ system degrades with the size of the lists.
 Examining the profiling of the most demanding Restconf PUT case, most
 cycles are spent on handling writing and copying the existing datastore.
 
-Concluding, the 
+Note that the experiments here contains _very_ simple
+data-structures. A more realistic complex example will require more
+CPU effort. Ad-hoc measurement of a more complex datastructure,
+generated four times the duration of the simple yang model in this work.
 
 ## 6. Future work
 
-* Improve access of single list elements to sub-linear performance.
+* Improve access of individual elements to sub-linear performance.
 * CLI access on large lists (not included in this study)
 
 ## 7. References

test/plot_perf.sh

@@ -270,7 +270,52 @@ plot(){
     echo # newline
 }
 
+# Run an operation, iterate from <from> to <to> in increment of <step> 
+# Each operation do <reqs> times
+# args: <op> <protocol> <from> <step> <to> <reqs> <cand> <run>
+# <reqs>=0 means all in one go
+startup(){
+    from=$1
+    step=$2
+    to=$3
+    mode=startup
+
+    if [ $# -ne 3 ]; then
+	exit "plot should be called with 3 arguments, got $#"
+    fi
+
+    # gnuplot file
+    gfile=$resdir/startup-$arch    
+    new "Create file $gfile"
+    echo -n "" > $gfile
+
+    # Startup db: load with n entries
+    dbfile=$dir/${mode}_db
+    sudo touch $dbfile
+    sudo chmod 666 $dbfile
+    for (( n=$from; n<=$to; n=$n+$step )); do  
+	new "startup-$arch $n"
+	new "Generate $n entries to $dbfile"
+	echo -n "<config><x xmlns=\"urn:example:clixon\">" > $dbfile
+	for (( i=0; i<$n; i++ )); do  
+	    echo -n "<y><a>$i</a><b>$i</b></y>" >> $dbfile
+	done
+	echo "</x></config>" >> $dbfile
+
+	new "Startup backend once -s $mode -f $cfg -y $fyang"
+	echo -n "$n " >>  $gfile
+	{ time -p sudo $clixon_backend -F1 -D $DBG -s $mode -f $cfg -y $fyang 2> /dev/null; } 2>&1 |  awk '/real/ {print $2}' | tr , . >> $gfile
+
+    done
+    echo # newline
+}
+
 if $run; then
+
+    # Startup test before regular backend/restconf start since we only start
+    # backend a single time
+    startup $step $step $to
+
     new "test params: -f $cfg -y $fyang"
     if [ $BE -ne 0 ]; then
 	new "kill old backend"
@@ -296,6 +341,7 @@ to=$to0
     step=$step0
     reqs=$reqs0
 
+
     # Put all tests
     for proto in netconf restconf; do
 	new "$proto put all entries to candidate (restconf:running)"
@@ -342,6 +388,24 @@ fi # if run
 
 if $plot; then
 
+# 0. Startup
+gplot=""
+for a in $archs; do
+    gplot="$gplot \"$resdir/startup-$a\" title \"startup-$a\","
+done
+
+gnuplot -persist <<EOF
+set title "Clixon startup"
+set style data linespoint
+set xlabel "Entries"
+set ylabel "Time[s]"
+set grid
+set terminal $term
+set yrange [*:*]
+set output "$resdir/clixon-startup.$term"
+plot $gplot
+EOF
+
 # 1. Get config
 gplot=""
 for a in $archs; do

test/test_perf.sh

@@ -5,7 +5,7 @@
 s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
 
 # Number of list/leaf-list entries in file
-: ${perfnr:=10000}
+: ${perfnr:=20000}
 
 # Number of requests made get/put
 : ${perfreq:=100}
@@ -59,6 +59,13 @@ cat <<EOF > $cfg
 </clixon-config>
 EOF
 
+if [ $BE -ne 0 ]; then
+    new "kill old backend"
+    sudo clixon_backend -zf $cfg -y $fyang
+    if [ $? -ne 0 ]; then
+	err
+    fi
+fi
 # Try startup mode w startup
 for mode in startup running; do
     file=$dir/${mode}_db
@@ -113,7 +120,6 @@ new "netconf write large config"
 expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 # Here, there are $perfnr entries in candidate
-
 new "netconf write large config again"
 expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"