experiments/clixon
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

startup measurements: test, plot and report

Browse source
This commit is contained in:
Olof hagsand 2019-04-23 11:57:18 +02:00
parent 6a0628141a
commit 77ad42f1ce
4 changed files with 149 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
doc/scaling/clixon-startup.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 6.1 KiB

35
doc/scaling/large-lists.md
View file

@ -12,10 +12,17 @@ Olof Hagsand, 2019-04-17
## 1. Background ## 1. Background
Clixon can handle large configurations. Here, large number of elements Clixon can handle large configurations. Here, measurements using a
in a "flat" list is presented. There are other scaling usecases, large number of elements in a simple "flat" list is analysed. This
such as large configuratin "depth", large number of requesting includes starting up with alarge existing database; initializing an
clients, etc. empty database with a large number of entries, accessing single
entries with a large database, etc.
In short, the results show a linear dependency on the number of
entries. This is OK for startup scenarions, but single-enrty (transactional) operations need improvement.
There are other scaling usecases, such as large configuratin "depth",
large number of requesting clients, etc.
Thanks to [Netgate](www.netgate.com) for supporting this work. Thanks to [Netgate](www.netgate.com) for supporting this work.
@ -41,7 +48,7 @@ The basic case is a large list, according to the following Yang specification:
``` ```
where `a` is a unique key and `b` is a payload, useful in replace operations. where `a` is a unique key and `b` is a payload, useful in replace operations.
XML lists with `N` elements are generated based on With this XML lists with `N` elements are generated based on
this configuration, eg for `N=10`: this configuration, eg for `N=10`:
``` ```
<y><a>0</a><b>0</b></y> <y><a>0</a><b>0</b></y>
@ -66,6 +73,7 @@ Requests are either made over the _whole_ dataset, or for one specific element.
Operations of single elements (transactions) are made in a burst of Operations of single elements (transactions) are made in a burst of
random elements, typically 100. random elements, typically 100.
## 3. Tests ## 3. Tests
All details of the setup are in the [test script](../../test/plot_perf.sh). All details of the setup are in the [test script](../../test/plot_perf.sh).
@ -76,6 +84,7 @@ All tests measure the "real" time of a command on a lightly loaded
machine using the Linux command `time(1)`. machine using the Linux command `time(1)`.
The following tests were made (for each architecture and protocol): The following tests were made (for each architecture and protocol):
* Write `N` entries into the startup configuration. The clixon_backend was started with options `-1s startup`.
* Write `N` entries in one single operation. (With an empty datastore) * Write `N` entries in one single operation. (With an empty datastore)
* Read `N` entries in one single operation. (With a datastore of `N` entries) * Read `N` entries in one single operation. (With a datastore of `N` entries)
* Commit `N` entries (With a candidate of `N` entries and empty running) * Commit `N` entries (With a candidate of `N` entries and empty running)
@ -83,7 +92,7 @@ The following tests were made (for each architecture and protocol):
* Write/Replace 1 entry (In a datastore of `N` entries) * Write/Replace 1 entry (In a datastore of `N` entries)
* Delete 1 entry (In a datastore of `N` entries) * Delete 1 entry (In a datastore of `N` entries)
The tests are made using Netconf and Restconf, except commit which is made only for Netconf. The tests are made using Netconf and Restconf, except commit which is made only for Netconf and startup where protocol is irrelevant.
### Architecture and OS ### Architecture and OS
@ -118,9 +127,12 @@ The tests were made on the following hardware, all running Ubuntu Linux:
## 4. Results ## 4. Results
### Access of the whole datastore
This section shows the results of the measurements as defined in [Tests](#tests). This section shows the results of the measurements as defined in [Tests](#tests).
### Startup
![Startup](clixon-startup.png "Startup")
### Access of the whole datastore
![Get config](clixon-get-0.png "Get config") ![Get config](clixon-get-0.png "Get config")
@ -196,11 +208,14 @@ system degrades with the size of the lists.
Examining the profiling of the most demanding Restconf PUT case, most Examining the profiling of the most demanding Restconf PUT case, most
cycles are spent on handling writing and copying the existing datastore. cycles are spent on handling writing and copying the existing datastore.
Concluding, the Note that the experiments here contains _very_ simple
data-structures. A more realistic complex example will require more
CPU effort. Ad-hoc measurement of a more complex datastructure,
generated four times the duration of the simple yang model in this work.
## 6. Future work ## 6. Future work
* Improve access of single list elements to sub-linear performance. * Improve access of individual elements to sub-linear performance.
* CLI access on large lists (not included in this study) * CLI access on large lists (not included in this study)
## 7. References ## 7. References

122
test/plot_perf.sh
View file

@ -270,9 +270,54 @@ plot(){
echo # newline echo # newline
} }
# Run an operation, iterate from <from> to <to> in increment of <step>
# Each operation do <reqs> times
# args: <op> <protocol> <from> <step> <to> <reqs> <cand> <run>
# <reqs>=0 means all in one go
startup(){
from=$1
step=$2
to=$3
mode=startup
if [ $# -ne 3 ]; then
exit "plot should be called with 3 arguments, got $#"
fi
# gnuplot file
gfile=$resdir/startup-$arch
new "Create file $gfile"
echo -n "" > $gfile
# Startup db: load with n entries
dbfile=$dir/${mode}_db
sudo touch $dbfile
sudo chmod 666 $dbfile
for (( n=$from; n<=$to; n=$n+$step )); do
new "startup-$arch $n"
new "Generate $n entries to $dbfile"
echo -n "<config><x xmlns=\"urn:example:clixon\">" > $dbfile
for (( i=0; i<$n; i++ )); do
echo -n "<y><a>$i</a><b>$i</b></y>" >> $dbfile
done
echo "</x></config>" >> $dbfile
new "Startup backend once -s $mode -f $cfg -y $fyang"
echo -n "$n " >> $gfile
{ time -p sudo $clixon_backend -F1 -D $DBG -s $mode -f $cfg -y $fyang 2> /dev/null; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $gfile
done
echo # newline
}
if $run; then if $run; then
new "test params: -f $cfg -y $fyang"
if [ $BE -ne 0 ]; then # Startup test before regular backend/restconf start since we only start
# backend a single time
startup $step $step $to
new "test params: -f $cfg -y $fyang"
if [ $BE -ne 0 ]; then
new "kill old backend" new "kill old backend"
sudo clixon_backend -zf $cfg -y $fyang sudo clixon_backend -zf $cfg -y $fyang
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
@ -280,41 +325,42 @@ if [ $BE -ne 0 ]; then
fi fi
new "start backend -s init -f $cfg -y $fyang" new "start backend -s init -f $cfg -y $fyang"
start_backend -s init -f $cfg -y $fyang start_backend -s init -f $cfg -y $fyang
fi fi
new "kill old restconf daemon" new "kill old restconf daemon"
sudo pkill -u www-data -f "/www-data/clixon_restconf" sudo pkill -u www-data -f "/www-data/clixon_restconf"
new "start restconf daemon" new "start restconf daemon"
start_restconf -f $cfg -y $fyang start_restconf -f $cfg -y $fyang
new "waiting" new "waiting"
sleep $RCWAIT sleep $RCWAIT
to=$to0 to=$to0
step=$step0 step=$step0
reqs=$reqs0 reqs=$reqs0
# Put all tests
for proto in netconf restconf; do # Put all tests
for proto in netconf restconf; do
new "$proto put all entries to candidate (restconf:running)" new "$proto put all entries to candidate (restconf:running)"
plot put $proto $step $step $to 0 0 0 # all candidate 0 running 0 plot put $proto $step $step $to 0 0 0 # all candidate 0 running 0
done done
# Get all tests # Get all tests
for proto in netconf restconf; do for proto in netconf restconf; do
new "$proto get all entries from running" new "$proto get all entries from running"
plot get $proto $step $step $to 0 n n # start w full datastore plot get $proto $step $step $to 0 n n # start w full datastore
done done
# Netconf commit all # Netconf commit all
new "Netconf commit all entries from candidate to running" new "Netconf commit all entries from candidate to running"
plot commit netconf $step $step $to 0 n 0 # candidate full running empty plot commit netconf $step $step $to 0 n 0 # candidate full running empty
# Transactions get/put/delete # Transactions get/put/delete
reqs=$reqs0 reqs=$reqs0
for proto in netconf restconf; do for proto in netconf restconf; do
new "$proto get $reqs from full database" new "$proto get $reqs from full database"
plot get $proto $step $step $to $reqs n n plot get $proto $step $step $to $reqs n n
@ -323,12 +369,12 @@ for proto in netconf restconf; do
new "$proto delete $reqs from full database(replace / alter values)" new "$proto delete $reqs from full database(replace / alter values)"
plot delete $proto $step $step $to $reqs n n plot delete $proto $step $step $to $reqs n n
done done
new "Kill restconf daemon" new "Kill restconf daemon"
stop_restconf stop_restconf
if [ $BE -ne 0 ]; then if [ $BE -ne 0 ]; then
new "Kill backend" new "Kill backend"
# Check if premature kill # Check if premature kill
pid=`pgrep -u root -f clixon_backend` pid=`pgrep -u root -f clixon_backend`
@ -337,11 +383,29 @@ if [ $BE -ne 0 ]; then
fi fi
# kill backend # kill backend
stop_backend -f $cfg stop_backend -f $cfg
fi fi
fi # if run fi # if run
if $plot; then if $plot; then
# 0. Startup
gplot=""
for a in $archs; do
gplot="$gplot \"$resdir/startup-$a\" title \"startup-$a\","
done
gnuplot -persist <<EOF
set title "Clixon startup"
set style data linespoint
set xlabel "Entries"
set ylabel "Time[s]"
set grid
set terminal $term
set yrange [*:*]
set output "$resdir/clixon-startup.$term"
plot $gplot
EOF
# 1. Get config # 1. Get config
gplot="" gplot=""
for a in $archs; do for a in $archs; do

10
test/test_perf.sh
View file

@ -5,7 +5,7 @@
s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
# Number of list/leaf-list entries in file # Number of list/leaf-list entries in file
: ${perfnr:=10000} : ${perfnr:=20000}
# Number of requests made get/put # Number of requests made get/put
: ${perfreq:=100} : ${perfreq:=100}
@ -59,6 +59,13 @@ cat <<EOF > $cfg
</clixon-config> </clixon-config>
EOF EOF
if [ $BE -ne 0 ]; then
new "kill old backend"
sudo clixon_backend -zf $cfg -y $fyang
if [ $? -ne 0 ]; then
err
fi
fi
# Try startup mode w startup # Try startup mode w startup
for mode in startup running; do for mode in startup running; do
file=$dir/${mode}_db file=$dir/${mode}_db
@ -113,7 +120,6 @@ new "netconf write large config"
expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$" expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
# Here, there are $perfnr entries in candidate # Here, there are $perfnr entries in candidate
new "netconf write large config again" new "netconf write large config again"
expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$" expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"