experiments/clixon
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

startup measurements: test, plot and report

Browse source
This commit is contained in:
Olof hagsand 2019-04-23 11:57:18 +02:00
parent 6a0628141a
commit 77ad42f1ce
4 changed files with 149 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
doc/scaling/clixon-startup.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 6.1 KiB

35
doc/scaling/large-lists.md
View file

@ -12,10 +12,17 @@ Olof Hagsand, 2019-04-17
## 1. Background ## 1. Background
Clixon can handle large configurations. Here, large number of elements Clixon can handle large configurations. Here, measurements using a
in a "flat" list is presented. There are other scaling usecases, large number of elements in a simple "flat" list is analysed. This
such as large configuratin "depth", large number of requesting includes starting up with alarge existing database; initializing an
clients, etc. empty database with a large number of entries, accessing single
entries with a large database, etc.
In short, the results show a linear dependency on the number of
entries. This is OK for startup scenarions, but single-enrty (transactional) operations need improvement.
There are other scaling usecases, such as large configuratin "depth",
large number of requesting clients, etc.
Thanks to [Netgate](www.netgate.com) for supporting this work. Thanks to [Netgate](www.netgate.com) for supporting this work.
@ -41,7 +48,7 @@ The basic case is a large list, according to the following Yang specification:
``` ```
where `a` is a unique key and `b` is a payload, useful in replace operations. where `a` is a unique key and `b` is a payload, useful in replace operations.
XML lists with `N` elements are generated based on With this XML lists with `N` elements are generated based on
this configuration, eg for `N=10`: this configuration, eg for `N=10`:
``` ```
<y><a>0</a><b>0</b></y> <y><a>0</a><b>0</b></y>
@ -66,6 +73,7 @@ Requests are either made over the _whole_ dataset, or for one specific element.
Operations of single elements (transactions) are made in a burst of Operations of single elements (transactions) are made in a burst of
random elements, typically 100. random elements, typically 100.
## 3. Tests ## 3. Tests
All details of the setup are in the [test script](../../test/plot_perf.sh). All details of the setup are in the [test script](../../test/plot_perf.sh).
@ -76,6 +84,7 @@ All tests measure the "real" time of a command on a lightly loaded
machine using the Linux command `time(1)`. machine using the Linux command `time(1)`.
The following tests were made (for each architecture and protocol): The following tests were made (for each architecture and protocol):
* Write `N` entries into the startup configuration. The clixon_backend was started with options `-1s startup`.
* Write `N` entries in one single operation. (With an empty datastore) * Write `N` entries in one single operation. (With an empty datastore)
* Read `N` entries in one single operation. (With a datastore of `N` entries) * Read `N` entries in one single operation. (With a datastore of `N` entries)
* Commit `N` entries (With a candidate of `N` entries and empty running) * Commit `N` entries (With a candidate of `N` entries and empty running)
@ -83,7 +92,7 @@ The following tests were made (for each architecture and protocol):
* Write/Replace 1 entry (In a datastore of `N` entries) * Write/Replace 1 entry (In a datastore of `N` entries)
* Delete 1 entry (In a datastore of `N` entries) * Delete 1 entry (In a datastore of `N` entries)
The tests are made using Netconf and Restconf, except commit which is made only for Netconf. The tests are made using Netconf and Restconf, except commit which is made only for Netconf and startup where protocol is irrelevant.
### Architecture and OS ### Architecture and OS
@ -118,9 +127,12 @@ The tests were made on the following hardware, all running Ubuntu Linux:
## 4. Results ## 4. Results
### Access of the whole datastore
This section shows the results of the measurements as defined in [Tests](#tests). This section shows the results of the measurements as defined in [Tests](#tests).
### Startup
![Startup](clixon-startup.png "Startup")
### Access of the whole datastore
![Get config](clixon-get-0.png "Get config") ![Get config](clixon-get-0.png "Get config")
@ -196,11 +208,14 @@ system degrades with the size of the lists.
Examining the profiling of the most demanding Restconf PUT case, most Examining the profiling of the most demanding Restconf PUT case, most
cycles are spent on handling writing and copying the existing datastore. cycles are spent on handling writing and copying the existing datastore.
Concluding, the Note that the experiments here contains _very_ simple
data-structures. A more realistic complex example will require more
CPU effort. Ad-hoc measurement of a more complex datastructure,
generated four times the duration of the simple yang model in this work.
## 6. Future work ## 6. Future work
* Improve access of single list elements to sub-linear performance. * Improve access of individual elements to sub-linear performance.
* CLI access on large lists (not included in this study) * CLI access on large lists (not included in this study)
## 7. References ## 7. References

168
test/plot_perf.sh
View file

@ -270,78 +270,142 @@ plot(){
echo # newline echo # newline
} }
# Run an operation, iterate from <from> to <to> in increment of <step>
# Each operation do <reqs> times
# args: <op> <protocol> <from> <step> <to> <reqs> <cand> <run>
# <reqs>=0 means all in one go
startup(){
from=$1
step=$2
to=$3
mode=startup
if [ $# -ne 3 ]; then
exit "plot should be called with 3 arguments, got $#"
fi
# gnuplot file
gfile=$resdir/startup-$arch
new "Create file $gfile"
echo -n "" > $gfile
# Startup db: load with n entries
dbfile=$dir/${mode}_db
sudo touch $dbfile
sudo chmod 666 $dbfile
for (( n=$from; n<=$to; n=$n+$step )); do
new "startup-$arch $n"
new "Generate $n entries to $dbfile"
echo -n "<config><x xmlns=\"urn:example:clixon\">" > $dbfile
for (( i=0; i<$n; i++ )); do
echo -n "<y><a>$i</a><b>$i</b></y>" >> $dbfile
done
echo "</x></config>" >> $dbfile
new "Startup backend once -s $mode -f $cfg -y $fyang"
echo -n "$n " >> $gfile
{ time -p sudo $clixon_backend -F1 -D $DBG -s $mode -f $cfg -y $fyang 2> /dev/null; } 2>&1 | awk '/real/ {print $2}' | tr , . >> $gfile
done
echo # newline
}
if $run; then if $run; then
new "test params: -f $cfg -y $fyang"
if [ $BE -ne 0 ]; then # Startup test before regular backend/restconf start since we only start
new "kill old backend" # backend a single time
sudo clixon_backend -zf $cfg -y $fyang startup $step $step $to
if [ $? -ne 0 ]; then
err new "test params: -f $cfg -y $fyang"
if [ $BE -ne 0 ]; then
new "kill old backend"
sudo clixon_backend -zf $cfg -y $fyang
if [ $? -ne 0 ]; then
err
fi
new "start backend -s init -f $cfg -y $fyang"
start_backend -s init -f $cfg -y $fyang
fi fi
new "start backend -s init -f $cfg -y $fyang"
start_backend -s init -f $cfg -y $fyang
fi
new "kill old restconf daemon" new "kill old restconf daemon"
sudo pkill -u www-data -f "/www-data/clixon_restconf" sudo pkill -u www-data -f "/www-data/clixon_restconf"
new "start restconf daemon" new "start restconf daemon"
start_restconf -f $cfg -y $fyang start_restconf -f $cfg -y $fyang
new "waiting" new "waiting"
sleep $RCWAIT sleep $RCWAIT
to=$to0 to=$to0
step=$step0 step=$step0
reqs=$reqs0 reqs=$reqs0
# Put all tests
for proto in netconf restconf; do
new "$proto put all entries to candidate (restconf:running)"
plot put $proto $step $step $to 0 0 0 # all candidate 0 running 0
done
# Get all tests # Put all tests
for proto in netconf restconf; do for proto in netconf restconf; do
new "$proto get all entries from running" new "$proto put all entries to candidate (restconf:running)"
plot get $proto $step $step $to 0 n n # start w full datastore plot put $proto $step $step $to 0 0 0 # all candidate 0 running 0
done done
# Netconf commit all # Get all tests
new "Netconf commit all entries from candidate to running" for proto in netconf restconf; do
plot commit netconf $step $step $to 0 n 0 # candidate full running empty new "$proto get all entries from running"
plot get $proto $step $step $to 0 n n # start w full datastore
done
# Transactions get/put/delete # Netconf commit all
reqs=$reqs0 new "Netconf commit all entries from candidate to running"
for proto in netconf restconf; do plot commit netconf $step $step $to 0 n 0 # candidate full running empty
new "$proto get $reqs from full database"
plot get $proto $step $step $to $reqs n n
new "$proto put $reqs to full database(replace / alter values)" # Transactions get/put/delete
plot put $proto $step $step $to $reqs n n reqs=$reqs0
for proto in netconf restconf; do
new "$proto get $reqs from full database"
plot get $proto $step $step $to $reqs n n
new "$proto delete $reqs from full database(replace / alter values)" new "$proto put $reqs to full database(replace / alter values)"
plot delete $proto $step $step $to $reqs n n plot put $proto $step $step $to $reqs n n
done
new "Kill restconf daemon" new "$proto delete $reqs from full database(replace / alter values)"
stop_restconf plot delete $proto $step $step $to $reqs n n
done
if [ $BE -ne 0 ]; then new "Kill restconf daemon"
new "Kill backend" stop_restconf
# Check if premature kill
pid=`pgrep -u root -f clixon_backend` if [ $BE -ne 0 ]; then
if [ -z "$pid" ]; then new "Kill backend"
err "backend already dead" # Check if premature kill
pid=`pgrep -u root -f clixon_backend`
if [ -z "$pid" ]; then
err "backend already dead"
fi
# kill backend
stop_backend -f $cfg
fi fi
# kill backend
stop_backend -f $cfg
fi
fi # if run fi # if run
if $plot; then if $plot; then
# 0. Startup
gplot=""
for a in $archs; do
gplot="$gplot \"$resdir/startup-$a\" title \"startup-$a\","
done
gnuplot -persist <<EOF
set title "Clixon startup"
set style data linespoint
set xlabel "Entries"
set ylabel "Time[s]"
set grid
set terminal $term
set yrange [*:*]
set output "$resdir/clixon-startup.$term"
plot $gplot
EOF
# 1. Get config # 1. Get config
gplot="" gplot=""
for a in $archs; do for a in $archs; do

10
test/test_perf.sh
View file

@ -5,7 +5,7 @@
s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
# Number of list/leaf-list entries in file # Number of list/leaf-list entries in file
: ${perfnr:=10000} : ${perfnr:=20000}
# Number of requests made get/put # Number of requests made get/put
: ${perfreq:=100} : ${perfreq:=100}
@ -59,6 +59,13 @@ cat <<EOF > $cfg
</clixon-config> </clixon-config>
EOF EOF
if [ $BE -ne 0 ]; then
new "kill old backend"
sudo clixon_backend -zf $cfg -y $fyang
if [ $? -ne 0 ]; then
err
fi
fi
# Try startup mode w startup # Try startup mode w startup
for mode in startup running; do for mode in startup running; do
file=$dir/${mode}_db file=$dir/${mode}_db
@ -113,7 +120,6 @@ new "netconf write large config"
expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$" expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
# Here, there are $perfnr entries in candidate # Here, there are $perfnr entries in candidate
new "netconf write large config again" new "netconf write large config again"
expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$" expecteof_file "/usr/bin/time -f %e $clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"