experiments/clixon
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Further optimizions and bugfixing of that

Browse source
This commit is contained in:
Olof hagsand 2019-04-16 12:09:21 +02:00
parent 71da2ac6cb
commit d46ca41c8b
11 changed files with 482 additions and 352 deletions
Download patch file Download diff file Expand all files Collapse all files

1
README.md
View file

@ -26,6 +26,7 @@ support.
* [Runtime](#runtime) * [Runtime](#runtime)
* [Clixon project page](http://www.clicon.org) * [Clixon project page](http://www.clicon.org)
* [Tests and CI](test/README.md) * [Tests and CI](test/README.md)
* [Scaling: large lists](doc/large-lists.md)
* [Containers](docker/README.md) * [Containers](docker/README.md)
* [Roadmap](doc/ROADMAP.md) * [Roadmap](doc/ROADMAP.md)
* [Reference manual](#reference) * [Reference manual](#reference)

134
doc/large-lists.md Normal file
View file

@ -0,0 +1,134 @@
# Large lists in Clixon
* [Background](#background)
* [Overview](#overview)
* [Test descriptions]#test-descriptions)
## Background
Clixon is a configuration management tool. In this paper the case of
a large number of "flat" list and leaf-list entries are investigated.
There may be other scaling usecases, such as large configuratin
"depth", large number of requesting clients, etc. However, these are
not investigated here.
## Overview
The basic case is a large list, according to the following Yang specification:
```
list y {
key "a";
leaf a {
type int32;
}
leaf b {
type string;
}
}
```
where `a` is a unique key and `b` is a payload, useful in replace operations.
There is also a leaf-list as follows:
```
leaf-list c {
type string;
}
```
XML lists with `N` elements are generated based on
this configuration, eg for `N=10`:
```
<y><a>0</a><b>0</b></y>
<y><a>1</a><b>1</b></y>
<y><a>2</a><b>2</b></y>
<y><a>3</a><b>3</b></y>
<y><a>4</a><b>4</b></y>
<y><a>5</a><b>5</b></y>
<y><a>6</a><b>6</b></y>
<y><a>7</a><b>7</b></y>
<y><a>8</a><b>8</b></y>
<y><a>9</a><b>9</b></y>
```
Requests are made using a random function, a request on the list above will on the form:
```
curl -G http://localhost/restconf/data/y=(rnd%$N)
```
## Test descriptions
### Limitations
Test were not made using CLI interaction.
### Setup
The setup consisted of the following components running on the same machine:
* A clixon backend daemon
* A clixon restconf daemon
* An nginx daemon daemon
* A netconf client program
* curl client
* A bash terminal and test script [plot_perf.sh](../test/plot_perf.sh)
* Gnuplot for generating plots
### Config file
The following Clixon config file was used:
```
<clixon-config xmlns="http://clicon.org/config">
<CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
<CLICON_YANG_DIR>$dir</CLICON_YANG_DIR>
<CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
<CLICON_YANG_MODULE_MAIN>scaling</CLICON_YANG_MODULE_MAIN>
<CLICON_SOCK>/usr/local/var/example/example.sock</CLICON_SOCK>
<CLICON_BACKEND_PIDFILE>/usr/local/var/example/example.pidfile</CLICON_BACKEND_PIDFILE>
<CLICON_RESTCONF_PRETTY>false</CLICON_RESTCONF_PRETTY>
<CLICON_XMLDB_DIR>$dir</CLICON_XMLDB_DIR>
<CLICON_XMLDB_PRETTY>false</CLICON_XMLDB_PRETTY>
</clixon-config>
```
where `$dir` and `$cfg`are local files. For more info see [plot_perf.sh].
### Testcases
All tests measure the "real" time of a command on a lightly loaded
machine using the Linux command `time(1)`.
The following tests were made (for each architecture and protocol):
* Write `N` entries in one single operation. (With an empty datastore)
* Read `N` entries in one single operation. (With a datastore of `N` entries)
* Commit `N` entries (With a candidate of `N` entries and empty running)
* Read 1 entry (In a datastore of `N` entries)
* Write/Replace 1 entry (In a datastore of `N` entries)
* Delete 1 entry (In a datastore of `N` entries)
### Protocols
The tests are made using:
* Netconf[RFC6241] and
* Restconf[RFC8040].
Notably, CLI tests are for future study.
### Architectures
The tests were made on the following hardware, all running Ubuntu Linux:
* [i686] dual Intel Core Duo processor (IBM Thinkpad X60), 3GB memory
* arm 32-bit (Raspberry PI 3)
* x86 64-bit (Intel NUC)
### Operating systems
On i686:
```
Linux version 4.4.0-143-generic (buildd@lgw01-amd64-037) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #169-Ubuntu SMP Thu Feb 7 07:56:51 UTC 2019
```
## Results
## References
[RFC6241](https://tools.ietf.org/html/rfc6241) "Network Configuration Protocol (NETCONF)"
[RFC8040](https://tools.ietf.org/html/rfc8040) "RESTCONF Protocol"
[i686](https://ark.intel.com/content/www/us/en/ark/products/27235/intel-core-duo-processor-t2400-2m-cache-1-83-ghz-667-mhz-fsb.html)
[plot_perf.sh](../test/plot_perf.sh) Test script

7
docker/system/README.md
View file

@ -54,3 +54,10 @@ To check status and then kill it:
``` ```
You trigger the test scripts inside the container using `make test`. You trigger the test scripts inside the container using `make test`.
## Changing code
If you want to edit clixon code so it runs in the container?
You either
(1) "persistent": make your changes in the actual clixon code and commit; make clean to remove the local clone; make test again
(2) "volatile" edit the local clone; make test.

2
lib/src/clixon_xml.c
View file

@ -814,7 +814,7 @@ xml_cv_set(cxobj *x,
* name "name". * name "name".
* *
* @param[in] x_up Base XML object * @param[in] x_up Base XML object
* @param[in] name shell wildcard pattern to match with node name * @param[in] name Node name
* *
* @retval xmlobj if found. * @retval xmlobj if found.
* @retval NULL if no such node found. * @retval NULL if no such node found.

8
lib/src/clixon_xml_map.c
View file

@ -2021,6 +2021,7 @@ api_path2xml_vec(char **vec,
cxobj *x = NULL; cxobj *x = NULL;
yang_stmt *y = NULL; yang_stmt *y = NULL;
yang_stmt *ymod; yang_stmt *ymod;
yang_stmt *ykey;
char *namespace = NULL; char *namespace = NULL;
if ((nodeid = vec[0]) == NULL || strlen(nodeid)==0){ if ((nodeid = vec[0]) == NULL || strlen(nodeid)==0){
@ -2104,7 +2105,12 @@ api_path2xml_vec(char **vec,
/* Create keys */ /* Create keys */
while ((cvi = cvec_each(cvk, cvi)) != NULL) { while ((cvi = cvec_each(cvk, cvi)) != NULL) {
keyname = cv_string_get(cvi); keyname = cv_string_get(cvi);
if ((xn = xml_new(keyname, x, NULL)) == NULL) if ((ykey = yang_find(y, Y_LEAF, keyname)) == NULL){
clicon_err(OE_XML, 0, "List statement \"%s\" has no key leaf \"%s\"",
yang_argument_get(y), keyname);
goto done;
}
if ((xn = xml_new(keyname, x, ykey)) == NULL)
goto done; goto done;
xml_type_set(xn, CX_ELMNT); xml_type_set(xn, CX_ELMNT);
if ((xb = xml_new("body", xn, NULL)) == NULL) if ((xb = xml_new("body", xn, NULL)) == NULL)

261
lib/src/clixon_xml_sort.c
View file

@ -89,7 +89,8 @@ xml_cv_cache(cxobj *x,
uint8_t fraction = 0; uint8_t fraction = 0;
char *body; char *body;
body = xml_body(x); if ((body = xml_body(x)) == NULL)
body="";
if ((cv = xml_cv(x)) != NULL) if ((cv = xml_cv(x)) != NULL)
goto ok; goto ok;
if ((y = xml_spec(x)) == NULL) if ((y = xml_spec(x)) == NULL)
@ -191,13 +192,21 @@ xml_child_spec(cxobj *x,
* @param[in] x1 object 1 * @param[in] x1 object 1
* @param[in] x2 object 2 * @param[in] x2 object 2
* @param[in] same If set, x1 and x2 are member of same parent & enumeration * @param[in] same If set, x1 and x2 are member of same parent & enumeration
* is used * is used (see explanation below)
* @retval 0 If equal * @retval 0 If equal
* @retval <0 If x1 is less than x2 * @retval <0 If x1 is less than x2
* @retval >0 If x1 is greater than x2 * @retval >0 If x1 is greater than x2
* @see xml_cmp1 Similar, but for one object * @see xml_cmp1 Similar, but for one object
*
* There are distinct calls for this function:
* 1. For sorting in an existing list of XML children
* 2. For searching of an existing element in a list
* In the first case, there is a special case for "ordered-by-user", where
* if they have the same yang-spec, the existing order is used as tie-breaker.
* In other words, if order-by-system, or if the case (2) above, the existing
* order is ignored and the actual xml element contents is examined.
* @note empty value/NULL is smallest value * @note empty value/NULL is smallest value
* @note xml_enumerate_children must have been called prior to this call * @note some error cases return as -1 (qsort cant handle errors)
* @note some error cases return as -1 (qsort cant handle errors) * @note some error cases return as -1 (qsort cant handle errors)
*/ */
int int
@ -252,10 +261,12 @@ xml_cmp(cxobj *x1,
/* Now y1==y2, same Yang spec, can only be list or leaf-list, /* Now y1==y2, same Yang spec, can only be list or leaf-list,
* But first check exceptions, eg config false or ordered-by user * But first check exceptions, eg config false or ordered-by user
* otherwise sort according to key * otherwise sort according to key
* If the two elements are in the same list, and they are ordered-by user
* then do not look more into equivalence, use the enumeration in the
* existing list.
*/ */
if (yang_config(y1)==0 || if (same &&
yang_find(y1, Y_ORDERED_BY, "user") != NULL){ (yang_config(y1)==0 || yang_find(y1, Y_ORDERED_BY, "user") != NULL)){
if (same)
equal = nr1-nr2; equal = nr1-nr2;
goto done; /* Ordered by user or state data : maintain existing order */ goto done; /* Ordered by user or state data : maintain existing order */
} }
@ -289,8 +300,10 @@ xml_cmp(cxobj *x1,
else{ else{
if (xml_cv_cache(x1b, &cv1) < 0) /* error case */ if (xml_cv_cache(x1b, &cv1) < 0) /* error case */
goto done; goto done;
assert(cv1);
if (xml_cv_cache(x2b, &cv2) < 0) /* error case */ if (xml_cv_cache(x2b, &cv2) < 0) /* error case */
goto done; goto done;
assert(cv2);
if ((equal = cv_cmp(cv1, cv2)) != 0) if ((equal = cv_cmp(cv1, cv2)) != 0)
goto done; goto done;
} }
@ -315,92 +328,6 @@ xml_cmp_qsort(const void* arg1,
return xml_cmp(*(struct xml**)arg1, *(struct xml**)arg2, 1); return xml_cmp(*(struct xml**)arg1, *(struct xml**)arg2, 1);
} }
/*! Compare xml object
* @param[in] x XML node to compare with
* @param[in] y The yang spec of x
* @param[in] name Name to compare with x
* @param[in] keyword Yang keyword (stmt type) to compare w x/y
* @param[in] keynr Length of keyvec/keyval vector when applicable
* @param[in] keyvec Array of of yang key identifiers
* @param[in] keyval Array of of yang key values
* @param[out] userorder If set, this yang order is user ordered, linear search
* @retval 0 If equal (or userorder set)
* @retval <0 if arg1 is less than arg2
* @retval >0 if arg1 is greater than arg2
* @see xml_cmp Similar, but for two objects
* @note Does not care about y type of value as xml_cmp
*/
static int
xml_cmp1(cxobj *x,
yang_stmt *y,
char *name,
enum rfc_6020 keyword,
int keynr,
char **keyvec,
char **keyval,
cg_var **keycvec,
int *userorder)
{
char *b;
cxobj *xb;
int i;
char *keyname;
char *key;
int match = 0;
cg_var *cv;
/* state data = userorder */
if (userorder && yang_config(y)==0)
*userorder=1;
/* Check if same yang spec (order in yang stmt list) */
switch (keyword){
case Y_CONTAINER: /* Match with name */
case Y_LEAF: /* Match with name */
match = strcmp(name, xml_name(x));
break;
case Y_LEAF_LIST: /* Match with name and value */
if (userorder && yang_find(y, Y_ORDERED_BY, "user") != NULL)
*userorder=1;
if ((b=xml_body(x)) == NULL)
match = 1;
else{
if (keycvec[0]){
if (xml_cv_cache(x, &cv) < 0) /* error case */
goto done;
match = cv_cmp(keycvec[0], cv);
}
else
match = strcmp(keyval[0], b);
}
break;
case Y_LIST: /* Match with array of key values */
if (userorder && yang_find(y, Y_ORDERED_BY, "user") != NULL)
*userorder=1;
/* All must match */
for (i=0; i<keynr; i++){
keyname = keyvec[i];
key = keyval[i];
if ((xb = xml_find(x, keyname)) == NULL)
break; /* error case */
if ((b = xml_body(xb)) == NULL)
break; /* error case */
if (xml_cv_cache(xb, &cv) < 0) /* error case */
goto done;
if (keycvec[i]){
if ((match = cv_cmp(keycvec[i], cv)) != 0)
break;
}
else
if ((match = strcmp(key, b)) != 0)
break;
}
break;
default:
break;
}
done:
return match;
}
/*! Sort children of an XML node /*! Sort children of an XML node
* Assume populated by yang spec. * Assume populated by yang spec.
@ -428,40 +355,37 @@ xml_sort(cxobj *x,
/*! Special case search for ordered-by user where linear sort is used /*! Special case search for ordered-by user where linear sort is used
*/ */
static cxobj * static cxobj *
xml_search_userorder(cxobj *x0, xml_search_userorder(cxobj *xp,
cxobj *x1,
yang_stmt *y, yang_stmt *y,
char *name,
int yangi, int yangi,
int mid, int mid)
enum rfc_6020 keyword,
int keynr,
char **keyvec,
char **keyval,
cg_var **keycvec)
{ {
int i; int i;
cxobj *xc; cxobj *xc;
for (i=mid+1; i<xml_child_nr(x0); i++){ /* First increment */ for (i=mid+1; i<xml_child_nr(xp); i++){ /* First increment */
xc = xml_child_i(x0, i); xc = xml_child_i(xp, i);
y = xml_spec(xc); y = xml_spec(xc);
if (yangi!=yang_order(y)) if (yangi!=yang_order(y))
break; break;
if (xml_cmp1(xc, y, name, keyword, keynr, keyvec, keyval, keycvec, NULL) == 0) if (xml_cmp(xc, x1, 0) == 0)
return xc; return xc;
} }
for (i=mid-1; i>=0; i--){ /* Then decrement */ for (i=mid-1; i>=0; i--){ /* Then decrement */
xc = xml_child_i(x0, i); xc = xml_child_i(xp, i);
y = xml_spec(xc); y = xml_spec(xc);
if (yangi!=yang_order(y)) if (yangi!=yang_order(y))
break; break;
if (xml_cmp1(xc, y, name, keyword, keynr, keyvec, keyval, keycvec, NULL) == 0) if (xml_cmp(xc, x1, 0) == 0)
return xc; return xc;
} }
return NULL; /* Not found */ return NULL; /* Not found */
} }
/*! /*!
* @param[in] xp Parent xml node.
* @param[in] yangi Yang order * @param[in] yangi Yang order
* @param[in] keynr Length of keyvec/keyval vector when applicable * @param[in] keynr Length of keyvec/keyval vector when applicable
* @param[in] keyvec Array of of yang key identifiers * @param[in] keyvec Array of of yang key identifiers
@ -470,14 +394,10 @@ xml_search_userorder(cxobj *x0,
* @param[in] upper Lower bound of childvec search interval * @param[in] upper Lower bound of childvec search interval
*/ */
static cxobj * static cxobj *
xml_search1(cxobj *x0, xml_search1(cxobj *xp,
char *name, cxobj *x1,
int userorder,
int yangi, int yangi,
enum rfc_6020 keyword,
int keynr,
char **keyvec,
char **keyval,
cg_var **keycvec,
int low, int low,
int upper) int upper)
{ {
@ -485,61 +405,65 @@ xml_search1(cxobj *x0,
int cmp; int cmp;
cxobj *xc; cxobj *xc;
yang_stmt *y; yang_stmt *y;
int userorder= 0;
if (upper < low) if (upper < low)
return NULL; /* not found */ return NULL; /* not found */
mid = (low + upper) / 2; mid = (low + upper) / 2;
if (mid >= xml_child_nr(x0)) /* beyond range */ if (mid >= xml_child_nr(xp)) /* beyond range */
return NULL; return NULL;
xc = xml_child_i(x0, mid); xc = xml_child_i(xp, mid);
if ((y = xml_spec(xc)) == NULL) if ((y = xml_spec(xc)) == NULL)
return NULL; return NULL;
cmp = yangi-yang_order(y); cmp = yangi-yang_order(y);
/* Here is right yang order == same yang? */ /* Here is right yang order == same yang? */
if (cmp == 0){ if (cmp == 0){
cmp = xml_cmp1(xc, y, name, keyword, keynr, keyvec, keyval, keycvec, &userorder); if (userorder){
if (userorder && cmp) /* Look inside this yangi order */ return xml_search_userorder(xp, x1, y, yangi, mid);
return xml_search_userorder(x0, y, name, yangi, mid, keyword, keynr, keyvec, keyval, keycvec); }
else /* Ordered by system */
cmp = xml_cmp(x1, xc, 0);
} }
if (cmp == 0) if (cmp == 0)
return xc; return xc;
else if (cmp < 0) else if (cmp < 0)
return xml_search1(x0, name, yangi, keyword, return xml_search1(xp, x1, userorder, yangi, low, mid-1);
keynr, keyvec, keyval, keycvec, low, mid-1);
else else
return xml_search1(x0, name, yangi, keyword, return xml_search1(xp, x1, userorder, yangi, mid+1, upper);
keynr, keyvec, keyval, keycvec, mid+1, upper);
return NULL; return NULL;
} }
/*! Find XML children using binary search /*! Find XML child under xp matching x1 using binary search
* @param[in] yangi yang child order * @param[in] xp Parent xml node.
* @param[in] yangi Yang child order
* @param[in] keynr Length of keyvec/keyval vector when applicable * @param[in] keynr Length of keyvec/keyval vector when applicable
* @param[in] keyvec Array of of yang key identifiers * @param[in] keyvec Array of of yang key identifiers
* @param[in] keyval Array of of yang key values * @param[in] keyval Array of of yang key values
*/ */
static cxobj * static cxobj *
xml_search(cxobj *x0, xml_search(cxobj *xp,
char *name, cxobj *x1,
int yangi, yang_stmt *yc)
enum rfc_6020 keyword,
int keynr,
char **keyvec,
char **keyval,
cg_var **keycvec)
{ {
cxobj *xa; cxobj *xa;
int low = 0; int low = 0;
int upper = xml_child_nr(x0); int upper = xml_child_nr(xp);
int userorder=0;
cxobj *xret = NULL;
int yangi;
/* Assume if there are any attributes, they are first in the list, mask /* Assume if there are any attributes, they are first in the list, mask
them by raising low to skip them */ them by raising low to skip them */
for (low=0; low<upper; low++) for (low=0; low<upper; low++)
if ((xa = xml_child_i(x0, low)) == NULL || xml_type(xa)!=CX_ATTR) if ((xa = xml_child_i(xp, low)) == NULL || xml_type(xa)!=CX_ATTR)
break; break;
return xml_search1(x0, name, yangi, keyword, keynr, keyvec, keyval, keycvec, /* Find if non-config and if ordered-by-user */
low, upper); if (yang_config(yc)==0)
userorder = 1;
else if (yc->ys_keyword == Y_LIST || yc->ys_keyword == Y_LEAF_LIST)
userorder = (yang_find(yc, Y_ORDERED_BY, "user") != NULL);
yangi = yang_order(yc);
xret = xml_search1(xp, x1, userorder, yangi, low, upper);
return xret;
} }
/*! Insert xn in xp:s sorted child list /*! Insert xn in xp:s sorted child list
@ -604,12 +528,6 @@ xml_insert2(cxobj *xp,
*/ */
} }
if (low +1 == upper){ /* termination criterium */ if (low +1 == upper){ /* termination criterium */
#if 0
if (xml_child_nr(xp) <= mid+1){
retval = mid;
goto done;
}
#endif
if (cmp<0) { if (cmp<0) {
retval = mid; retval = mid;
goto done; goto done;
@ -733,15 +651,8 @@ match_base_child(cxobj *x0,
int retval = -1; int retval = -1;
cvec *cvk = NULL; /* vector of index keys */ cvec *cvk = NULL; /* vector of index keys */
cg_var *cvi; cg_var *cvi;
char *b;
cxobj *xb; cxobj *xb;
char *keyname; char *keyname;
char keynr = 0;
char **keyval = NULL;
char **keyvec = NULL;
cg_var **keycvec = NULL;
int i;
int yorder;
cxobj *x0c = NULL; cxobj *x0c = NULL;
yang_stmt *y0c; yang_stmt *y0c;
yang_stmt *y0p; yang_stmt *y0p;
@ -767,19 +678,10 @@ match_base_child(cxobj *x0,
case Y_LEAF: /* Equal regardless */ case Y_LEAF: /* Equal regardless */
break; break;
case Y_LEAF_LIST: /* Match with name and value */ case Y_LEAF_LIST: /* Match with name and value */
keynr = 1; if (xml_body(x1c) == NULL){ /* Treat as empty string */
if ((keyval = calloc(keynr+1, sizeof(char*))) == NULL){ // assert(0);
clicon_err(OE_UNIX, errno, "calloc");
goto done;
}
if ((keyval[0] = xml_body(x1c)) == NULL)
goto ok; goto ok;
if ((keycvec = calloc(keynr+1, sizeof(cg_var*))) == NULL){
clicon_err(OE_UNIX, errno, "calloc");
goto done;
} }
if (xml_cv_cache(x1c, &keycvec[0]) < 0) /* error case */
goto done;
break; break;
case Y_LIST: /* Match with key values */ case Y_LIST: /* Match with key values */
cvk = yc->ys_cvec; /* Use Y_LIST cache, see ys_populate_list() */ cvk = yc->ys_cvec; /* Use Y_LIST cache, see ys_populate_list() */
@ -787,51 +689,22 @@ match_base_child(cxobj *x0,
* Then create two vectors one with names and one with values of x1c, * Then create two vectors one with names and one with values of x1c,
* ec: keyvec: [a,b,c] keyval: [1,2,3] * ec: keyvec: [a,b,c] keyval: [1,2,3]
*/ */
cvi = NULL; keynr = 0; cvi = NULL;
while ((cvi = cvec_each(cvk, cvi)) != NULL)
keynr++;
if ((keyval = calloc(keynr+1, sizeof(char*))) == NULL){
clicon_err(OE_UNIX, errno, "calloc");
goto done;
}
if ((keyvec = calloc(keynr+1, sizeof(char*))) == NULL){
clicon_err(OE_UNIX, errno, "calloc");
goto done;
}
if ((keycvec = calloc(keynr+1, sizeof(char*))) == NULL){
clicon_err(OE_UNIX, errno, "calloc");
goto done;
}
cvi = NULL; i = 0;
while ((cvi = cvec_each(cvk, cvi)) != NULL) { while ((cvi = cvec_each(cvk, cvi)) != NULL) {
keyname = cv_string_get(cvi); keyname = cv_string_get(cvi);
keyvec[i] = keyname; // keyvec[i] = keyname;
if ((xb = xml_find(x1c, keyname)) == NULL) if ((xb = xml_find(x1c, keyname)) == NULL){
goto ok; goto ok;
if ((b = xml_body(xb)) == NULL)
goto ok;
keyval[i] = b;
if (xml_cv_cache(xb, &keycvec[i]) < 0) /* error case */
goto done;
i++;
} }
break; }
default: default:
break; break;
} }
/* Get match. */ /* Get match. */
yorder = yang_order(yc); x0c = xml_search(x0, x1c, yc);
x0c = xml_search(x0, xml_name(x1c), yorder, yc->ys_keyword, keynr, keyvec, keyval, keycvec);
ok: ok:
*x0cp = x0c; *x0cp = x0c;
retval = 0; retval = 0;
done:
if (keyval)
free(keyval);
if (keyvec)
free(keyvec);
if (keycvec)
free(keycvec);
return retval; return retval;
} }

8
lib/src/clixon_yang_module.c
View file

@ -159,7 +159,8 @@ yang_modules_revision(clicon_handle h)
} }
/*! Actually build the yang modules state XML tree /*! Actually build the yang modules state XML tree
*/ * @see RFC7895
*/
static int static int
yms_build(clicon_handle h, yms_build(clicon_handle h,
yang_stmt *yspec, yang_stmt *yspec,
@ -197,8 +198,11 @@ yms_build(clicon_handle h,
cprintf(cb,"<name>%s</name>", ymod->ys_argument); cprintf(cb,"<name>%s</name>", ymod->ys_argument);
if ((ys = yang_find(ymod, Y_REVISION, NULL)) != NULL) if ((ys = yang_find(ymod, Y_REVISION, NULL)) != NULL)
cprintf(cb,"<revision>%s</revision>", ys->ys_argument); cprintf(cb,"<revision>%s</revision>", ys->ys_argument);
else else{
/* RFC7895 1 If no (such) revision statement exists, the module's or
submodule's revision is the zero-length string. */
cprintf(cb,"<revision></revision>"); cprintf(cb,"<revision></revision>");
}
if ((ys = yang_find(ymod, Y_NAMESPACE, NULL)) != NULL) if ((ys = yang_find(ymod, Y_NAMESPACE, NULL)) != NULL)
cprintf(cb,"<namespace>%s</namespace>", ys->ys_argument); cprintf(cb,"<namespace>%s</namespace>", ys->ys_argument);
else else

2
test/lib.sh
View file

@ -221,7 +221,7 @@ expectfn(){
if [ $r != $retval ]; then if [ $r != $retval ]; then
echo -e "\e[31m\nError ($r != $retval) in Test$testnr [$testname]:" echo -e "\e[31m\nError ($r != $retval) in Test$testnr [$testname]:"
echo -e "\e[0m:" echo -e "\e[0m:"
return exit -1
fi fi
# if [ $r != 0 ]; then # if [ $r != 0 ]; then
# return # return

345
test/plot_perf.sh
View file

@ -1,6 +1,8 @@
#!/bin/bash #!/bin/bash
# Transactions per second for large lists read/write plotter using gnuplot # Transactions per second for large lists read/write plotter using gnuplot
# What do I want to plot? # What do I want to plot?
# First: on i32, i64, arm32
# PART 1: Basic load
# 1. How long to write 100K entries? # 1. How long to write 100K entries?
# - netconf / restconf # - netconf / restconf
# - list / leaf-list # - list / leaf-list
@ -9,28 +11,36 @@
# - list / leaf-list # - list / leaf-list
# 3. How long to commit 100K entries? (netconf) # 3. How long to commit 100K entries? (netconf)
# - list / leaf-list # - list / leaf-list
# 4. In database 100K entries. How many read operations per second? #
# PART 2: Load 100K entries. Commit.
# 4. How many read operations per second?
# - netconf/ restconf # - netconf/ restconf
# - list / leaf-list # - list / leaf-list
# 5. 100K entries. How many write operations per second? # 5. How many write operations per second?
# - netconf / restconf # - netconf / restconf
# - list / leaf-list # - list / leaf-list
# 6. 100K entries. How may delete operations per second? # 6. How may delete operations per second?
# - netconf / restconf # - netconf / restconf
# - list / leaf-list # - list / leaf-list
# The script uses bash builtin "time" command which is somewhat difficult to
# understand. See: https://linux.die.net/man/1/bash # pipelines
# You essentially have to do: { time stuff; } 2>&1
# See: https://stackoverflow.com/questions/26784870/parsing-the-output-of-bashs-time-builtin
# Magic line must be first in script (see README.md) # Magic line must be first in script (see README.md)
s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
#max=2000 # Nr of db entries # op from step to reqs
#step=200 to=1000
#reqs=500 step=100
reqs=100
# Global variables # Global variables
APPNAME=example APPNAME=example
cfg=$dir/plot-conf.xml cfg=$dir/plot-conf.xml
fyang=$dir/plot.yang fyang=$dir/plot.yang
fconfig=$dir/config fxml=$dir/data.xml
fjson=$dir/data.json
# For memcheck # For memcheck
# clixon_netconf="valgrind --leak-check=full --show-leak-kinds=all clixon_netconf" # clixon_netconf="valgrind --leak-check=full --show-leak-kinds=all clixon_netconf"
@ -46,7 +56,7 @@ module scaling{
list y { list y {
key "a"; key "a";
leaf a { leaf a {
type string; type uint32;
} }
leaf b { leaf b {
type string; type string;
@ -65,107 +75,199 @@ cat <<EOF > $cfg
<CLICON_YANG_DIR>$dir</CLICON_YANG_DIR> <CLICON_YANG_DIR>$dir</CLICON_YANG_DIR>
<CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR> <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
<CLICON_YANG_MODULE_MAIN>scaling</CLICON_YANG_MODULE_MAIN> <CLICON_YANG_MODULE_MAIN>scaling</CLICON_YANG_MODULE_MAIN>
<CLICON_SOCK>/usr/local/var/$APPNAME/$APPNAME.sock</CLICON_SOCK> <CLICON_SOCK>/usr/local/var/example/example.sock</CLICON_SOCK>
<CLICON_BACKEND_PIDFILE>/usr/local/var/example/$APPNAME.pidfile</CLICON_BACKEND_PIDFILE> <CLICON_BACKEND_PIDFILE>/usr/local/var/example/example.pidfile</CLICON_BACKEND_PIDFILE>
<CLICON_RESTCONF_PRETTY>false</CLICON_RESTCONF_PRETTY> <CLICON_RESTCONF_PRETTY>false</CLICON_RESTCONF_PRETTY>
<CLICON_XMLDB_DIR>$dir</CLICON_XMLDB_DIR> <CLICON_XMLDB_DIR>$dir</CLICON_XMLDB_DIR>
<CLICON_XMLDB_PRETTY>false</CLICON_XMLDB_PRETTY> <CLICON_XMLDB_PRETTY>false</CLICON_XMLDB_PRETTY>
</clixon-config> </clixon-config>
EOF EOF
# Run function # Generate file with n entries
# args: <i> <reqs> <mode> # argument: <n> <proto>
# where mode is one of: genfile(){
# readlist writelist restreadlist restwritelist if [ $2 = netconf ]; then
runfn(){ echo -n "<rpc><edit-config><target><candidate/></target><default-operation>replace</default-operation><config><x xmlns=\"urn:example:clixon\">" > $fxml
nr=$1 # Number of entries in DB for (( i=0; i<$1; i++ )); do
reqs=$2 echo -n "<y><a>$i</a><b>$i</b></y>" >> $fxml
operation=$3
# echo "runfn nr=$nr reqs=$reqs mode=$mode"
# new "generate config with $nr list entries"
echo -n "<rpc><edit-config><target><candidate/></target><default-operation>replace</default-operation><config><x xmlns=\"urn:example:clixon\">" > $fconfig
for (( i=0; i<$nr; i++ )); do
case $mode in
readlist|writelist|restreadlist|restwritelist)
echo -n "<y><a>$i</a><b>$i</b></y>" >> $fconfig
;;
writeleaflist)
echo -n "<c>$i</c>" >> $fconfig
;;
esac
done done
echo "</x></config></edit-config></rpc>]]>]]>" >> $fxml
else # restconf
echo -n '{"scaling:x":{"y":[' > $fjson
for (( i=0; i<$1; i++ )); do
if [ $i -ne 0 ]; then
echo -n ',' >> $fjson
fi
echo -n "{\"a\":$i,\"b\":\"$i\"}" >> $fjson
done
echo ']}}' >> $fjson
fi
}
echo "</x></config></edit-config></rpc>]]>]]>" >> $fconfig # Run netconffunction
# args: <op> <proto> <load> <n> <reqs>
# where proto is one of:
# netconf, restconf
# where op is one of:
# writeall readall commitall read write
runnet(){
op=$1
n=$2 # Number of entries in DB
reqs=$3
# new "netconf write $nr entry to backend" echo -n "$n " >> $dir/$op-netconf-$reqs
expecteof_file "$clixon_netconf -qf $cfg -y $fyang" "$fconfig" "^<rpc-reply><ok/></rpc-reply>]]>]]>$" case $op in
write)
case $mode in if [ $reqs = 0 ]; then # Write all in one go
readlist) genfile $n netconf;
# new "netconf GET list $reqs" { time -p cat $fxml | $clixon_netconf -qf $cfg -y $fyang ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs
time -p for (( i=0; i<$reqs; i++ )); do else # reqs != 0
rnd=$(( ( RANDOM % $nr ) )) { time -p for (( i=0; i<$reqs; i++ )); do
echo "<rpc><get-config><source><candidate/></source><filter type=\"xpath\" select=\"/x/y[a=$rnd][b=$rnd]\" /></get-config></rpc>]]>]]>" rnd=$(( ( RANDOM % $n ) ));
done | $clixon_netconf -qf $cfg -y $fyang > /dev/null echo "<rpc><edit-config><target><candidate/></target><config><x xmlns=\"urn:example:clixon\"><y><a>$rnd</a><b>$rnd</b></y></x></config></edit-config></rpc>]]>]]>";
done | $clixon_netconf -qf $cfg -y $fyang ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs
fi
;; ;;
writelist) read)
# new "netconf WRITE list $reqs" if [ $reqs = 0 ]; then # Read all in one go
time -p for (( i=0; i<$reqs; i++ )); do { time -p echo "<rpc><get-config><source><running/></source></get-config></rpc>]]>]]>" | $clixon_netconf -qf $cfg -y $fyang > /dev/null ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs
else # reqs != 0
{ time -p for (( i=0; i<$reqs; i++ )); do
rnd=$(( ( RANDOM % $nr ) )) rnd=$(( ( RANDOM % $nr ) ))
echo "<rpc><edit-config><target><candidate/></target><config><x xmlns=\"urn:example:clixon\"><y><a>$rnd</a><b>$rnd</b></y></x></config></edit-config></rpc>]]>]]>" echo "<rpc><edit-config><target><candidate/></target><config><x><y><a>$rnd</a><b>$rnd</b></y></x></config></edit-config></rpc>]]>]]>"
done | $clixon_netconf -qf $cfg -y $fyang > /dev/null done | $clixon_netconf -qf $cfg -y $fyang; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs
fi
;; ;;
restreadlist) delete)
# new "restconf GET list $reqs" { time -p for (( i=0; i<$reqs; i++ )); do
time -p for (( i=0; i<$reqs; i++ )); do
rnd=$(( ( RANDOM % $nr ) )) rnd=$(( ( RANDOM % $nr ) ))
curl -sSG http://localhost/restconf/data/scaling:x/y=$rnd > /dev/null echo "<rpc><edit-config><target><candidate/></target><config><x><y><a>$rnd</a><b>$rnd</b></y></x></config></edit-config></rpc>]]>]]>"
done done | $clixon_netconf -qf $cfg -y $fyang; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs
;; ;;
writeleaflist) commit)
# new "netconf GET leaf-list $reqs" { time -p echo "<rpc><commit/></rpc>]]>]]>" | $clixon_netconf -qf $cfg -y $fyang > /dev/null ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-netconf-$reqs
time -p for (( i=0; i<$reqs; i++ )); do ;;
rnd=$(( ( RANDOM % $nr ) )) *)
echo "<rpc><edit-config><target><candidate/></target><config><x xmlns=\"urn:example:clixon\"><c>$rnd</c></x></config></edit-config></rpc>]]>]]>" err "Operation not supported" "$op"
done | $clixon_netconf -qf $cfg -y $fyang > /dev/null exit
;; ;;
esac esac
# new "discard test"
expecteof "$clixon_netconf -qf $cfg -y $fyang" "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
} }
# Step # Run restconf function
# args: <i> <reqs> <mode> # args: <op> <proto> <load> <n> <reqs>
stepfn(){ # where proto is one of:
i=$1 # netconf, restconf
reqs=$2 # where op is one of:
mode=$3 # writeall readall commitall read write
runrest(){
op=$1
n=$2 # Number of entries in DB
reqs=$3
>&2 echo "stepfn $mode: i=$i reqs=$reqs" echo -n "$n " >> $dir/$op-restconf-$reqs
echo -n "" > $fconfig case $op in
t=$(TEST=%e runfn $i $reqs $mode 2>&1 | awk '/real/ {print $2}') write)
#TEST=%e runfn $i $reqs $mode 2>&1 if [ $reqs = 0 ]; then # Write all in one go
# t is time in secs of $reqs -> transactions per second. $reqs genfile $n restconf
p=$(echo "$reqs/$t" | bc -lq) # restconf @- means from stdin
# p is transactions per second. { time -p curl -sS -X PUT -d @$fjson http://localhost/restconf/data/scaling:x ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs
# write to gnuplot file: $dir/$mode else # Small requests
echo "$i $p" >> $dir/$mode { time -p for (( i=0; i<$reqs; i++ )); do
rnd=$(( ( RANDOM % $n ) ));
curl -sS -X PUT http://localhost/restconf/data/scaling:x/y=$rnd -d "{\"scaling:y\":{\"a\":$rnd,\"b\":\"$rnd\"}}"
done ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs
#
fi
;;
read)
if [ $reqs = 0 ]; then # Read all in one go
{ time -p curl -sS -X GET http://localhost/restconf/data/scaling:x > /dev/null; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs
else # Small requests
{ time -p for (( i=0; i<$reqs; i++ )); do
rnd=$(( ( RANDOM % $n ) ));
curl -sS -X GET http://localhost/restconf/data/scaling:x/y=$rnd
done ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs
fi
;;
delete)
{ time -p for (( i=0; i<$reqs; i++ )); do
rnd=$(( ( RANDOM % $n ) ));
curl -sS -X GET http://localhost/restconf/data/scaling:x/y=$rnd
done ; } 2>&1 | awk '/real/ {print $2}' >> $dir/$op-restconf-$reqs
;;
*)
err "Operation not supported" "$op"
exit
;;
esac
} }
# Run once
#args: <step> <reqs> <max>
once(){
# Input Parameters
step=$1
reqs=$2
max=$3
echo "oncefn step=$step reqs=$reqs max=$max" commit(){
new "test params: -f $cfg -y $fyang" # commit to running
if [ $BE -ne 0 ]; then expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
}
reset(){
# delete all in candidate
expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><edit-config><target><candidate/></target><default-operation>none</default-operation><config operation='delete'/></edit-config></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
# commit to running
commit
}
# Load n entries into candidate
# Args: <n>
load(){
# Generate file ($fxml)
genfile $1 netconf
# Write it to backend in one chunk
expecteof_file "$clixon_netconf -qf $cfg -y $fyang" "$fxml" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
}
# Run an operation, iterate from <from> to <to> in increment of <step>
# Each operation do <reqs> times
# args: <op> <protocol> <from> <step> <to> <reqs> <cand> <run>
# <reqs>=0 means all in one go
# <cand> <run> means a priori loaded into datastore
plot(){
op=$1
proto=$2
from=$3
step=$4
to=$5
reqs=$6
can=$7
run=$8
if [ $# -ne 8 ]; then
exit "plot should be called with 8 arguments, got $#"
fi
# reset file
new "Create file $dir/$op-$proto-$reqs"
echo "" > $dir/$op-$proto-$reqs
for (( n=$from; n<=$to; n=$n+$step )); do
reset
if [ $can = n ]; then
load $n
if [ $run = n ]; then
commit
fi
fi
new "$op-$proto-$reqs $n"
if [ $proto = netconf ]; then
runnet $op $n $reqs
else
runrest $op $n $reqs
fi
done
echo # newline
}
new "test params: -f $cfg -y $fyang"
if [ $BE -ne 0 ]; then
new "kill old backend" new "kill old backend"
sudo clixon_backend -zf $cfg -y $fyang sudo clixon_backend -zf $cfg -y $fyang
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
@ -173,40 +275,46 @@ once(){
fi fi
new "start backend -s init -f $cfg -y $fyang" new "start backend -s init -f $cfg -y $fyang"
start_backend -s init -f $cfg -y $fyang start_backend -s init -f $cfg -y $fyang
fi fi
new "kill old restconf daemon" new "kill old restconf daemon"
sudo pkill -u www-data -f "/www-data/clixon_restconf" sudo pkill -u www-data -f "/www-data/clixon_restconf"
new "start restconf daemon" new "start restconf daemon"
start_restconf -f $cfg -y $fyang start_restconf -f $cfg -y $fyang
new "waiting" new "waiting"
sleep $RCWAIT sleep $RCWAIT
new "Intial steps as start" for proto in netconf restconf; do
for (( i=10; i<=$step; i=i+10 )); do new "$proto write all entries to candidate (restconf:running)"
stepfn $i $reqs readlist plot write $proto $step $step $to 0 0 0 # all candidate 0 running 0
stepfn $i $reqs writelist done
stepfn $i $reqs restreadlist
stepfn $i $reqs writeleaflist
done
rnd=$(( ( RANDOM % $step ) ))
echo "curl -sSG http://localhost/restconf/data/scaling:x/y=$rnd"
curl -sSG http://localhost/restconf/data/scaling:x/y=$rnd
exit
new "Actual steps"
for (( i=$step; i<=$max; i=i+$step )); do
stepfn $i $reqs readlist
stepfn $i $reqs writelist
stepfn $i $reqs restreadlist
stepfn $i $reqs writeleaflist
done
new "Kill restconf daemon" for proto in netconf restconf; do
stop_restconf new "$proto read all entries from running"
plot read netconf $step $step $to 0 n n # start w full datastore
done
if [ $BE -ne 0 ]; then new "Netconf commit all entries from candidate to running"
plot commit netconf $step $step $to 0 n 0 # candidate full running empty
reqs=100
for proto in netconf restconf; do
new "$proto read $reqs from full database"
plot read $proto $step $step $to $reqs n n
new "$proto Write $reqs to full database(replace / alter values)"
plot write $proto $step $step $to $reqs n n
new "$proto delete $reqs from full database(replace / alter values)"
plot delete $proto $step $step $to $reqs n n
done
new "Kill restconf daemon"
stop_restconf
if [ $BE -ne 0 ]; then
new "Kill backend" new "Kill backend"
# Check if premature kill # Check if premature kill
pid=`pgrep -u root -f clixon_backend` pid=`pgrep -u root -f clixon_backend`
@ -215,12 +323,9 @@ exit
fi fi
# kill backend # kill backend
stop_backend -f $cfg stop_backend -f $cfg
fi fi
}
# step=200 reqs=500 max=2000
once 200 500 1000
arch=$(arch)
gnuplot -persist <<EOF gnuplot -persist <<EOF
set title "Clixon transactions per second r/w large lists" font ",14" textcolor rgbcolor "royalblue" set title "Clixon transactions per second r/w large lists" font ",14" textcolor rgbcolor "royalblue"
set xlabel "entries" set xlabel "entries"

2
test/test_restconf2.sh
View file

@ -95,7 +95,7 @@ expectfn 'curl -s -X POST -d {"example:cont1":{"interface":{"type":"regular"}}}
new "restconf POST initial tree" new "restconf POST initial tree"
expectfn 'curl -s -X POST -d {"example:cont1":{"interface":{"name":"local0","type":"regular"}}} http://localhost/restconf/data' 0 "" expectfn 'curl -s -X POST -d {"example:cont1":{"interface":{"name":"local0","type":"regular"}}} http://localhost/restconf/data' 0 ""
new "restconf GET datastore intial" new "restconf GET datastore initial"
expectfn "curl -s -X GET http://localhost/restconf/data/example:cont1" 0 '{"example:cont1": {"interface": \[{"name": "local0","type": "regular"}\]}}' expectfn "curl -s -X GET http://localhost/restconf/data/example:cont1" 0 '{"example:cont1": {"interface": \[{"name": "local0","type": "regular"}\]}}'
new "restconf GET interface subtree" new "restconf GET interface subtree"