experiments/clixon
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Cleaning up code for xml insignificant whitespace removal

Browse source 
Experimenal explicit index search code
This commit is contained in:
Olof hagsand 2020-02-07 14:59:57 +01:00
parent c7d6f69a85
commit a674af6f2c
32 changed files with 493 additions and 180 deletions
Download patch file Download diff file Expand all files Collapse all files

14
CHANGELOG.md
View file

@ -14,6 +14,9 @@
[search](https://clixon-docs.readthedocs.io/en/latest/xml.html#searching-in-xml)
### API changes on existing features (you may need to change your code)
* Session-id CLI functionality delayed: "lazy evaluation"
* C-api: Changed `clicon_session_id_get(clicon_handle h, uint32_t *id)`
* From a cli perspective this is a revert to 4.1 behaviour, where the cli does not immediately exit on start if the backend is not running, but with the new session-id function
* On failed validation of leafrefs, error message changed from: `No such leaf` to `No leaf <name> matching path <path>`.
* CLI Error message (clicon_rpc_generate_error()) changed when backend returns netconf error to be more descriptive:
* Original: `Config error: Validate failed. Edit and try again or discard changes: Invalid argument`
@ -21,8 +24,6 @@
### Minor changes
* Session-id CLI functionality delayed: "lazy evaluation"
* From a cli perspective this is a revert to 4.1 behaviour, where the cli does not immediately exit on start if the backend is not running, but with the new session-id function
* Obsoleted and removed XMLDB format "tree". This function did not work. Only xml and json allowed.
* Test framework
* Added `-- -S <file>` command-line to main example to be able to return any state to main example.
@ -33,19 +34,16 @@
### Corrected Bugs
* Fixed: Pretty-printed XML using prefixes not parsed correctly.
* eg `<a:x> <y/></a:x>` could lead to errors, wheras (`<x> <y/></x>`) works fine.
* XML namespace merge bug fixed. Example: two xmlns attributes could both survive a merge whereas one should replace the other.
## 4.3.1 (2 February 2020)
Patch release based on testing by Dave Cornejo, Netgate
### Corrected Bugs
* Compile option `VALIDATE_STATE_XML` introduced in `include/custom.h` to control whether code for state data validation is compiled or not.
* Fixed: Validation of user state data led to wrong validation, if state relied on config data, eg leafref/must/when etc.
* Fixed: No revision in yang module led to errors in validation of state data
* Fixed: Leafref validation did not cover case of when the "path" statement is declared within a typedef, only if it was declared in the data part directly under leaf.
* Fixed: Yang `must` xpath statements containing prefixes stopped working due to namespace context updates
## 4.3.0 (1 January 2020)
There were several issues with multiple namespaces with augmented yangs in 4.2 that have been fixed in 4.3. Some other highlights include: several issues with XPaths including "canonical namespace context" support, a reorganization of the YANG files shipped with the release, and a wildchar in the CLICON_MODE variable.

5
apps/backend/backend_client.c
View file

@ -1499,8 +1499,9 @@ from_client_msg(clicon_handle h,
ce->ce_id = id;
/* Populate incoming XML tree with yang -
* should really have been dealt with by decode above
* maybe not necessary since it should be */
if (xml_spec_populate_rpc(h, x, yspec) < 0)
* but it still is needed - test_cli debug test fails
*/
if (xml_spec_populate_rpc_input(h, x, yspec) < 0)
goto done;
if ((ret = xml_yang_validate_rpc(h, x, &xret)) < 0)
goto done;

2
apps/netconf/netconf_main.c
View file

@ -149,7 +149,7 @@ netconf_input_packet(clicon_handle h,
free(str0);
if ((xrpc=xpath_first(xreq, NULL, "//rpc")) != NULL){
isrpc++;
if (xml_spec_populate_rpc(h, xrpc, yspec) < 0)
if (xml_spec_populate_rpc_input(h, xrpc, yspec) < 0)
goto done;
if ((ret = xml_yang_validate_rpc(h, xrpc, &xret)) < 0)
goto done;

2
apps/restconf/restconf_methods_post.c
View file

@ -861,7 +861,7 @@ api_operations_post(clicon_handle h,
}
#endif
/* 6. Validate incoming RPC and fill in defaults */
if (xml_spec_populate_rpc(h, xtop, yspec) < 0) /* */
if (xml_spec_populate_rpc_input(h, xtop, yspec) < 0) /* */
goto done;
if ((ret = xml_yang_validate_rpc(h, xtop, &xret)) < 0)
goto done;

14
include/clixon_custom.h
View file

@ -65,16 +65,18 @@
*/
#define XPATH_LIST_OPTIMIZE
/*! Add search indexes, so that binary search can be made for non-key list indexes
/*! Add explicit search indexes, so that binary search can be made for non-key list indexes
* This also applies if there are multiple keys and you want to search on only the second for
* example.
*/
#undef XML_EXTRA_INDEX
#undef XML_EXPLICIT_INDEX
/*! Validate user state callback content
* Use may register state callbacks using ca_statedata callback
* When this option is set, the XML returned from the callback is validated after merging with the running
* db. If it fails, an internal error is returned to the originating user.
* If the option is not set, the XML returned by the user is not validated.
* User register state callbacks using the ca_statedata callback
* When this option is set, the XML returned from the callback is validated after merging with the
* running db. If it fails, an internal error is returned to the originating user.
* If the option is not set, the XML returned by the user is not validated. This could be useful if
* there is some error in the validation, as a safety catch.
* (Eventually this option will be permanently enabled)
*/
#define VALIDATE_STATE_XML

4
lib/clixon/clixon_xml.h
View file

@ -35,6 +35,8 @@
* XML support functions.
* @see https://www.w3.org/TR/2008/REC-xml-20081126
* https://www.w3.org/TR/2009/REC-xml-names-20091208/
* Canonical XML version (just for info)
* https://www.w3.org/TR/xml-c14n
*/
#ifndef _CLIXON_XML_H
#define _CLIXON_XML_H
@ -158,9 +160,9 @@ cxobj *xml_wrap(cxobj *xc, char *tag);
int xml_purge(cxobj *xc);
int xml_child_rm(cxobj *xp, int i);
int xml_rm(cxobj *xc);
int xml_rm_children(cxobj *x, enum cxobj_type type);
int xml_rootchild(cxobj *xp, int i, cxobj **xcp);
int xml_rootchild_node(cxobj *xp, cxobj *xc);
int xml_enumerate_children(cxobj *xp);
int xml_enumerate_reset(cxobj *xp);
int xml_enumerate_get(cxobj *x);

4
lib/clixon/clixon_xml_map.h
View file

@ -59,10 +59,10 @@ int xml_tree_prune_flagged(cxobj *xt, int flag, int test);
int xml_default(cxobj *x, void *arg);
int xml_sanity(cxobj *x, void *arg);
int xml_non_config_data(cxobj *xt, void *arg);
int xml_spec_populate_rpc(clicon_handle h, cxobj *x, yang_stmt *yspec);
int xml_spec_populate_rpc_input(clicon_handle h, cxobj *x, yang_stmt *yspec);
int xml_spec_populate(cxobj *x, void *arg);
int xml2xpath(cxobj *x, char **xpath);
int check_namespaces(cxobj *x0, cxobj *x1, cxobj *x1p);
int assign_namespaces(cxobj *x0, cxobj *x1, cxobj *x1p);
int xml_merge(cxobj *x0, cxobj *x1, yang_stmt *yspec, char **reason);
int yang_enum_int_value(cxobj *node, int32_t *val);

2
lib/clixon/clixon_yang.h
View file

@ -51,7 +51,7 @@
* Yang flags used in
*/
#define YANG_FLAG_MARK 0x01 /* (Dynamic) marker for dynamic algorithms, eg expand */
#ifdef XML_EXTRA_INDEX
#ifdef XML_EXPLICIT_INDEX
#define YANG_FLAG_INDEX 0x02 /* This yang node under list is (extra) index. --> you can access
* list elements using this index with binary search */
#endif

2
lib/src/Makefile.in
View file

@ -79,7 +79,7 @@ SRC = clixon_sig.c clixon_uid.c clixon_log.c clixon_err.c clixon_event.c \
clixon_sha1.c clixon_datastore.c clixon_datastore_write.c clixon_datastore_read.c \
clixon_netconf_lib.c clixon_stream.c clixon_nacm.c
YACCOBJS := lex.clixon_xml_parse.o clixon_xml_parse.tab.o \
YACCOBJS = lex.clixon_xml_parse.o clixon_xml_parse.tab.o \
lex.clixon_yang_parse.o clixon_yang_parse.tab.o \
lex.clixon_json_parse.o clixon_json_parse.tab.o \
lex.clixon_xpath_parse.o clixon_xpath_parse.tab.o \

1
lib/src/clixon_data.c
View file

@ -615,4 +615,3 @@ clicon_session_id_set(clicon_handle h,
clicon_hash_add(cdat, "session-id", &id, sizeof(uint32_t));
return 0;
}

4
lib/src/clixon_datastore_write.c
View file

@ -348,7 +348,7 @@ text_modify(clicon_handle h,
* Check if namespace exists in x0 parent
* if not add new binding and replace in x0.
*/
if (check_namespaces(x1, x0, x0p) < 0)
if (assign_namespaces(x1, x0, x0p) < 0)
goto done;
changed++;
if (op==OP_NONE)
@ -514,7 +514,7 @@ text_modify(clicon_handle h,
* Check if namespace exists in x0 parent
* if not add new binding and replace in x0.
*/
if (check_namespaces(x1, x0, x0p) < 0)
if (assign_namespaces(x1, x0, x0p) < 0)
goto done;
if (op==OP_NONE)
xml_flag_set(x0, XML_FLAG_NONE); /* Mark for potential deletion */

7
lib/src/clixon_json.c
View file

@ -1156,9 +1156,12 @@ json_parse(char *str,
goto done;
if (ret == 0)
goto fail;
/* Populate, ie associate xml nodes with yang specs */
if (xml_apply0(xt, CX_ELMNT, xml_spec_populate, yspec) < 0)
/* Populate, ie associate xml nodes with yang specs.
* XXX But this wrong if xt is not on top-level, can give false positives.
*/
if (xml_apply0(xt, CX_ELMNT, xml_spec_populate, yspec) < 0){
goto done;
}
if (xml_apply0(xt, CX_ELMNT, xml_sort, NULL) < 0)
goto done;
/* Now find leafs with identityrefs (+transitive) and translate

4
lib/src/clixon_options.c
View file

@ -330,8 +330,10 @@ clicon_option_add(clicon_handle h,
if (strcmp(name, "CLICON_FEATURE")==0 ||
strcmp(name, "CLICON_YANG_DIR")==0){
if ((x = clicon_conf_xml(h)) == NULL)
if ((x = clicon_conf_xml(h)) == NULL){
clicon_err(OE_UNIX, ENOENT, "option %s not found (clicon_conf_xml_set has not been called?)", name);
goto done;
}
if (xml_parse_va(&x, NULL, "<%s>%s</%s>",
name, value, name) < 0)
goto done;

4
lib/src/clixon_path.c
View file

@ -1394,7 +1394,7 @@ instance_id_resolve(clixon_path *cplist,
goto done;
}
/*! Search XML tree using Clixon path struct
/*! Search XML tree using (internal) Clixon path struct
*
* @param[in] xt Top xml-tree where to search
* @param[in] yt Yang statement of top symbol (can be yang-spec if top-level)
@ -1612,7 +1612,7 @@ clixon_xml_find_instance_id(cxobj *xt,
goto done;
if (debug)
clixon_path_print(stderr, cplist);
/* Resolve module:name to yang-stmt, fail if not successful */
/* Resolve module:name to pointer to yang-stmt, fail if not successful */
if ((ret = instance_id_resolve(cplist, yt)) < 0)
goto done;
if (ret == 0)

1
lib/src/clixon_plugin.c
View file

@ -400,7 +400,6 @@ clixon_plugin_exit(clicon_handle h)
return 0;
}
/*! Run the restconf user-defined credentials callback if present
* Find first authentication callback and call that, then return.
* The callback is to set the authenticated user

32
lib/src/clixon_xml.c
View file

@ -35,6 +35,8 @@
* XML support functions.
* @see https://www.w3.org/TR/2008/REC-xml-20081126
* https://www.w3.org/TR/2009/REC-xml-names-20091208
* Canonical XML version (just for info)
* https://www.w3.org/TR/xml-c14n
*/
#ifdef HAVE_CONFIG_H
@ -1276,6 +1278,35 @@ xml_rm(cxobj *xc)
return retval;
}
/*! Remove all children of specific type
* @param[in] x XML node
* @param[in] type Remove all children of xn of this type
* @retval 0 OK
* @retval -1 Error
*/
int
xml_rm_children(cxobj *x,
enum cxobj_type type)
{
int retval = -1;
cxobj *xc;
int i;
for (i=0; i<xml_child_nr(x);){
xc = xml_child_i(x, i);
if (xml_type(xc) != type){
i++;
continue;
}
if (xml_child_rm(x, i) < 0)
goto done;
xml_free(xc);
}
retval = 0;
done:
return retval;
}
/*! Remove top XML object and all children except a single child
* Given a root xml node, and the i:th child, remove the child from its parent
* and return it, remove the parent and all other children. (unwrap)
@ -1934,7 +1965,6 @@ _xml_parse(const char *str,
return -1;
}
ya.ya_xparent = xt;
ya.ya_skipspace = 1; /* remove all non-terminal bodies (strip pretty-print) */
ya.ya_yspec = yspec;
if (clixon_xml_parsel_init(&ya) < 0)
goto done;

96
lib/src/clixon_xml_map.c
View file

@ -878,24 +878,23 @@ xml_non_config_data(cxobj *xt,
return retval;
}
/*! Add yang specification backpointer to rpc
/*! Find yang spec association of XML node for incoming RPC
*
* Incoming RPC has an "input" structure that is not taken care of by xml_spec_populate
* @param[in] xt XML tree node
* @param[in] arg Yang spec
* @retval 0 OK
* @retval -1 Error
* @note This may be unnecessary if yspec is set on creation
* @note For subs to anyxml nodes will not have spec set
* @note No validation is done,... XXX
* The
* @code
* xml_apply(xc, CX_ELMNT, xml_spec_populate, yspec)
* xml_apply(xc, CX_ELMNT, xml_spec_populate_rpc_input, yspec)
* @endcode
* @see xml_spec_populate
* @see xml_spec_populate For other generic cases
*/
int
xml_spec_populate_rpc(clicon_handle h,
cxobj *xrpc,
yang_stmt *yspec)
xml_spec_populate_rpc_input(clicon_handle h,
cxobj *xrpc,
yang_stmt *yspec)
{
int retval = -1;
yang_stmt *yrpc = NULL; /* yang node */
@ -932,17 +931,24 @@ xml_spec_populate_rpc(clicon_handle h,
return retval;
}
/*! Add yang specification backpointer to XML node
/*! Find yang spec association of XML node
*
* This may be unnecessary if yspec is set on manual creation. Also note that for incoming or outgoing RPC
* need specialized function.
* XXX: maybe it can be built into the same function, but you dont know whether it is input or output rpc, so
* it seems like you need specialized functions.
* @param[in] xt XML tree node
* @param[in] arg Yang spec
* @note This may be unnecessary if yspec is set on creation
* @note For subs to anyxml nodes will not have spec set
* @note No validation is done,... XXX
* @retval 0 OK
* @retval -1 Error
* @code
* xml_apply(xc, CX_ELMNT, xml_spec_populate, yspec)
* xml_apply(xc, CX_ELMNT, xml_spec_populate, yspec)
* @endcode
* @note For subs to anyxml nodes will not have spec set
* @see xml_spec_populate_rpc_input for incoming rpc
* XXX: can give false positives
*/
#undef DEBUG
#undef DEBUG /* Set this for debug */
int
xml_spec_populate(cxobj *x,
void *arg)
@ -966,18 +972,19 @@ xml_spec_populate(cxobj *x,
if (xml2ns(x, xml_prefix(x), &ns) < 0)
goto done;
if (xp && (yparent = xml_spec(xp)) != NULL){
clicon_debug(1, "%s yang parent:%s", __FUNCTION__, yang_argument_get(yparent));
y = yang_find_datanode(yparent, name);
}
else if (yspec){
else if (yspec){ /* XXX this gives false positives */
if (ys_module_by_xml(yspec, x, &ymod) < 0)
goto done;
/* ymod is "real" module, name may belong to included submodule */
if (ymod != NULL){
#ifdef DEBUG
clicon_debug(1, "%s %s mod:%s", __FUNCTION__, name, yang_argument_get(ymod));
clicon_debug(1, "%s %s mod:%s", __FUNCTION__, name, yang_argument_get(ymod));
#endif
y = yang_find_schemanode(ymod, name);
}
y = yang_find_schemanode(ymod, name);
}
#ifdef DEBUG
else
clicon_debug(1, "%s %s mod:NULL", __FUNCTION__, name);
@ -993,19 +1000,34 @@ xml_spec_populate(cxobj *x,
clicon_debug(1, "%s y:%s", __FUNCTION__, yang_argument_get(y));
#endif
/* Assign spec only if namespaces match */
if (strcmp(ns, nsy) == 0)
xml_spec_set(x, y);
if (strcmp(ns, nsy) == 0){
#if 0
/* Cornercase:
* The XML parser may have kept pretty-printed whitespaces in empty nodes, eg "<x> </x>"
* since it is valid leaf(-list) content.
* But if it is not a container, list or something, else, the content should be stripped.
* But we cannot do this because of false positives (above)
*/
if (xml_spec(x) == NULL && /* Not assigned yet, ensure to do just once,... */
yang_keyword_get(y) != Y_LEAF &&
yang_keyword_get(y) != Y_LEAF_LIST){
if (xml_rm_children(x, CX_BODY) < 0)
goto done;
}
#endif
xml_spec_set(x, y);
}
}
else{
#ifdef DEBUG
else
clicon_debug(1, "%s y:NULL", __FUNCTION__);
#endif
}
retval = 0;
done:
return retval;
}
/*! Given an XML node, build an xpath to root, internal function
* @retval 0 OK
* @retval -1 Error. eg XML malformed
@ -1159,19 +1181,19 @@ xmlns_assign(cxobj *x)
goto done;
*/
int
check_namespaces(cxobj *x0, /* source */
cxobj *x1, /* target */
cxobj *x1p)
assign_namespaces(cxobj *x0, /* source */
cxobj *x1, /* target */
cxobj *x1p)
{
int retval = -1;
char *namespace = NULL;
char *prefix0 = NULL;;
char *prefix1 = NULL;
char *prefixb = NULL; /* identityref body prefix */
cvec *nsc0 = NULL;
cvec *nsc = NULL;
int isroot;
char *pexist = NULL;
int retval = -1;
char *namespace = NULL;
char *prefix0 = NULL;;
char *prefix1 = NULL;
char *prefixb = NULL; /* identityref body prefix */
cvec *nsc0 = NULL;
cvec *nsc = NULL;
int isroot;
char *pexist = NULL;
yang_stmt *y;
/* XXX: need to identify root better than hiereustics and strcmp,... */
@ -1313,7 +1335,7 @@ xml_merge1(cxobj *x0, /* the target */
if (xml_value_set(x0b, x1bstr) < 0)
goto done;
}
if (check_namespaces(x1, x0, x0p) < 0)
if (assign_namespaces(x1, x0, x0p) < 0)
goto done;
} /* if LEAF|LEAF_LIST */
else { /* eg Y_CONTAINER, Y_LIST */
@ -1321,7 +1343,7 @@ xml_merge1(cxobj *x0, /* the target */
if ((x0 = xml_new(x1name, NULL, (yang_stmt*)y0)) == NULL)
goto done;
}
if (check_namespaces(x1, x0, x0p) < 0)
if (assign_namespaces(x1, x0, x0p) < 0)
goto done;
/* Loop through children of the modification tree */
x1c = NULL;

1
lib/src/clixon_xml_parse.h
View file

@ -50,7 +50,6 @@ struct xml_parse_yacc_arg{
cxobj *ya_xelement; /* xml active element */
cxobj *ya_xparent; /* xml parent element*/
int ya_skipspace; /* If set, remove all non-terminal bodies (strip pretty-print) */
yang_stmt *ya_yspec; /* If set, top-level yang-spec */
int ya_lex_state; /* lex return state */
};

4
lib/src/clixon_xml_parse.l
View file

@ -34,8 +34,8 @@
* XML parser
* @see https://www.w3.org/TR/2008/REC-xml-20081126
* https://www.w3.org/TR/2009/REC-xml-names-20091208
*
* Canonical XML version (just for info)
* https://www.w3.org/TR/xml-c14n
*/
%{

123
lib/src/clixon_xml_parse.y
View file

@ -34,6 +34,8 @@
* XML parser
* @see https://www.w3.org/TR/2008/REC-xml-20081126
* https://www.w3.org/TR/2009/REC-xml-names-20091208
* Canonical XML version (just for info)
* https://www.w3.org/TR/xml-c14n
*/
%union {
char *string;
@ -72,6 +74,7 @@
#include "clixon_log.h"
#include "clixon_queue.h"
#include "clixon_hash.h"
#include "clixon_string.h"
#include "clixon_handle.h"
#include "clixon_yang.h"
#include "clixon_xml.h"
@ -127,14 +130,12 @@ xml_parse_whitespace(struct xml_parse_yacc_arg *ya,
ya->ya_xelement = NULL; /* init */
/* If there is an element already, only add one whitespace child
* otherwise, keep all whitespace.
* otherwise, keep all whitespace. See code in xml_parse_bslash
*/
#if 1
for (i=0; i<xml_child_nr(xp); i++){
if (xml_type(xml_child_i(xp, i)) == CX_ELMNT)
goto ok; /* Skip if already element */
}
#endif
if (xn == NULL){
if ((xn = xml_new("body", xp, NULL)) == NULL)
goto done;
@ -250,47 +251,51 @@ xml_parse_endslash_post(struct xml_parse_yacc_arg *ya)
return 0;
}
/*! Called at </name> */
/*! A content terminated by <name>...</name> or <prefix:name>...</prefix:name> is ready
*
* Any whitespace between the subelements to a non-leaf is
* insignificant, i.e., an implementation MAY insert whitespace
* characters between subelements and are therefore stripped, but see comment in code below.
* @param[in] ya XML parser yacc handler struct
* @param[in] prefix
* @param[in] name
*/
static int
xml_parse_bslash1(struct xml_parse_yacc_arg *ya,
char *name)
xml_parse_bslash(struct xml_parse_yacc_arg *ya,
char *prefix,
char *name)
{
int retval = -1;
cxobj *x = ya->ya_xelement;
cxobj *xc;
char *prefix0;
char *name0;
if (strcmp(xml_name(x), name)){
clicon_err(OE_XML, XMLPARSE_ERRNO, "XML parse sanity check failed: %s vs %s",
xml_name(x), name);
goto done;
}
if (xml_prefix(x)!=NULL){
clicon_err(OE_XML, XMLPARSE_ERRNO, "XML parse sanity check failed: %s:%s vs %s",
xml_prefix(x), xml_name(x), name);
/* These are existing tags */
prefix0 = xml_prefix(x);
name0 = xml_name(x);
/* Check name or prerix unequal from begin-tag */
if (clicon_strcmp(name0, name) ||
clicon_strcmp(prefix0, prefix)){
clicon_err(OE_XML, XMLPARSE_ERRNO, "Sanity check failed: %s%s%s vs %s%s%s",
prefix0?prefix0:"", prefix0?":":"", name0,
prefix?prefix:"", prefix?":":"", name);
goto done;
}
/* Strip pretty-print. Ad-hoc algorithm
* It ok with x:[body], but not with x:[ex,body]
* It is also ok with x:[attr,body]
* So the rule is: if there is at least on element, then remove all bodies?
* So the rule is: if there is at least on element, then remove all bodies.
* See also code in xml_parse_whitespace
* But there is more: when YANG is assigned, if not leaf/leaf-lists, then all contents should
* be stripped, see xml_spec_populate()
*/
if (ya->ya_skipspace){
xc = NULL;
while ((xc = xml_child_each(x, xc, CX_ELMNT)) != NULL)
break;
if (xc != NULL){ /* at least one element */
int i;
for (i=0; i<xml_child_nr(x);){
xc = xml_child_i(x, i);
if (xml_type(xc) != CX_BODY){
i++;
continue;
}
if (xml_child_rm(x, i) < 0)
goto done;
xml_free(xc);
}
}
xc = NULL;
while ((xc = xml_child_each(x, xc, CX_ELMNT)) != NULL)
break;
if (xc != NULL){ /* at least one element */
if (xml_rm_children(x, CX_BODY) < 0) /* remove all bodies */
goto done;
}
retval = 0;
done:
@ -298,56 +303,6 @@ xml_parse_bslash1(struct xml_parse_yacc_arg *ya,
return retval;
}
/*! Called at </namespace:name> */
static int
xml_parse_bslash2(struct xml_parse_yacc_arg *ya,
char *namespace,
char *name)
{
int retval = -1;
cxobj *x = ya->ya_xelement;
cxobj *xc;
if (strcmp(xml_name(x), name)){
clicon_err(OE_XML, XMLPARSE_ERRNO, "Sanity check failed: %s:%s vs %s:%s",
xml_prefix(x),
xml_name(x),
namespace,
name);
goto done;
}
if (xml_prefix(x)==NULL ||
strcmp(xml_prefix(x), namespace)){
clicon_err(OE_XML, XMLPARSE_ERRNO, "Sanity check failed: %s:%s vs %s:%s",
xml_prefix(x),
xml_name(x),
namespace,
name);
goto done;
}
/* Strip pretty-print. Ad-hoc algorithm
* It ok with x:[body], but not with x:[ex,body]
* It is also ok with x:[attr,body]
* So the rule is: if there is at least on element, then remove all bodies?
*/
if (ya->ya_skipspace){
xc = NULL;
while ((xc = xml_child_each(x, xc, CX_ELMNT)) != NULL)
break;
if (xc != NULL){ /* at least one element */
xc = NULL;
while ((xc = xml_child_each(x, xc, CX_BODY)) != NULL) {
xml_value_set(xc, ""); /* XXX remove */
}
}
}
retval = 0;
done:
free(name);
free(namespace);
return retval;
}
/*! Parse XML attribute
* Special cases:
* - DefaultAttName: xmlns
@ -450,10 +405,10 @@ element1 : ESLASH {_YA->ya_xelement = NULL;
endtag : BSLASH NAME '>'
{ clicon_debug(2, "endtag -> < </ NAME>");
if (xml_parse_bslash1(_YA, $2) < 0) YYABORT; }
if (xml_parse_bslash(_YA, NULL, $2) < 0) YYABORT; }
| BSLASH NAME ':' NAME '>'
{ if (xml_parse_bslash2(_YA, $2, $4) < 0) YYABORT;
{ if (xml_parse_bslash(_YA, $2, $4) < 0) YYABORT;
clicon_debug(2, "endtag -> < </ NAME:NAME >"); }
;

45
lib/src/clixon_xml_sort.c
View file

@ -579,7 +579,7 @@ xml_find_keys1(cxobj *xp,
* @see xml_find_index for a generic search function
*/
static int
xml_find_keys(cxobj *xp,
xml_find_keys(cxobj *xp,
cxobj *x1,
yang_stmt *yc,
int skip1,
@ -1160,6 +1160,36 @@ xml_find_index_yang(cxobj *xp,
goto done;
}
#ifdef XML_EXPLICIT_INDEX
/*! API for search in XML child list with yang available
* @retval 1 OK
* @retval 0 Revert, try again with no-yang search (or explicit index)
* @retval -1 Error
* XXX: can merge with xml_find_index_yang in some way, similar code
*/
static int
xml_find_index_explicit(cxobj *xp,
yang_stmt *yc,
cvec *cvk,
cxobj ***xvec,
size_t *xlen)
{
int retval = -1;
cg_var *cvi;
if (yang_keyword_get(yc) == Y_LIST && cvec_len(cvk) == 1){
cvi = cvec_i(cvk, 0);
goto revert;
}
retval = 1; /* OK */
done:
return retval;
revert: /* means try name-only*/
retval = 0;
goto done;
}
#endif /* XML_EXPLICIT_INDEX */
/*! API for search in XML child list using indexes and binary search if applicable
*
* Generic search function as alternative to xml_find() and others that finds YANG associated
@ -1237,9 +1267,18 @@ clixon_xml_find_index(cxobj *xp,
if (yc){
if ((ret = xml_find_index_yang(xp, yc, cvk, xvec, xlen)) < 0)
goto done;
if (ret == 0)
if (xml_find_noyang_name(xp, namespace, name, cvk, xvec, xlen) < 0)
if (ret == 0){ /* This means yang method did not work for some reason
* such as not being list key indexes in cvk, for example
*/
#ifdef XML_EXPLICIT_INDEX
/* Check if (exactly one) explicit indexes in cvk */
if ((ret = xml_find_index_explicit(xp, yc, cvk, xvec, xlen)) < 0)
goto done;
if (ret == 0)
#endif
if (xml_find_noyang_name(xp, namespace, name, cvk, xvec, xlen) < 0)
goto done;
}
}
else
if (xml_find_noyang_name(xp, namespace, name, cvk, xvec, xlen) < 0)

73
lib/src/clixon_yang.c
View file

@ -84,6 +84,10 @@
#include "clixon_yang_type.h"
#include "clixon_yang_internal.h" /* internal included by this file only, not API*/
#ifdef XML_EXPLICIT_INDEX
static int yang_search_index_extension(clicon_handle h, yang_stmt *yext, yang_stmt *ys);
#endif
/*
* Local variables
*/
@ -1824,9 +1828,17 @@ ys_populate_unknown(clicon_handle h,
goto done;
}
}
#ifdef XML_EXPLICIT_INDEX
/* Add explicit index extension */
if ((retval = yang_search_index_extension(h, yext, ys)) < 0) {
clicon_debug(1, "plugin_extension() failed");
return -1;
}
#endif
/* Make extension callbacks that may alter yang structure */
if (clixon_plugin_extension(h, yext, ys) < 0)
goto done;
retval = 0;
done:
if (prefix)
@ -2634,14 +2646,61 @@ yang_type_cache_free(yang_type_cache *ycache)
return 0;
}
#ifdef XML_EXTRA_INDEX
/*! Mark element as index in list
#ifdef XML_EXPLICIT_INDEX
/*! Mark element as search_index in list
* @retval 0 OK
* @retval -1 Error
*/
int
yang_list_index_add(yang_stmt *yi)
yang_list_index_add(yang_stmt *ys)
{
assert(yang_parent_get(yi) && yang_keyword_get(yang_parent_get(yi)) == Y_LIST);
yi->ys_flags |= YANG_FLAG_INDEX;
return 0;
int retval = -1;
yang_stmt *yp;
if ((yp = yang_parent_get(ys)) == NULL ||
yang_keyword_get(yp) != Y_LIST){
clicon_log(LOG_WARNING, "search_index should in a list");
goto ok;
}
yang_flag_set(ys, YANG_FLAG_INDEX);
ok:
retval = 0;
// done:
return retval;
}
#endif /* XML_EXTRA_INDEX */
/*! Callback for yang clixon search_index extension
*
* @param[in] h Clixon handle
* @param[in] yext Yang node of extension
* @param[in] ys Yang node of (unknown) statement belonging to extension
* @retval 0 OK (warnings may appear)
* @retval -1 Error
*/
int
yang_search_index_extension(clicon_handle h,
yang_stmt *yext,
yang_stmt *ys)
{
int retval = -1;
char *extname;
char *modname;
yang_stmt *ymod;
yang_stmt *yp;
ymod = ys_module(yext);
modname = yang_argument_get(ymod);
extname = yang_argument_get(yext);
if (strcmp(modname, "clixon-config") != 0 || strcmp(extname, "search_index") != 0)
goto ok;
clicon_debug(1, "%s Enabled extension:%s:%s", __FUNCTION__, modname, extname);
yp = yang_parent_get(ys);
if (yang_list_index_add(yp) < 0)
goto done;
ok:
retval = 0;
done:
return retval;
}
#endif /* XML_EXPLICIT_INDEX */

2
test/lib.sh
View file

@ -70,7 +70,7 @@ testname=
# for starting a backend
: ${BETIMEOUT:=10}
# If set, enable debugging (of backend)
# If set, enable debugging (of backend and restconf daemons)
: ${DBG:=0}
# Where to log restconf. Some systems may not have syslog,

5
test/test_api_path.sh
View file

@ -15,8 +15,11 @@ s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
# Number of list/leaf-list entries
: ${nr:=100}
# Number of tests to generate XML for
max=7
# XML file (alt provide it in stdin after xpath)
for (( i=1; i<7; i++ )); do
for (( i=1; i<$max; i++ )); do
eval xml$i=$dir/xml$i.xml
done
ydir=$dir/yang

4
test/test_cli.sh
View file

@ -113,9 +113,11 @@ expectfn "$clixon_cli -1 -f $cfg -l o load /tmp/foo" 0 "^$"
new "cli check load"
expectfn "$clixon_cli -1 -f $cfg -l o show conf cli" 0 "interfaces interface eth/0/0 ipv4 enabled true"
new "cli debug"
new "cli debug set"
expectfn "$clixon_cli -1 -f $cfg -l o debug level 1" 0 "^$"
# How to test this?
new "cli debug reset"
expectfn "$clixon_cli -1 -f $cfg -l o debug level 0" 0 "^$"
new "cli rpc"

7
test/test_instance_id.sh
View file

@ -9,8 +9,11 @@ s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
# Number of list/leaf-list entries
: ${nr:=100}
# Number of tests to generate XML for
max=9
# XML file (alt provide it in stdin after xpath)
for (( i=1; i<9; i++ )); do
for (( i=1; i<$max; i++ )); do
eval xml$i=$dir/xml$i.xml
done
ydir=$dir/yang
@ -19,7 +22,7 @@ if [ ! -d $ydir ]; then
mkdir $ydir
fi
# XPATH binary search in ordered-by system lists
# Instance-id PATH binary search in ordered-by system lists
cat <<EOF > $ydir/moda.yang
module moda{
namespace "urn:example:a";

21
test/test_leafref_state.sh
View file

@ -63,6 +63,7 @@ module leafref{
}
EOF
# This is state data writte to file that backend reads from (on request)
cat <<EOF > $fstate
<sender-state xmlns="urn:example:example">
<ref>x</ref>
@ -83,7 +84,7 @@ if [ $BE -ne 0 ]; then
wait_backend
fi
# Test top-level, default prefix, wring leafref prefix and typedef path
# Test top-level, default prefix, wrong leafref prefix and typedef path
XML=$(cat <<EOF
<sender-config xmlns="urn:example:example">
<name>x</name>
@ -127,6 +128,24 @@ expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get content="nonconfig"><filter ty
new "netconf get /sender-config config-only"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get content="config"><filter type="xpath" select="/sender-config" xmlns="urn:example:example"/></get></rpc>]]>]]>' '^<rpc-reply><data><sender-config xmlns="urn:example:example"><name>x</name></sender-config></data></rpc-reply>]]>]]>$'
# Negative tests,
# Double xmlns attribute
cat <<EOF > $fstate
<sender-config xmlns="urn:example:example">
<name>x</name>
</sender-config>
EOF
new "Merge same tree - check double xmlns attribute"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get content="all"><filter type="xpath" select="/"/></get></rpc>]]>]]>' '^<rpc-reply><data><sender-config xmlns="urn:example:example"><name>x</name></sender-config></data></rpc-reply>]]>]]>$'
# Back to original
cat <<EOF > $fstate
<sender-state xmlns="urn:example:example">
<ref>x</ref>
</sender-state>
EOF
# delete x, add y
XML=$(cat <<EOF
<sender-config xmlns="urn:example:example" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" nc:operation="delete">

154
test/test_netconf_whitespace.sh Executable file
View file

@ -0,0 +1,154 @@
#!/usr/bin/env bash
# Test netconf / xml whitespaces
# Insignicant and significant
# See https://www.w3.org/TR/xml-c14n
# where "clean", "dirty" and "mixed" whitespaces are defined
# <clean> </clean>
# <dirty> A B </dirty>
# <mixed>
# A
# <clean> </clean>
# B
# <dirty> A B </dirty>
# C
# </mixed>
# Magic line must be first in script (see README.md)
s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
APPNAME=example
cfg=$dir/conf_yang.xml
fyang=$dir/whitespace.yang
cat <<EOF > $cfg
<clixon-config xmlns="http://clicon.org/config">
<CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
<CLICON_FEATURE>ietf-netconf:startup</CLICON_FEATURE>
<CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
<CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
<CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
<CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
<CLICON_CLI_MODE>$APPNAME</CLICON_CLI_MODE>
<CLICON_SOCK>/usr/local/var/$APPNAME/$APPNAME.sock</CLICON_SOCK>
<CLICON_BACKEND_DIR>/usr/local/lib/$APPNAME/backend</CLICON_BACKEND_DIR>
<CLICON_BACKEND_PIDFILE>/usr/local/var/$APPNAME/$APPNAME.pidfile</CLICON_BACKEND_PIDFILE>
<CLICON_XMLDB_DIR>$dir</CLICON_XMLDB_DIR>
</clixon-config>
EOF
cat <<EOF > $fyang
module whitespace{
yang-version 1.1;
namespace "urn:example:whitespace";
prefix fi;
container x{
presence true;
list y {
key a;
leaf a{
type string;
}
container b{
presence true;
}
}
}
}
EOF
new "test params: -f $cfg"
echo '<config><ex:x xmlns:ex="urn:example:whitespace">
<ex:y> <ex:a>foo</ex:a>\n <ex:b> </ex:b></ex:y> </ex:x></config></edit-config></rpc>]]>]]>$start</config>' > $dir/startup_db
if [ $BE -ne 0 ]; then
new "kill old backend"
sudo clixon_backend -zf $cfg -s startup
if [ $? -ne 0 ]; then
err
fi
new "start backend -s startup -f $cfg"
start_backend -s startup -f $cfg
fi
new "waiting"
wait_backend
new "get startup"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><ex:x xmlns:ex="urn:example:whitespace"><ex:y><ex:a>foo</ex:a><ex:b/></ex:y></ex:x></data></rpc-reply>]]>]]>$'
new "remove"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:whitespace" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" nc:operation="delete"/></config></edit-config></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
new "commit"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "add insignificant 'dirty' whitespaces"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:whitespace"> \t \ <y> <a>foo</a>\n </y> </x></config></edit-config></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
new "get stripped whitespace"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><x xmlns="urn:example:whitespace"><y><a>foo</a></y></x></data></rpc-reply>]]>]]>$'
new "validate"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "netconf discard-changes"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "add insignificant 'dirty' whitespaces with prefixes"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><ex:x xmlns:ex="urn:example:whitespace"> \t \ <ex:y> <ex:a>foo</ex:a>\n </ex:y> </ex:x></config></edit-config></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
new "get stripped whitespace"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><ex:x xmlns:ex="urn:example:whitespace"><ex:y><ex:a>foo</ex:a></ex:y></ex:x></data></rpc-reply>]]>]]>$'
new "validate"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "netconf discard-changes"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "add insignificant 'clean' whitespaces"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:whitespace"> \n \t </x></config></edit-config></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
new "add more insignificant 'clean' whitespaces"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:whitespace"><y><a>foo</a><b> </b></y></x></config></edit-config></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
new "get stripped whitespace"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><x xmlns="urn:example:whitespace"><y><a>foo</a><b/></y></x></data></rpc-reply>]]>]]>'
new "validate"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "netconf discard-changes"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "add significant whitespaces"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:whitespace"><y><a> foo
bar </a></y></x></config></edit-config></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
new "get significant whitespace"
expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><x xmlns="urn:example:whitespace"><y><a> foo
bar </a></y></x></data></rpc-reply>]]>]]>'
new "validate"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
new "netconf discard-changes"
expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
if [ $BE -eq 0 ]; then
exit # BE
fi
new "Kill backend"
# Check if premature kill
pid=$(pgrep -u root -f clixon_backend)
if [ -z "$pid" ]; then
err "backend already dead"
fi
# kill backend
stop_backend -f $cfg
rm -rf $dir

6
test/test_xml.sh
View file

@ -163,7 +163,7 @@ XML=$(cat <<EOF
</html:html>
EOF
)
expecteof "$clixon_util_xml -o" 0 "$XML" "$XML"
expecteof "$clixon_util_xml -o" 0 "$XML" '^<html:html xmlns:html="http://www.w3.org/1999/xhtml"><html:head><html:title>Frobnostication</html:title></html:head><html:body><html:p><html:a href="http://frob.example.com">here.</html:a></html:p></html:body></html:html>$'
new "Second example 6.1 from https://www.w3.org/TR/2009/REC-xml-names-20091208"
XML=$(cat <<EOF
@ -176,8 +176,8 @@ XML=$(cat <<EOF
</bk:book>
EOF
)
expecteof "$clixon_util_xml -o" 0 "$XML" "$XML"
expecteof "$clixon_util_xml -o" 0 "$XML" '^<bk:book xmlns:bk="urn:loc.gov:books" xmlns:isbn="urn:ISBN:0-395-36341-6"><bk:title>Cheaper by the Dozen</bk:title><isbn:number>1568491379</isbn:number></bk:book>$'
rm -rf $dir
# unset conditional parameters

13
util/clixon_util_path.c
View file

@ -102,11 +102,20 @@ main(int argc,
cbuf *cb = NULL;
int api_path_p = 0; /* api-path or instance-id */
clicon_handle h;
struct stat st;
struct stat st;
cxobj *xcfg = NULL;
/* In the startup, logs to stderr & debug flag set later */
clicon_log_init("api-path", LOG_DEBUG, CLICON_LOG_STDERR);
/* Initialize clixon handle */
if ((h = clicon_handle_init()) == NULL)
goto done;
/* Initialize config tree (needed for -Y below) */
if ((xcfg = xml_new("clixon-config", NULL, NULL)) == NULL)
goto done;
if (clicon_conf_xml_set(h, xcfg) < 0)
goto done;
optind = 1;
opterr = 0;
while ((c = getopt(argc, argv, UTIL_PATH_OPTS)) != -1)
@ -121,7 +130,7 @@ main(int argc,
case 'f': /* XML file */
filename = optarg;
if ((fd = open(filename, O_RDONLY)) < 0){
clicon_err(OE_UNIX, errno, "open(%s)", argv[1]);
clicon_err(OE_UNIX, errno, "open(%s)", optarg);
goto done;
}
break;

6
util/clixon_util_xml.c
View file

@ -113,8 +113,14 @@ main(int argc,
/* In the startup, logs to stderr & debug flag set later */
clicon_log_init(__FILE__, LOG_INFO, CLICON_LOG_STDERR);
/* Initialize clixon handle */
if ((h = clicon_handle_init()) == NULL)
goto done;
if ((xcfg = xml_new("clixon-config", NULL, NULL)) == NULL)
goto done;
if (clicon_conf_xml_set(h, xcfg) < 0)
goto done;
xcfg = xml_new("clixon-config", NULL, NULL);
clicon_conf_xml_set(h, xcfg);

11
util/clixon_util_xpath.c
View file

@ -136,12 +136,19 @@ main(int argc,
struct stat st;
cvec *nsc = NULL;
int canonical = 0;
cxobj *xcfg = NULL;
/* In the startup, logs to stderr & debug flag set later */
clicon_log_init("xpath", LOG_DEBUG, CLICON_LOG_STDERR);
/* Initialize clixon handle */
if ((h = clicon_handle_init()) == NULL)
goto done;
/* Initialize config tree (needed for -Y below) */
if ((xcfg = xml_new("clixon-config", NULL, NULL)) == NULL)
goto done;
if (clicon_conf_xml_set(h, xcfg) < 0)
goto done;
optind = 1;
opterr = 0;
while ((c = getopt(argc, argv, XPATH_OPTS)) != -1)