Merge branch 'master' of https://github.com/clicon/clixon

CHANGELOG.md

@@ -16,6 +16,8 @@
   * Added clixon_util_regexp utility function
   * Added extensive regexp test [test/test_pattern.sh] for both posix and libxml2
   * Added regex cache to type resolution
+* Yang "refine" feature supported
+  * According to RFC 7950 7.13.2
 * Yang "min-element" and "max-element" feature supported
   * According to RFC 7950 7.7.4 and 7.7.5
   * See (tests)[test/test_minmax.sh]
@@ -68,6 +70,29 @@
 ### API changes on existing features (you may need to change your code)
 
 * All hash_ functions have been prefixed with `clicon_` to avoid name collision with other packages (frr)
+* RESTCONF strict namespace validation of data in POST and PUT.
+  * Accepted:
+  ```
+    curl -X PUT http://localhost/restconf/data/mod:a -d {"mod:a":"x"}
+  ```
+  * Not accepted (must prefix "a" with module):
+  ```
+    curl -X PUT http://localhost/restconf/data/mod:a -d {"a":"x"}
+  ```
+  * Undefine `RESTCONF_NS_DATA_CHECK` in include/clixon_custom.h to disable strict check.
+* Many validation functions have changed error parameter from cbuf to xml tree. 
+  * XML trees are more flexible for utility tools
+  * If you use these(mostly internal), you need to change the error function: `generic_validate, from_validate_common, xml_yang_validate_all_top, xml_yang_validate_all, xml_yang_validate_add, xml_yang_validate_rpc, xml_yang_validate_list_key_only`
+* Replaced `CLIXON_DATADIR` with two configurable options defining where Clixon installs Yang files.
+  * use `--with-yang-installdir=DIR` to install Clixon yang files in DIR
+  * use `--with-std-yang-installdir=DIR` to install standard yang files that Clixon may use in DIR 
+  * Default is (as before) `/usr/local/share/clixon`
+* New clixon-config@2019-06-05.yang revision
+  * Added: `CLICON_YANG_REGEXP, CLICON_CLI_TAB_MODE, CLICON_CLI_HIST_FILE, CLICON_CLI_HIST_SIZE, CLICON_XML_CHANGELOG, CLICON_XML_CHANGELOG_FILE`.
+  * Renamed: `CLICON_XMLDB_CACHE` to `CLICON_DATASTORE_CACHE` and type changed.
+  * Deleted: `CLICON_XMLDB_PLUGIN, CLICON_USE_STARTUP_CONFIG`;
+* New clixon-lib@2019-06-05.yang revision
+  * Added: ping rpc added (for liveness)
 * Added compiled regexp parameter as part of internal yang type resolution functions
   * `yang_type_resolve()`, `yang_type_get()`
 * All internal `ys_populate_*()` functions (except ys_populate()) have switched parameters: `clicon_handle, yang_stmt *)`
@@ -167,6 +192,11 @@
 
 ### Minor changes
 
+* `startup_extraxml` triggers unnecessary validation
+  * Renamed startup_db_reset -> xmldb_db_reset (its a general function)
+  * In startup_extraxml(), check if reset callbacks or extraxml file actually makes and changes to the tmp db.
+* Print CLICON_YANG_DIR and CLICON_FEATURE lists on startup with debug flag 
+* Extended `util/clixon_util_xml` with yang and validate functionality so it can be used as a stand-alone utility for validating XML/JSON files
 * JSON parse and print improvements
   * Integrated parsing with namespace translation and yang spec lookup
 * Added CLIgen tab-modes in config option CLICON_CLI_TAB_MODE, which means you can control the behaviour of `<tab>` in the CLI.
@@ -212,6 +242,8 @@
 	
 ### Corrected Bugs
 
+* Fixed a problem with some netconf error messages caused restconf daemon to exit due to no XML encoding
+* Check cligen tab mode, dont start if CLICON_CLI_TAB_MODE is undefined
 * Startup transactions did not mark added tree with XML_FLAG_ADD as it should.
 * Restconf PUT different keys detected (thanks @dcornejo) and fixed
   * This was accepted but shouldn't be: `PUT http://restconf/data/A=hello/B -d '{"B":"goodbye"}'`

README.md

@@ -10,7 +10,7 @@ support.
   * [Frequently asked questions (FAQ)](doc/FAQ.md)
   * [Hello world](example/hello/README.md)
   * [Changelog](CHANGELOG.md)
-  * [Installation](#installation)
+  * [Installation](doc/INSTALL.md)
   * [Licenses](#licenses)
   * [Support](#support)
   * [Dependencies](#dependencies)
@@ -47,21 +47,6 @@ Users of Clixon currently include:
 
 See also [Clicon project page](http://clicon.org).
 
-Clixon runs on Linux, [FreeBSD port](https://www.freshports.org/devel/clixon) and Mac/Apple. CPU architecures include x86_64, i686, ARM32.
-
-## Installation
-
-A typical installation is as follows:
-```
-     configure	       	       # Configure clixon to platform
-     make                      # Compile
-     sudo make install         # Install libs, binaries, and config-files
-     sudo make install-include # Install include files (for compiling)
-```
-
-One [example application](example/README.md) is provided, a IETF IP YANG datamodel with
-generated CLI, Netconf and restconf interface.
-
 ## Licenses
 
 Clixon is open-source and dual licensed. Either Apache License, Version 2.0 or GNU
@@ -113,7 +98,7 @@ Clixon follows:
 However, the following YANG syntax modules are not implemented (reference to RFC7950 in parenthesis):
 - deviation (7.20.3)
 - action (7.15)
-- refine (7.13.2)
+- augment in a uses sub-clause (7.17) (module-level augment is implemented)
 - status (7.21.2)
 - extension (7.19)
 - YIN (13)
@@ -149,23 +134,22 @@ The standards covered include:
 Not supported:
 - !DOCTYPE (ie DTD)
 
-Historically, Clixon has not until 3.9 made strict namespace
+The following xpath axes are supported:
-enforcing. For example, the following non-strict netconf was
+- CHILD, DESCENDANT, DESCENDANT_OR_SELF, SELF, and PARENT
-previously accepted:
+
-```
+The following xpath axes are _not_ supported:
-     <rpc><my-own-method/></rpc> 
+- PRECEEDING, PRECEEDING_SIBLING, NAMESPACE, FOLLOWING_SIBLING, FOLLOWING, ANCESTOR,ANCESTOR_OR_SELF, ATTRIBUTE 
-```
+
-In 3.9, the same statement should be, for example:
+Note that base netconf namespace syntax is not enforced but recommended, which means that the following two expressions are treated equivalently:
-```
-     <rpc><my-own-method xmlns="urn:example:my-own"/></rpc> 
-```
-Note that base netconf syntax is still not enforced but recommended:
 ```
+     <rpc>
+        <my-own-method xmlns="urn:example:my-own"/>
+     </rpc> 
      <rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
         <my-own-method xmlns="urn:example:my-own"/>
      </rpc> 
 ```
-
+All other namespaces are enforced.
 
 ## Netconf
 

apps/backend/backend_client.c

@@ -411,6 +411,7 @@ from_client_edit_config(clicon_handle h,
     cbuf               *cbx = NULL; /* Assist cbuf */
     int                 ret;
     char               *username;
+    cxobj              *xret = NULL;
 
     username = clicon_username_get(h);
     if ((yspec =  clicon_dbspec_yang(h)) == NULL){
@@ -470,10 +471,13 @@ from_client_edit_config(clicon_handle h,
 	    goto ok;
 	}
 	/* xmldb_put (difflist handling) requires list keys */
-	if ((ret = xml_yang_validate_list_key_only(h, xc, cbret)) < 0)
+	if ((ret = xml_yang_validate_list_key_only(h, xc, &xret)) < 0)
 	    goto done;
-	if (ret == 0)
+	if (ret == 0){
+	    if (clicon_xml2cbuf(cbret, xret, 0, 0) < 0)
+		goto done;
 	    goto ok;
+	}
 	/* Cant do this earlier since we dont have a yang spec to
 	 * the upper part of the tree, until we get the "config" tree.
 	 */
@@ -493,6 +497,8 @@ from_client_edit_config(clicon_handle h,
  ok:
     retval = 0;
  done:
+    if (xret)
+	xml_free(xret);
     if (cbx)
 	cbuf_free(cbx);
     clicon_debug(1, "%s done cbret:%s", __FUNCTION__, cbuf_get(cbret));	
@@ -1121,6 +1127,7 @@ from_client_msg(clicon_handle        h,
     yang_stmt           *ye;
     yang_stmt           *ymod;
     cxobj               *xnacm = NULL;
+    cxobj               *xret = NULL;
     
     clicon_debug(1, "%s", __FUNCTION__);
     yspec = clicon_dbspec_yang(h); 
@@ -1147,10 +1154,13 @@ from_client_msg(clicon_handle        h,
      * maybe not necessary since it should be */
     if (xml_spec_populate_rpc(h, x, yspec) < 0)
     	goto done;
-    if ((ret = xml_yang_validate_rpc(h, x, cbret)) < 0)
+    if ((ret = xml_yang_validate_rpc(h, x, &xret)) < 0)
 	goto done;
-    if (ret == 0)
+    if (ret == 0){
+	if (clicon_xml2cbuf(cbret, xret, 0, 0) < 0)
+	    goto done;
 	goto reply;
+    }
     xe = NULL;
     username = xml_find_value(x, "username");
     /* May be used by callbacks, etc */
@@ -1222,6 +1232,8 @@ from_client_msg(clicon_handle        h,
     retval = 0;
   done:  
     clicon_debug(1, "%s retval:%d", __FUNCTION__, retval);
+    if (xret)
+	xml_free(xret);
     if (xt)
 	xml_free(xt);
     if (cbret)

apps/backend/backend_commit.c

@@ -81,7 +81,7 @@
  * are if code comes via XML/NETCONF.
  * @param[in]   yspec   Yang spec
  * @param[in]   td      Transaction data
- * @param[out]  cbret   Cligen buffer containing netconf error (if retval == 0)
+ * @param[out]  xret    Error XML tree. Free with xml_free after use
  * @retval     -1       Error
  * @retval      0       Validation failed (with cbret set)
  * @retval      1       Validation OK       
@@ -90,7 +90,7 @@ static int
 generic_validate(clicon_handle       h,
 		 yang_stmt          *yspec,
 		 transaction_data_t *td,
-		 cbuf               *cbret)
+		 cxobj             **xret)
 {
     int             retval = -1;
     cxobj          *x1;
@@ -100,7 +100,7 @@ generic_validate(clicon_handle       h,
     int             ret;
 
     /* All entries */
-    if ((ret = xml_yang_validate_all_top(h, td->td_target, cbret)) < 0) 
+    if ((ret = xml_yang_validate_all_top(h, td->td_target, xret)) < 0) 
 	goto done;
     if (ret == 0)
 	goto fail;
@@ -109,7 +109,7 @@ generic_validate(clicon_handle       h,
 	x1 = td->td_scvec[i]; /* source changed */
 	x2 = td->td_tcvec[i]; /* target changed */
 	/* Should this be recursive? */
-	if ((ret = xml_yang_validate_add(h, x2, cbret)) < 0)
+	if ((ret = xml_yang_validate_add(h, x2, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -119,7 +119,7 @@ generic_validate(clicon_handle       h,
 	x1 = td->td_dvec[i];
 	ys = xml_spec(x1);
 	if (ys && yang_mandatory(ys) && yang_config(ys)==0){
-	    if (netconf_missing_element(cbret, "protocol", xml_name(x1), "Missing mandatory variable") < 0)
+	    if (netconf_missing_element_xml(xret, "protocol", xml_name(x1), "Missing mandatory variable") < 0)
 		goto done;
 	    goto fail;
 	}
@@ -127,7 +127,7 @@ generic_validate(clicon_handle       h,
     /* added entries */
     for (i=0; i<td->td_alen; i++){
 	x2 = td->td_avec[i];
-	if ((ret = xml_yang_validate_add(h, x2, cbret)) < 0)
+	if ((ret = xml_yang_validate_add(h, x2, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -175,6 +175,7 @@ startup_common(clicon_handle       h,
     modstate_diff_t    *msd = NULL;
     cxobj              *xt = NULL;
     cxobj              *x;
+    cxobj              *xret = NULL;
     
     /* If CLICON_XMLDB_MODSTATE is enabled, then get the db XML with 
      * potentially non-matching module-state in msd
@@ -225,11 +226,13 @@ startup_common(clicon_handle       h,
     /* 5. Make generic validation on all new or changed data.
        Note this is only call that uses 3-values */
     clicon_debug(1, "Validating startup %s", db);
-    if ((ret = generic_validate(h, yspec, td, cbret)) < 0)
+    if ((ret = generic_validate(h, yspec, td, &xret)) < 0)
 	goto done;
-    if (ret == 0)
+    if (ret == 0){
+	if (clicon_xml2cbuf(cbret, xret, 0, 0) < 0)
+	    goto done;
 	goto fail; /* STARTUP_INVALID */
-
+    }
     /* 6. Call plugin transaction validate callbacks */
     if (plugin_transaction_validate(h, td) < 0)
 	goto done;
@@ -240,6 +243,8 @@ startup_common(clicon_handle       h,
  ok:
     retval = 1;
  done:
+    if (xret)
+	xml_free(xret);
     if (xt)
 	xml_free(xt);
     if (msd)
@@ -374,6 +379,7 @@ startup_commit(clicon_handle  h,
  * and call application callback validations.
  * @param[in] h         Clicon handle
  * @param[in] candidate The candidate database. The wanted backend state
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @retval   -1       Error - or validation failed (but cbret not set)
  * @retval    0       Validation failed (with cbret set)
  * @retval    1       Validation OK       
@@ -385,7 +391,7 @@ static int
 from_validate_common(clicon_handle       h, 
 		     char               *candidate,
 		     transaction_data_t *td,
-		     cbuf               *cbret)
+		     cxobj             **xret)
 {
     int         retval = -1;
     yang_stmt  *yspec;
@@ -409,7 +415,7 @@ from_validate_common(clicon_handle       h,
      * But xml_diff requires some basic validation, at least check that yang-specs
      * have been assigned
      */
-    if ((ret = xml_yang_validate_all_top(h, td->td_target, cbret)) < 0)
+    if ((ret = xml_yang_validate_all_top(h, td->td_target, xret)) < 0)
 	goto done;
     if (ret == 0)
 	goto fail;
@@ -462,7 +468,7 @@ from_validate_common(clicon_handle       h,
 
     /* 5. Make generic validation on all new or changed data.
        Note this is only call that uses 3-values */
-    if ((ret = generic_validate(h, yspec, td, cbret)) < 0)
+    if ((ret = generic_validate(h, yspec, td, xret)) < 0)
 	goto done;
     if (ret == 0)
 	goto fail;
@@ -503,6 +509,7 @@ candidate_commit(clicon_handle h,
     int                 retval = -1;
     transaction_data_t *td = NULL;
     int                 ret;
+    cxobj              *xret = NULL;
 
      /* 1. Start transaction */
     if ((td = transaction_new()) == NULL)
@@ -511,10 +518,13 @@ candidate_commit(clicon_handle h,
     /* Common steps (with validate). Load candidate and running and compute diffs
      * Note this is only call that uses 3-values
      */
-    if ((ret = from_validate_common(h, candidate, td, cbret)) < 0)
+    if ((ret = from_validate_common(h, candidate, td, &xret)) < 0)
 	goto done;
-    if (ret == 0)
+    if (ret == 0){
+	if (clicon_xml2cbuf(cbret, xret, 0, 0) < 0)
+	    goto done;
 	goto fail;
+    }
 
      /* 7. Call plugin transaction commit callbacks */
      if (plugin_transaction_commit(h, td) < 0)
@@ -563,7 +573,9 @@ candidate_commit(clicon_handle h,
 	 xmldb_get0_free(h, &td->td_src);
 	 transaction_free(td);
      }
-    return retval;
+     if (xret)
+	 xml_free(xret);
+     return retval;
  fail:
     retval = 0;
     goto done;
@@ -725,6 +737,7 @@ from_client_validate(clicon_handle h,
     transaction_data_t *td = NULL;
     int                 ret;
     char               *db;
+    cxobj             *xret = NULL;
 
     if ((db = netconf_db_find(xe, "source")) == NULL){
 	if (netconf_missing_element(cbret, "protocol", "source", NULL) < 0)
@@ -737,9 +750,15 @@ from_client_validate(clicon_handle h,
     if ((td = transaction_new()) == NULL)
 	goto done;
     /* Common steps (with commit) */
-    if ((ret = from_validate_common(h, db, td, cbret)) < 1){
+    if ((ret = from_validate_common(h, db, td, &xret)) < 1){
+	/* A little complex due to several sources of validation fails or errors.
+	 * (1) xerr is set -> translate to cbret; (2) cbret set use that; otherwise
+	 * use clicon_err. */
+	if (xret && clicon_xml2cbuf(cbret, xret, 0, 0) < 0)
+	    goto done;
 	plugin_transaction_abort(h, td);
-	if (netconf_operation_failed(cbret, "application", clicon_err_reason)< 0)
+	if (!cbuf_len(cbret) &&
+	    netconf_operation_failed(cbret, "application", clicon_err_reason)< 0)
 	    goto done;
 	goto ok;
     }
@@ -773,6 +792,8 @@ from_client_validate(clicon_handle h,
 	 xmldb_get0_free(h, &td->td_src);
 	 transaction_free(td);
     }
+    if (xret)
+	xml_free(xret);
     return retval;
 } /* from_client_validate */
 

apps/backend/backend_main.c

@@ -642,7 +642,7 @@ main(int    argc,
     switch (startup_mode){
     case SM_INIT: /* Scratch running and start from empty */
 	/* [Delete and] create running db */
-	if (startup_db_reset(h, "running") < 0)
+	if (xmldb_db_reset(h, "running") < 0)
 	    goto done;
     case SM_NONE: /* Fall through *
 		   * Load plugins and call plugin_init() */

apps/backend/backend_startup.c

@@ -70,22 +70,6 @@
 #include "backend_commit.h"
 #include "backend_startup.h"
 
-/*! Create an XML database. If it exists already, delete it before creating
- * @param[in]  h   Clixon handle
- * @param[in]  db  Symbolic database name, eg "candidate", "running"
- */
-int
-startup_db_reset(clicon_handle h, 
-		 char         *db)
-{
-    if (xmldb_exists(h, db) == 1){
-	if (xmldb_delete(h, db) != 0 && errno != ENOENT) 
-	    return -1;
-    }
-    if (xmldb_create(h, db) < 0)
-	return -1;
-    return 0;
-}
 
 /*! Merge db1 into db2 without commit 
  * @retval   -1       Error
@@ -235,34 +219,45 @@ startup_extraxml(clicon_handle        h,
 		 cbuf                *cbret)
 {
     int         retval = -1;
-    char       *db = "tmp";
+    char       *tmp_db = "tmp";
     int         ret;
     cxobj	*xt = NULL; /* Potentially upgraded XML */
     
     /* Clear tmp db */
-    if (startup_db_reset(h, db) < 0)
+    if (xmldb_db_reset(h, tmp_db) < 0)
 	goto done;
     /* Application may define extra xml in its reset function*/
-    if (clixon_plugin_reset(h, db) < 0)   
+    if (clixon_plugin_reset(h, tmp_db) < 0)   
 	goto done;
     /* Extra XML can also be added via file */
     if (file){
 	/* Parse and load file into tmp db */
-	if ((ret = load_extraxml(h, file, db, cbret)) < 0)
+	if ((ret = load_extraxml(h, file, tmp_db, cbret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
     }
+    /* 
+     * Check if tmp db is empty.
+     * It should be empty if extra-xml is null and reset plugins did nothing
+     * then skip validation.
+     */
+    if (xmldb_get(h, tmp_db, NULL, &xt) < 0)
+	goto done;
+    if (xt==NULL || xml_child_nr(xt)==0) 
+	goto ok;
+    xml_free(xt);
+    xt = NULL;
     /* Validate the tmp db and return possibly upgraded xml in xt
      */
-    if ((ret = startup_validate(h, db, &xt, cbret)) < 0)
+    if ((ret = startup_validate(h, tmp_db, &xt, cbret)) < 0)
 	goto done;
     if (ret == 0)
 	goto fail;
     if (xt==NULL || xml_child_nr(xt)==0) 
 	goto ok;
     /* Merge tmp into running (no commit) */
-    if ((ret = db_merge(h, db, "running", cbret)) < 0)
+    if ((ret = db_merge(h, tmp_db, "running", cbret)) < 0)
 	goto fail;
     if (ret == 0)
 	goto fail;
@@ -270,7 +265,7 @@ startup_extraxml(clicon_handle        h,
     retval = 1;
  done:
     xmldb_get0_free(h, &xt);
-    if (xmldb_delete(h, db) != 0 && errno != ENOENT) 
+    if (xmldb_delete(h, tmp_db) != 0 && errno != ENOENT) 
 	return -1;
     return retval;
  fail:
@@ -306,7 +301,7 @@ startup_failsafe(clicon_handle h)
     /* Copy original running to tmp as backup (restore if error) */
     if (xmldb_copy(h, "running", "tmp") < 0)
 	goto done;
-    if (startup_db_reset(h, "running") < 0)
+    if (xmldb_db_reset(h, "running") < 0)
 	goto done;
     ret = candidate_commit(h, db, cbret);
     if (ret != 1)

apps/backend/backend_startup.h

@@ -40,7 +40,6 @@
 /*
  * Prototypes
  */ 
-int startup_db_reset(clicon_handle h, char *db);
 int startup_mode_startup(clicon_handle h, char *db, cbuf *cbret);
 int startup_extraxml(clicon_handle h, char *file, cbuf *cbret);
 int startup_failsafe(clicon_handle h);

apps/cli/cli_main.c

@@ -283,6 +283,7 @@ main(int argc, char **argv)
     yang_stmt     *yspecfg = NULL; /* For config XXX clixon bug */
     struct passwd *pw;
     char          *str;
+    int            tabmode;
     
     /* Defaults */
     once = 0;
@@ -542,7 +543,11 @@ main(int argc, char **argv)
 	clicon_log(LOG_WARNING, "No such cli mode: %s (Specify cli mode with CLICON_CLI_MODE in config file or -m <mode> on command line", cli_syntax_mode(h));
 
     /* CLIgen tab mode, ie how <tab>s behave */
-    cligen_tabmode_set(cli_cligen(h), clicon_cli_tab_mode(h));
+    if ((tabmode = clicon_cli_tab_mode(h)) < 0){
+	fprintf(stderr, "FATAL: CLICON_CLI_TAB_MODE not set\n");
+	goto done;
+    }
+    cligen_tabmode_set(cli_cligen(h), tabmode);
 
     if (logclisyntax)
 	cli_logsyntax_set(h, logclisyntax);

apps/netconf/netconf_main.c

@@ -84,7 +84,7 @@ static int ignore_packet_errors = 1;
  */
 static int
 netconf_input_packet(clicon_handle h, 
-			cbuf         *cb)
+		     cbuf         *cb)
 {
     int        retval = -1;
     char      *str;
@@ -125,9 +125,10 @@ netconf_input_packet(clicon_handle h,
         isrpc++;
 	if (xml_spec_populate_rpc(h, xrpc, yspec) < 0)
 	    goto done;
-	if ((ret = xml_yang_validate_rpc(h, xrpc, cbret)) < 0) 
+	if ((ret = xml_yang_validate_rpc(h, xrpc, &xret)) < 0) 
 	    goto done;
 	if (ret == 0){
+	    clicon_xml2cbuf(cbret, xret, 0, 0);
 	    netconf_output_encap(1, cbret, "rpc-error");
 	    goto ok;
 	}
@@ -155,7 +156,7 @@ netconf_input_packet(clicon_handle h,
 		netconf_output_encap(1, cbret, "rpc-error");
 		goto done;
 	    }
-	    if ((xc = xml_child_i(xret,0))!=NULL){
+	    if ((xc = xml_child_i(xret, 0))!=NULL){
 		xa=NULL;
 		/* Copy message-id attribute from incoming to reply. 
 		 * RFC 6241:

apps/netconf/netconf_rpc.c

@@ -547,6 +547,7 @@ netconf_application_rpc(clicon_handle h,
     yang_stmt     *yinput;
     yang_stmt     *youtput;
     cxobj         *xoutput;
+    cxobj         *xerr = NULL;
     cbuf          *cb = NULL;
     cbuf          *cbret = NULL;
     int            ret;
@@ -587,15 +588,13 @@ netconf_application_rpc(clicon_handle h,
 	    xml_spec_set(xn, yinput); /* needed for xml_spec_populate */
 	    if (xml_apply(xn, CX_ELMNT, xml_spec_populate, yspec) < 0)
 		goto done;
-	    if ((ret = xml_yang_validate_all_top(h, xn, cbret)) < 0)
+	    if ((ret = xml_yang_validate_all_top(h, xn, &xerr)) < 0)
-		goto done;
+		goto done;
-	    if (ret == 0){
+	    if (ret > 0 && (ret = xml_yang_validate_add(h, xn, &xerr)) < 0)
-		netconf_output_encap(1, cbret, "rpc-error");
-		goto ok;
-	    }
-	    if ((ret = xml_yang_validate_add(h, xn, cbret)) < 0)
 		goto done;
 	    if (ret == 0){
+		if (clicon_xml2cbuf(cbret, xerr, 0, 0) < 0)
+		    goto done;		
 		netconf_output_encap(1, cbret, "rpc-error");
 		goto ok;
 	    }
@@ -622,15 +621,13 @@ netconf_application_rpc(clicon_handle h,
 	    if (xml_apply(xoutput, CX_ELMNT, xml_spec_populate, yspec) < 0)
 		goto done;
 
-	    if ((ret = xml_yang_validate_all_top(h, xoutput, cbret)) < 0)
+	    if ((ret = xml_yang_validate_all_top(h, xoutput, &xerr)) < 0)
-		goto done;
+		goto done;
-	    if (ret == 0){
+	    if (ret > 0 && (ret = xml_yang_validate_add(h, xoutput, &xerr)) < 0)
-		clicon_log(LOG_WARNING, "Errors in output netconf %s", cbuf_get(cbret));
-		goto ok;
-	    }
-	    if ((ret = xml_yang_validate_add(h, xoutput, cbret)) < 0)
 		goto done;
 	    if (ret == 0){
+		if (clicon_xml2cbuf(cbret, xerr, 0, 0) < 0)
+		    goto done;		
 		clicon_log(LOG_WARNING, "Errors in output netconf %s", cbuf_get(cbret));
 		goto ok;
 	    }
@@ -641,6 +638,8 @@ netconf_application_rpc(clicon_handle h,
  ok:
     retval = 0;
  done:
+    if (xerr)
+	xml_free(xerr);
     if (cb)
 	cbuf_free(cb);
     if (cbret)

apps/restconf/restconf_methods.c

@@ -437,13 +437,17 @@ api_data_post(clicon_handle h,
 {
     int        retval = -1;
     enum operation_type op = OP_CREATE;
+    cxobj     *xdata0 = NULL; /* Original -d data struct (including top symbol) */
+    cxobj     *xdata;         /* -d data (without top symbol)*/
     int        i;
-    cxobj     *xdata = NULL;
     cbuf      *cbx = NULL;
-    cxobj     *xtop = NULL; /* xpath root */
+    cxobj     *xtop = NULL; /* top of api-path */
-    cxobj     *xbot = NULL;
+    cxobj     *xbot = NULL; /* bottom of api-path */
-    cxobj     *x;
+    yang_stmt *ybot = NULL; /* yang of xbot */
-    yang_stmt *y = NULL;
+#ifdef RESTCONF_NS_DATA_CHECK
+    yang_stmt *ymodapi = NULL; /* yang module of api-path (if any) */
+    yang_stmt *ymoddata = NULL; /* yang module of data (-d) */
+#endif
     yang_stmt *yspec;
     cxobj     *xa;
     cxobj     *xret = NULL;
@@ -469,8 +473,12 @@ api_data_post(clicon_handle h,
     /* Translate api_path to xtop/xbot */
     xbot = xtop;
     if (api_path){
-	if ((ret = api_path2xml(api_path, yspec, xtop, YC_DATANODE, 1, &xbot, &y)) < 0)
+	if ((ret = api_path2xml(api_path, yspec, xtop, YC_DATANODE, 1, &xbot, &ybot)) < 0)
 	    goto done;
+#ifdef RESTCONF_NS_DATA_CHECK
+	if (ybot)
+	    ymodapi=ys_module(ybot);
+#endif
 	if (ret == 0){ /* validation failed */
 	    if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 		goto done;
@@ -486,7 +494,7 @@ api_data_post(clicon_handle h,
     }
     /* Parse input data as json or xml into xml */
     if (parse_xml){
-	if (xml_parse_string(data, NULL, &xdata) < 0){
+	if (xml_parse_string(data, NULL, &xdata0) < 0){
 	    if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 		goto done;
 	    if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -499,7 +507,7 @@ api_data_post(clicon_handle h,
 	}
     }
     else {
-	if ((ret = json_parse_str(data, yspec, &xdata, &xerr)) < 0){ 
+	if ((ret = json_parse_str(data, yspec, &xdata0, &xerr)) < 0){ 
 	    if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 		goto done;
 	    if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -523,7 +531,7 @@ api_data_post(clicon_handle h,
     /* 4.4.1: The message-body MUST contain exactly one instance of the
      * expected data resource. 
      */
-    if (xml_child_nr(xdata) != 1){
+    if (xml_child_nr(xdata0) != 1){
 	if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 	    goto done;
 	if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -534,15 +542,46 @@ api_data_post(clicon_handle h,
 	    goto done;
 	goto ok;
     }
-    x = xml_child_i(xdata,0);
+    xdata = xml_child_i(xdata0,0);
+#ifdef RESTCONF_NS_DATA_CHECK
+    /* If the api-path (above) defines a module, then xdata must have a prefix
+     * and it match the module defined in api-path. 
+     * In a POST, maybe there are cornercases where xdata (which is a child) and
+     * xbot (which is the parent) may have non-matching namespaces?
+     * This does not apply if api-path is / (no module)
+     */
+    if (ys_module_by_xml(yspec, xdata, &ymoddata) < 0)
+	goto done;
+    if (ymoddata && ymodapi){
+	if (ymoddata != ymodapi){
+	     if (netconf_malformed_message_xml(&xerr, "Data is not prefixed with matching namespace") < 0)
+		 goto done;
+	    if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
+	    		clicon_err(OE_XML, EINVAL, "rpc-error not found (internal error)");
+	    		goto done;
+	    }
+    if (debug){
+	cbuf *ccc=cbuf_new();
+	if (clicon_xml2cbuf(ccc, xe, 0, 0) < 0)
+	    goto done;
+	clicon_debug(1, "%s XE:%s", __FUNCTION__, cbuf_get(ccc));
+    }
+
+	    if (api_return_err(h, r, xe, pretty, use_xml) < 0)
+		goto done;
+	    goto ok;
+	}
+    }
+#endif  /* RESTCONF_NS_DATA_CHECK */
+
     /* Add operation (create/replace) as attribute */
-    if ((xa = xml_new("operation", x, NULL)) == NULL)
+    if ((xa = xml_new("operation", xdata, NULL)) == NULL)
 	goto done;
     xml_type_set(xa, CX_ATTR);
     if (xml_value_set(xa, xml_operation2str(op)) < 0)
 	goto done;
     /* Replace xbot with x, ie bottom of api-path with data */
-    if (xml_addsub(xbot, x) < 0)
+    if (xml_addsub(xbot, xdata) < 0)
 	goto done;
     /* Create text buffer for transfer to backend */
     if ((cbx = cbuf_new()) == NULL)
@@ -626,8 +665,8 @@ api_data_post(clicon_handle h,
 	xml_free(xretdis);
     if (xtop)
 	xml_free(xtop);
-    if (xdata)
+    if (xdata0)
-	xml_free(xdata);
+	xml_free(xdata0);
      if (cbx)
 	cbuf_free(cbx); 
    return retval;
@@ -636,11 +675,14 @@ api_data_post(clicon_handle h,
 
 /*! Check matching keys
  *
+ * Check that x1 and x2 are of type list/leaf-list and share the same key statements
+ * I.e that if x1=<list><key>b</key></list> then x2 = <list><key>b</key></list> as 
+ * well. Otherwise return -1.
  * @param[in] y        Yang statement, should be list or leaf-list
- * @param[in] xdata    XML data tree
+ * @param[in] x1       First XML tree (eg data)
- * @param[in] xapipath XML api-path tree
+ * @param[in] x2       Second XML tree (eg api-path)
  * @retval    0        Yes, keys match
- * @retval    -1        No keys do not match
+ * @retval    -1       No, keys do not match
  * If the target resource represents a YANG leaf-list, then the PUT
  * method MUST NOT change the value of the leaf-list instance.
  *
@@ -651,17 +693,17 @@ api_data_post(clicon_handle h,
  */
 static int
 match_list_keys(yang_stmt *y,
-		cxobj     *xdata,
+		cxobj     *x1,
-		cxobj     *xapipath)
+		cxobj     *x2)
 {
     int        retval = -1;
     cvec      *cvk = NULL; /* vector of index keys */
     cg_var    *cvi;
     char      *keyname;
-    cxobj     *xkeya; /* xml key object in api-path */
+    cxobj     *xkey1; /* xml key object of x1 */
-    cxobj     *xkeyd; /* xml key object in data */
+    cxobj     *xkey2; /* xml key object of x2 */
-    char      *keya;
+    char      *key1;
-    char      *keyd;
+    char      *key2;
 
     clicon_debug(1, "%s", __FUNCTION__);
     switch (yang_keyword_get(y)){
@@ -670,29 +712,30 @@ match_list_keys(yang_stmt *y,
 	cvi = NULL;
 	while ((cvi = cvec_each(cvk, cvi)) != NULL) {
 	    keyname = cv_string_get(cvi);	    
-	    if ((xkeya = xml_find(xapipath, keyname)) == NULL)
+	    if ((xkey2 = xml_find(x2, keyname)) == NULL)
 		goto done; /* No key in api-path */
-	    if ((keya = xml_body(xkeya)) == NULL)
+	    if ((key2 = xml_body(xkey2)) == NULL)
 		goto done;
-	    if ((xkeyd = xml_find(xdata, keyname)) == NULL)
+	    if ((xkey1 = xml_find(x1, keyname)) == NULL)
 		goto done; /* No key in data */
-	    if ((keyd = xml_body(xkeyd)) == NULL)
+	    if ((key1 = xml_body(xkey1)) == NULL)
 		goto done;
-	    if (strcmp(keya, keyd) != 0)
+	    if (strcmp(key2, key1) != 0)
 		goto done; /* keys dont match */
 	}
 	break;
     case Y_LEAF_LIST:
-	if ((keya = xml_body(xapipath)) == NULL)
+	if ((key2 = xml_body(x2)) == NULL)
 	    goto done; /* No key in api-path */
-	if ((keyd = xml_body(xdata)) == NULL)
+	if ((key1 = xml_body(x1)) == NULL)
 	    goto done; /* No key in data */
-	if (strcmp(keya, keyd) != 0)
+	if (strcmp(key2, key1) != 0)
 	    goto done; /* keys dont match */
 	break;
     default:
-	goto done;
+	goto ok;
     }
+ ok:
     retval = 0;
  done:
     clicon_debug(1, "%s retval:%d", __FUNCTION__, retval);
@@ -738,13 +781,17 @@ api_data_put(clicon_handle h,
     int        retval = -1;
     enum operation_type op = OP_REPLACE;
     int        i;
-    cxobj     *xdata = NULL;
+    cxobj     *xdata0 = NULL; /* Original -d data struct (including top symbol) */
+    cxobj     *xdata;         /* -d data (without top symbol)*/
     cbuf      *cbx = NULL;
-    cxobj     *xtop = NULL; /* xpath root */
+    cxobj     *xtop = NULL; /* top of api-path */
-    cxobj     *xbot = NULL;
+    cxobj     *xbot = NULL; /* bottom of api-path */
+    yang_stmt *ybot = NULL; /* yang of xbot */
+#ifdef RESTCONF_NS_DATA_CHECK
+    yang_stmt *ymodapi = NULL; /* yang module of api-path (if any) */
+    yang_stmt *ymoddata = NULL; /* yang module of data (-d) */
+#endif
     cxobj     *xparent;
-    cxobj     *x;
-    yang_stmt *y = NULL;
     yang_stmt *yp; /* yang parent */
     yang_stmt *yspec;
     cxobj     *xa;
@@ -757,6 +804,7 @@ api_data_put(clicon_handle h,
     char      *username;
     int        ret;
     char      *namespace0;
+    char      *dname;
 
     clicon_debug(1, "%s api_path:\"%s\" json:\"%s\"",
 		 __FUNCTION__, api_path0, data);
@@ -773,8 +821,12 @@ api_data_put(clicon_handle h,
     /* Translate api_path to xtop/xbot */
     xbot = xtop;
     if (api_path){
-	if ((ret = api_path2xml(api_path, yspec, xtop, YC_DATANODE, 1, &xbot, &y)) < 0)
+	if ((ret = api_path2xml(api_path, yspec, xtop, YC_DATANODE, 1, &xbot, &ybot)) < 0)
 	    goto done;
+#ifdef RESTCONF_NS_DATA_CHECK
+	if (ybot)
+	    ymodapi=ys_module(ybot);
+#endif
 	if (ret == 0){ /* validation failed */
 	    if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 		goto done;
@@ -788,9 +840,10 @@ api_data_put(clicon_handle h,
 	    goto ok;
 	}
     }
+
     /* Parse input data as json or xml into xml */
     if (parse_xml){
-	if (xml_parse_string(data, NULL, &xdata) < 0){
+	if (xml_parse_string(data, yspec, &xdata0) < 0){
 	    if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 		goto done;
 	    if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -803,7 +856,7 @@ api_data_put(clicon_handle h,
 	}
     }
     else{
-	if ((ret = json_parse_str(data, yspec, &xdata, &xerr)) < 0){
+	if ((ret = json_parse_str(data, yspec, &xdata0, &xerr)) < 0){
 	    if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 		goto done;
 	    if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -827,7 +880,7 @@ api_data_put(clicon_handle h,
     /* The message-body MUST contain exactly one instance of the
      * expected data resource. 
      */
-    if (xml_child_nr(xdata) != 1){
+    if (xml_child_nr(xdata0) != 1){
 	if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
 	    goto done;
 	if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -838,34 +891,56 @@ api_data_put(clicon_handle h,
 	    goto done;
 	goto ok;
     }
-    x = xml_child_i(xdata,0);
+    xdata = xml_child_i(xdata0,0);
+#ifdef RESTCONF_NS_DATA_CHECK
+    /* If the api-path (above) defines a module, then xdata must have a prefix
+     * and it match the module defined in api-path
+     * This does not apply if api-path is / (no module)
+     */
+    if (ys_module_by_xml(yspec, xdata, &ymoddata) < 0)
+	goto done;
+    if (ymoddata && ymodapi){
+	if (ymoddata != ymodapi){
+	    if (netconf_malformed_message_xml(&xerr, "Data is not prefixed with matching namespace") < 0)
+		goto done;
+	    if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
+		clicon_err(OE_XML, EINVAL, "rpc-error not found (internal error)");
+		goto done;
+	    }
+	    if (api_return_err(h, r, xe, pretty, use_xml) < 0)
+		goto done;
+	    goto ok;
+	}
+    }
+#endif /* RESTCONF_NS_DATA_CHECK */
+
     /* Add operation (create/replace) as attribute */
-    if ((xa = xml_new("operation", x, NULL)) == NULL)
+    if ((xa = xml_new("operation", xdata, NULL)) == NULL)
 	goto done;
     xml_type_set(xa, CX_ATTR);
     if (xml_value_set(xa, xml_operation2str(op)) < 0)
 	goto done;
-#if 0
+
-    if (debug){
+    /* Top-of tree, no api-path
-	cbuf *ccc=cbuf_new();
+     * Replace xparent with x, ie bottom of api-path with data 
-	if (clicon_xml2cbuf(ccc, xdata, 0, 0) < 0)
+     */	    
-	    goto done;
+    dname = xml_name(xdata);
-	clicon_debug(1, "%s DATA:%s", __FUNCTION__, cbuf_get(ccc));
+    if (api_path==NULL && strcmp(dname,"data")==0){
-    }
+	if (xml_addsub(NULL, xdata) < 0)
-#endif
-    /* Replace xparent with x, ie bottom of api-path with data */	    
-    if (api_path==NULL && strcmp(xml_name(x),"data")==0){
-	if (xml_addsub(NULL, x) < 0)
 	    goto done;
 	if (xtop)
 	    xml_free(xtop);
-	xtop = x;
+	xtop = xdata;
 	xml_name_set(xtop, "config");
     }
     else {
-	clicon_debug(1, "%s x:%s xbot:%s",__FUNCTION__, xml_name(x), xml_name(xbot));
+	/* There is an api-path that defines an element in the datastore tree.
+	 * Not top-of-tree.
+	 */
+	clicon_debug(1, "%s x:%s xbot:%s",__FUNCTION__, dname, xml_name(xbot));
+
 	/* Check same symbol in api-path as data */	    
-	if (strcmp(xml_name(x), xml_name(xbot))){
+	if (strcmp(dname, xml_name(xbot))){
 	    if (netconf_operation_failed_xml(&xerr, "protocol", "Not same symbol in api-path as data") < 0)
 		goto done;
 	    if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -884,27 +959,13 @@ api_data_put(clicon_handle h,
 	 * That is why the conditional is somewhat hairy
 	 */	    
 	xparent = xml_parent(xbot);
-#if 1
+	if (ybot){
-    if (debug){
+	    /* Ensure list keys match between uri and data. That is:
-	cbuf *ccc=cbuf_new();
+	     * If data is on the form: -d {"a":{"k":1}} where a is list or leaf-list
-	if (clicon_xml2cbuf(ccc, xtop, 0, 0) < 0)
+	     * then uri-path must be ../a=1
-	    goto done;
+	     * match_list_key() checks if this is true
-	clicon_debug(1, "%s AAA XPATH:%s", __FUNCTION__, cbuf_get(ccc));
+	     */
-    }
+	    if (match_list_keys(ybot, xdata, xbot) < 0){
-    if (debug){
-	cbuf *ccc=cbuf_new();
-	if (clicon_xml2cbuf(ccc, xdata, 0, 0) < 0)
-	    goto done;
-	clicon_debug(1, "%s DATA:%s", __FUNCTION__, cbuf_get(ccc));
-    }
-#endif
-
-	if (y){
-	    yp = yang_parent_get(y);
-	    if (((yang_keyword_get(y) == Y_LIST || yang_keyword_get(y) == Y_LEAF_LIST) &&
-		 match_list_keys(y, x, xbot) < 0) ||
-		(yp && yang_keyword_get(yp) == Y_LIST &&
-		 match_list_keys(yp, xml_parent(x), xparent) < 0)){
 		if (netconf_operation_failed_xml(&xerr, "protocol", "api-path keys do not match data keys") < 0)
 		    goto done;
 		if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
@@ -914,15 +975,40 @@ api_data_put(clicon_handle h,
 		if (api_return_err(h, r, xe, pretty, use_xml) < 0)
 		    goto done;
 		goto ok;
-		
+	    }
+	    /* Ensure keys in lists are not changed. That is:
+	     * If data is on the form: -d {"k":1} and its parent is a list "a"
+	     * then the uri-path must be "../a=1 (you cannot change a's key)"
+	     */
+	    if ((yp = yang_parent_get(ybot)) != NULL &&
+		yang_keyword_get(yp) == Y_LIST){
+		if ((ret = yang_key_match(yp, dname)) < 0)
+		    goto done;
+		if (ret == 1){ /* Match: xdata is a key */
+		    char *parbod = xml_find_body(xparent, dname);
+		    /* Check if the key is different from the one in uri-path,
+		     * or does not exist
+		     */
+		    if (parbod == NULL || strcmp(parbod, xml_body(xdata))){
+			if (netconf_operation_failed_xml(&xerr, "protocol", "api-path keys do not match data keys") < 0)
+			    goto done;
+			if ((xe = xpath_first(xerr, "rpc-error")) == NULL){
+			    clicon_err(OE_XML, EINVAL, "rpc-error not found (internal error)");
+			    goto done;
+			}
+			if (api_return_err(h, r, xe, pretty, use_xml) < 0)
+			    goto done;
+			goto ok;
+		    }
+		}
 	    }
 	}
 
 	xml_purge(xbot);
-	if (xml_addsub(xparent, x) < 0)
+	if (xml_addsub(xparent, xdata) < 0)
 	    goto done;
 	/* If we already have that default namespace, remove it in child */
-	if ((xa = xml_find_type(x, NULL, "xmlns", CX_ATTR)) != NULL){
+	if ((xa = xml_find_type(xdata, NULL, "xmlns", CX_ATTR)) != NULL){
 	    if (xml2ns(xparent, NULL, &namespace0) < 0)
 		goto done;
 	    /* Set xmlns="" default namespace attribute (if diff from default) */
@@ -945,6 +1031,7 @@ api_data_put(clicon_handle h,
     clicon_debug(1, "%s xml: %s api_path:%s",__FUNCTION__, cbuf_get(cbx), api_path);
     if (clicon_rpc_netconf(h, cbuf_get(cbx), &xret, NULL) < 0)
 	goto done;
+
     if ((xe = xpath_first(xret, "//rpc-error")) != NULL){
 	if (api_return_err(h, r, xe, pretty, use_xml) < 0)
 	    goto done;
@@ -1010,8 +1097,8 @@ api_data_put(clicon_handle h,
 	xml_free(xretdis);
     if (xtop)
 	xml_free(xtop);
-    if (xdata)
+    if (xdata0)
-	xml_free(xdata);
+	xml_free(xdata0);
      if (cbx)
 	cbuf_free(cbx); 
    return retval;
@@ -1454,14 +1541,9 @@ api_operations_post_output(clicon_handle h,
     cxobj     *xa;          /* xml attribute (xmlns) */
     cxobj     *x;
     cxobj     *xok;
-    cbuf      *cbret = NULL;
     int        isempty;
     
     //    clicon_debug(1, "%s", __FUNCTION__);
-    if ((cbret = cbuf_new()) == NULL){
-	clicon_err(OE_UNIX, 0, "cbuf_new");
-	goto done;
-    }
     /* Validate that exactly only <rpc-reply> tag */
     if ((xoutput = xml_child_i_type(xret, 0, CX_ELMNT)) == NULL ||
 	strcmp(xml_name(xoutput),"rpc-reply") != 0 ||
@@ -1499,14 +1581,12 @@ api_operations_post_output(clicon_handle h,
 #if 0
 	if (xml_apply(xoutput, CX_ELMNT, xml_spec_populate, yspec) < 0)
 	    goto done;
-	if ((ret = xml_yang_validate_all(xoutput, cbret)) < 0)
+	if ((ret = xml_yang_validate_all(xoutput, &xerr)) < 0)
 	    goto done;
 	if (ret == 1 &&
-	    (ret = xml_yang_validate_add(h, xoutput, cbret)) < 0)
+	    (ret = xml_yang_validate_add(h, xoutput, &xerr)) < 0)
 	    goto done;
 	if (ret == 0){ /* validation failed */
-	    if (xml_parse_string(cbuf_get(cbret), yspec, &xerr) < 0)
-		goto done;
 	    if ((xe = xpath_first(xerr, "rpc-reply/rpc-error")) == NULL){
 		clicon_err(OE_XML, EINVAL, "rpc-error not found (internal error)");
 		goto done;
@@ -1551,8 +1631,6 @@ api_operations_post_output(clicon_handle h,
     retval = 1;
  done:
     clicon_debug(1, "%s retval: %d", __FUNCTION__, retval);
-    if (cbret)
-	cbuf_free(cbret);
     if (xerr)
 	xml_free(xerr);
     return retval;
@@ -1738,14 +1816,12 @@ api_operations_post(clicon_handle h,
     /* 6. Validate incoming RPC and fill in defaults */
     if (xml_spec_populate_rpc(h, xtop, yspec) < 0) /*  */
 	goto done;
-    if ((ret = xml_yang_validate_rpc(h, xtop, cbret)) < 0)
+    if ((ret = xml_yang_validate_rpc(h, xtop, &xret)) < 0)
 	goto done;
     if (ret == 0){
-	if (xml_parse_string(cbuf_get(cbret), NULL, &xret) < 0)
+	if ((xe = xpath_first(xret, "rpc-error")) == NULL){
-	    goto done;
-	if ((xe = xpath_first(xret, "rpc-reply/rpc-error")) == NULL){
 	    clicon_err(OE_XML, EINVAL, "rpc-error not found (internal error)");
-	    goto done;
+	    goto ok;
 	}
 	if (api_return_err(h, r, xe, pretty, use_xml) < 0)
 	    goto done;

configure

@@ -621,7 +621,8 @@ ac_includes_default="\
 
 ac_subst_vars='LTLIBOBJS
 LIBOBJS
-CLIXON_DATADIR
+STD_YANG_INSTALLDIR
+YANG_INSTALLDIR
 EGREP
 GREP
 LEXLIB
@@ -637,7 +638,6 @@ wwwuser
 wwwdir
 enable_stdyangs
 with_restconf
-RANLIB
 SH_SUFFIX
 CLIXON_DEFAULT_CONFIG
 INSTALLFLAGS
@@ -718,6 +718,8 @@ with_restconf
 with_wwwuser
 with_configfile
 with_libxml2
+with_yang_installdir
+with_std_yang_installdir
 '
       ac_precious_vars='build_alias
 host_alias
@@ -1367,6 +1369,8 @@ Optional Packages:
   --with-wwwuser=<user>   Set www user different from www-data
   --with-configfile=FILE  set default path to config file
   --with-libxml2          use gnome/libxml2 regex engine
+  --with-yang-installdir=DIR  Install Clixon yang files here (default: ${prefix}/share/clixon)
+  --with-std-yang-installdir=DIR  Install standard yang files here (default: ${prefix}/share/clixon)
 
 Some influential environment variables:
   CC          C compiler command
@@ -2463,7 +2467,6 @@ test -z "$INSTALL_DATA" && INSTALL_DATA='${INSTALL} -m 644'
 
 
 
-
  # If yes, compile apps/restconf
 
 wwwdir=/www-data
@@ -4493,13 +4496,39 @@ fi
 done
 
 
-# CLIXON_DATADIR is where clixon installs the "system" yang files in yang/Makfile
+# YANG_INSTALLDIR is where clixon installs the Clixon yang files
-# This directory should most probably be included in each application,
+# (the files in in yang/clixon)
-# so each application designer may need to place CLIXON_DATADIR in their config
+# Each application designer may need to place CLIXON_YANG_DIR in their config:
-# (last in yang dir list):
+# <CLIXON-YANG-DIR>$YANG_INSTALLDIR</CLIXON-YANG_DIR>
-# <CLIXON-YANG-DIR>$CLIXON_DATADIR</CLIXON-YANG_DIR>
 
-CLIXON_DATADIR="${prefix}/share/clixon"
+# Check whether --with-yang-installdir was given.
+if test "${with_yang_installdir+set}" = set; then :
+  withval=$with_yang_installdir; YANG_INSTALLDIR="$withval"
+else
+  YANG_INSTALLDIR="${prefix}/share/clixon"
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: Clixon yang files are installed in ${YANG_INSTALLDIR}" >&5
+$as_echo "Clixon yang files are installed in ${YANG_INSTALLDIR}" >&6; }
+
+# STD_YANG_INSTALLDIR is where clixon installs standard yang files
+# (the files in in yang/standard)
+# that Clixon needs to run (or examples rely on). These may be retreived from
+# elsewhere (eg yangmodels repo)
+
+# Check whether --with-std-yang-installdir was given.
+if test "${with_std_yang_installdir+set}" = set; then :
+  withval=$with_std_yang_installdir; STD_YANG_INSTALLDIR="$withval"
+else
+  STD_YANG_INSTALLDIR="${prefix}/share/clixon"
+
+fi
+
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: Standard yang files are installed in ${STD_YANG_INSTALLDIR}" >&5
+$as_echo "Standard yang files are installed in ${STD_YANG_INSTALLDIR}" >&6; }
 
 # Default location for config file
 

configure.ac

@@ -89,7 +89,6 @@ AC_SUBST(INSTALLFLAGS)
 AC_SUBST(CLIXON_DEFAULT_CONFIG)
 AC_SUBST(LIBS)
 AC_SUBST(SH_SUFFIX)
-AC_SUBST(RANLIB)
 AC_SUBST(with_restconf) # If yes, compile apps/restconf
 AC_SUBST(enable_stdyangs) 
 AC_SUBST(wwwdir,/www-data)
@@ -232,13 +231,29 @@ fi
 #
 AC_CHECK_FUNCS(inet_aton sigaction sigvec strlcpy strsep strndup alphasort versionsort)
 
-# CLIXON_DATADIR is where clixon installs the "system" yang files in yang/Makfile
+# YANG_INSTALLDIR is where clixon installs the Clixon yang files
-# This directory should most probably be included in each application,
+# (the files in in yang/clixon)
-# so each application designer may need to place CLIXON_DATADIR in their config
+# Each application designer may need to place YANG_INSTALLDIR in their config:
-# (last in yang dir list):
+# <CLICON_YANG_DIR>$YANG_INSTALLDIR</CLICON_YANG_DIR>
-# <CLIXON-YANG-DIR>$CLIXON_DATADIR</CLIXON-YANG_DIR>
+AC_ARG_WITH(yang-installdir,
-AC_SUBST(CLIXON_DATADIR)
+            [  --with-yang-installdir=DIR  Install Clixon yang files here (default: ${prefix}/share/clixon) ],
-CLIXON_DATADIR="${prefix}/share/clixon"
+	    [YANG_INSTALLDIR="$withval"],
+    	    [YANG_INSTALLDIR="${prefix}/share/clixon"]		
+	    )
+AC_SUBST(YANG_INSTALLDIR)
+AC_MSG_RESULT(Clixon yang files are installed in ${YANG_INSTALLDIR})	
+
+# STD_YANG_INSTALLDIR is where clixon installs standard yang files
+# (the files in in yang/standard)
+# that Clixon needs to run (or examples rely on). These may be retreived from
+# elsewhere (eg yangmodels repo)
+AC_ARG_WITH(std-yang-installdir,
+            [  --with-std-yang-installdir=DIR  Install standard yang files here (default: ${prefix}/share/clixon) ],
+	    [STD_YANG_INSTALLDIR="$withval"],
+    	    [STD_YANG_INSTALLDIR="${prefix}/share/clixon"]		
+	    )
+AC_SUBST(STD_YANG_INSTALLDIR)
+AC_MSG_RESULT(Standard yang files are installed in ${STD_YANG_INSTALLDIR})	
 
 # Default location for config file
 AC_DEFINE_UNQUOTED(CLIXON_DEFAULT_CONFIG,"${CLIXON_DEFAULT_CONFIG}",[Location for apps to find default config file])

doc/FAQ.md

@@ -305,7 +305,7 @@ The following configuration file options control the loading of Yang files:
 - `CLICON_YANG_MODULE_REVISION` : Specifies a revision to the main module. 
 - `CLICON_YANG_MAIN_DIR` - Load all yang modules in this directory.
 
-Note that the special `CLIXON_DATADIR`, by default `/usr/local/share/clixon` should be included in the yang dir path for Clixon system files to be found.
+Note that the special `YANG_INSTALLDIR`, by default `/usr/local/share/clixon` should be included in the yang dir path for Clixon system files to be found.
 
 You can combine the options, however, if there are different variants
 of the same module, more specific options override less

doc/INSTALL.md

@@ -0,0 +1,47 @@
+# Building Clixon
+
+Clixon runs on Linux, [FreeBSD port](https://www.freshports.org/devel/clixon) and Mac/Apple. CPU architecures include x86_64, i686, ARM32.
+
+## Ubuntu Linux
+
+### Installing dependencies
+
+Install packages
+```
+sudo apt-get update
+sudo apt-get install flex bison fcgi-dev curl-dev
+```
+
+Install and build CLIgen
+```
+    git clone https://github.com/olofhagsand/cligen.git
+    cd cligen;
+    configure;
+    make;
+    make install
+```
+
+Add a user group, using groupadd and usermod:
+```
+  sudo groupadd clicon # 
+  sudo usermod -a -G clicon <user>
+  sudo usermod -a -G clicon www-data
+```
+
+
+### Build from source
+```
+     configure	       	       # Configure clixon to platform
+     make                      # Compile
+     sudo make install         # Install libs, binaries, and config-files
+     sudo make install-include # Install include files (for compiling)
+```
+
+## Alpine Linux
+Docker is used to build Alpine Linux 
+### Build docker image
+
+## FreeBSD
+### Package install
+### Build from source
+

example/hello/Makefile.in

@@ -45,7 +45,7 @@ libdir          = @exec_prefix@/lib
 APPNAME	        = hello
 
 # Here is where example yang appears
-CLIXON_DATADIR  = @CLIXON_DATADIR@
+YANG_INSTALLDIR  = @YANG_INSTALLDIR@
 # Install here if you want default clixon location:
 CLIXON_DEFAULT_CONFIG = @CLIXON_DEFAULT_CONFIG@
 
@@ -125,7 +125,7 @@ install: $(YANGSPECS) $(CLISPECS) $(PLUGINS) $(APPNAME).xml
 	install -d -m 0755 $(DESTDIR)$(libdir)/$(APPNAME)/clispec
 	install -m 0644 $(CLISPECS) $(DESTDIR)$(libdir)/$(APPNAME)/clispec
 	install -d -m 0755 $(DESTDIR)$(datarootdir)/$(APPNAME)/yang
-	install -m 0644 $(YANGSPECS) $(DESTDIR)$(DESTDIR)$(CLIXON_DATADIR)
+	install -m 0644 $(YANGSPECS) $(DESTDIR)$(DESTDIR)$(YANG_INSTALLDIR)
 	install -d -m 0755 $(DESTDIR)$(localstatedir)/$(APPNAME)
 
 # Uncomment for installing config file in /usr/local/etc instead

example/main/Makefile.in

@@ -44,7 +44,7 @@ libdir          = @exec_prefix@/lib
 
 APPNAME	        = example
 # Here is where example yang appears
-CLIXON_DATADIR  = @CLIXON_DATADIR@
+YANG_INSTALLDIR  = @YANG_INSTALLDIR@
 # Install here if you want default clixon location:
 CLIXON_DEFAULT_CONFIG = @CLIXON_DEFAULT_CONFIG@
 
@@ -129,7 +129,7 @@ install: $(YANGSPECS) $(CLISPECS) $(PLUGINS) $(APPNAME).xml
 	install -m 0644 $(APPNAME).xml $(DESTDIR)$(sysconfdir)
 #	install -m 0644 $(APPNAME).xml $(DESTDIR)$(CLIXON_DEFAULT_CONFIG)
 	install -d -m 0755 $(DESTDIR)$(datarootdir)/$(APPNAME)/yang
-	install -m 0644 $(YANGSPECS) $(DESTDIR)$(DESTDIR)$(CLIXON_DATADIR)
+	install -m 0644 $(YANGSPECS) $(DESTDIR)$(DESTDIR)$(YANG_INSTALLDIR)
 	install -d -m 0755 $(DESTDIR)$(libdir)/$(APPNAME)/cli
 	install -m 0644 $(INSTALLFLAGS) $(CLI_PLUGIN) $(DESTDIR)$(libdir)/$(APPNAME)/cli
 	install -d -m 0755 $(DESTDIR)$(libdir)/$(APPNAME)/backend

include/clixon_custom.h

@@ -41,6 +41,6 @@
  */
 #undef RPC_USERNAME_ASSERT
 
-/* Use new xml_insert code on sorted xml lists
+/* Make namespace check on RESTCONF PUT and POST -d data
  */
-#define USE_XML_INSERT
+#define RESTCONF_NS_DATA_CHECK

lib/clixon/clixon_datastore.h

@@ -62,5 +62,7 @@ int xmldb_islocked(clicon_handle h, const char *db);
 int xmldb_exists(clicon_handle h, const char *db);
 int xmldb_delete(clicon_handle h, const char *db);
 int xmldb_create(clicon_handle h, const char *db);
+/* utility functions */
+int xmldb_db_reset(clicon_handle h, char *db);
 
 #endif /* _CLIXON_DATASTORE_H */

lib/clixon/clixon_netconf_lib.h

@@ -32,7 +32,8 @@
   ***** END LICENSE BLOCK *****
 
  * Netconf library functions. See RFC6241
- * 
+ * Functions to generate a netconf error message come in two forms: xml-tree and 
+ * cbuf. XML tree is preferred.
  */
 #ifndef _CLIXON_NETCONF_LIB_H
 #define _CLIXON_NETCONF_LIB_H
@@ -60,16 +61,18 @@ int netconf_lock_denied(cbuf *cb, char *info, char *message);
 int netconf_resource_denied(cbuf *cb, char *type, char *message);
 int netconf_rollback_failed(cbuf *cb, char *type, char *message);
 int netconf_data_exists(cbuf *cb, char *message);
-int netconf_data_missing(cbuf *cb, char *message);
+int netconf_data_missing(cbuf *cb, char *missing_choice, char *message);
+int netconf_data_missing_xml(cxobj **xret, char *missing_choice, char *message);
 int netconf_operation_not_supported(cbuf *cb, char *type, char *message);
 int netconf_operation_failed(cbuf *cb, char *type, char *message);
 int netconf_operation_failed_xml(cxobj **xret, char *type, char *message);
 int netconf_malformed_message(cbuf *cb, char *message);
 int netconf_malformed_message_xml(cxobj **xret, char *message);
-int netconf_data_not_unique(cbuf *cb, cxobj *x,	cvec *cvk);
+int netconf_data_not_unique_xml(cxobj **xret, cxobj *x,	cvec *cvk);
-int netconf_minmax_elements(cbuf *cb, cxobj *x, int max);
+int netconf_minmax_elements_xml(cxobj **xret, cxobj *x, int max);
 int netconf_trymerge(cxobj *x, yang_stmt *yspec, cxobj **xret);
 int netconf_module_load(clicon_handle h);
 char *netconf_db_find(cxobj *xn, char *name);
+int netconf_err2cb(cxobj *xerr, cbuf **cberr);
 
 #endif /* _CLIXON_NETCONF_LIB_H */

lib/clixon/clixon_xml_map.h

@@ -48,11 +48,11 @@ int xml2txt(FILE *f, cxobj *x, int level);
 int xml2cli(FILE *f, cxobj *x, char *prepend, enum genmodel_type gt);
 int xml_yang_root(cxobj *x, cxobj **xr);
 int xmlns_assign(cxobj *x);
-int xml_yang_validate_rpc(clicon_handle h, cxobj *xrpc, cbuf *cbret);
+int xml_yang_validate_rpc(clicon_handle h, cxobj *xrpc, cxobj **xret);
-int xml_yang_validate_list_key_only(clicon_handle h, cxobj *xt, cbuf *cbret);
+int xml_yang_validate_list_key_only(clicon_handle h, cxobj *xt, cxobj **xret);
-int xml_yang_validate_add(clicon_handle h, cxobj *xt, cbuf *cbret);
+int xml_yang_validate_add(clicon_handle h, cxobj *xt, cxobj **xret);
-int xml_yang_validate_all(clicon_handle h, cxobj *xt, cbuf *cbret);
+int xml_yang_validate_all(clicon_handle h, cxobj *xt, cxobj **xret);
-int xml_yang_validate_all_top(clicon_handle h, cxobj *xt, cbuf *cbret);
+int xml_yang_validate_all_top(clicon_handle h, cxobj *xt, cxobj **xret);
 int xml2cvec(cxobj *xt, yang_stmt *ys, cvec **cvv0);
 int cvec2xml_1(cvec *cvv, char *toptag, cxobj *xp, cxobj **xt0);
 

lib/src/clixon_datastore.c

@@ -78,6 +78,7 @@
 #include "clixon_datastore_write.h"
 #include "clixon_datastore_read.h"
 
+
 /*! Translate from symbolic database name to actual filename in file-system
  * @param[in]   th       text handle handle
  * @param[in]   db       Symbolic database name, eg "candidate", "running"
@@ -446,3 +447,21 @@ xmldb_create(clicon_handle h,
 	close(fd);
     return retval;
 }
+
+/*! Create an XML database. If it exists already, delete it before creating
+ * Utility function.
+ * @param[in]  h   Clixon handle
+ * @param[in]  db  Symbolic database name, eg "candidate", "running"
+ */
+int
+xmldb_db_reset(clicon_handle h, 
+	       char         *db)
+{
+    if (xmldb_exists(h, db) == 1){
+	if (xmldb_delete(h, db) != 0 && errno != ENOENT) 
+	    return -1;
+    }
+    if (xmldb_create(h, db) < 0)
+	return -1;
+    return 0;
+}

lib/src/clixon_datastore_read.c

@@ -641,7 +641,8 @@ xmldb_get_zerocopy(clicon_handle       h,
  * @param[in]  db     Name of database to search in (filename including dir path
  * @param[in]  xpath  String with XPATH syntax. or NULL for all
  * @param[out] xret   Single return XML tree. Free with xml_free()
-
+ * @retval     0      OK
+ * @retval     -1     Error
  * @code
  *   if (xmldb_get(xh, "running", "/interfaces/interface[name="eth"]", &xt) < 0)
  *      err;

lib/src/clixon_datastore_write.c

@@ -154,17 +154,10 @@ text_modify(clicon_handle       h,
 			goto fail;
 		    permit = 1;
 		}
-		//		int iamkey=0;
-
-#ifdef USE_XML_INSERT
 		/* Add new xml node but without parent - insert when node fully
 		   copied (see changed conditional below) */
 		if ((x0 = xml_new(x1name, NULL, (yang_stmt*)y0)) == NULL)
 		    goto done;
-#else
-		if ((x0 = xml_new(x1name, x0p, (yang_stmt*)y0)) == NULL)
-		    goto done;		
-#endif
 		changed++;
 
 		/* Copy xmlns attributes  */
@@ -210,16 +203,14 @@ text_modify(clicon_handle       h,
 		    }
 		}
 	    }
-#ifdef USE_XML_INSERT
 	    if (changed){
 		if (xml_insert(x0p, x0) < 0)
 		    goto done;
 	    }
-#endif
 	    break;
 	case OP_DELETE:
 	    if (x0==NULL){
-		if (netconf_data_missing(cbret, "Data does not exist; cannot delete resource") < 0)
+		if (netconf_data_missing(cbret, NULL, "Data does not exist; cannot delete resource") < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -295,17 +286,12 @@ text_modify(clicon_handle       h,
 			goto fail;
 		    permit = 1;
 		}
-#ifdef USE_XML_INSERT
 		/* Add new xml node but without parent - insert when node fully
 		 * copied (see changed conditional below) 
 		 * Note x0 may dangle cases if exit before changed conditional
 		 */
 		if ((x0 = xml_new(x1name, NULL, (yang_stmt*)y0)) == NULL)
 		    goto done;
-#else
-		if ((x0 = xml_new(x1name, x0p, (yang_stmt*)y0)) == NULL)
-		    goto done;
-#endif
 		changed++;
 		/* Copy xmlns attributes  */
 		x1a = NULL;
@@ -367,16 +353,14 @@ text_modify(clicon_handle       h,
 		if (ret == 0)
 		    goto fail;
 	    }
-#ifdef USE_XML_INSERT
 	    if (changed){
 		if (xml_insert(x0p, x0) < 0)
 		    goto done;
 	    }
-#endif
 	    break;
 	case OP_DELETE:
 	    if (x0==NULL){
-		if (netconf_data_missing(cbret, "Data does not exist; cannot delete resource") < 0)
+		if (netconf_data_missing(cbret, NULL, "Data does not exist; cannot delete resource") < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -396,17 +380,11 @@ text_modify(clicon_handle       h,
 	    break;
 	} /* CONTAINER switch op */
     } /* else Y_CONTAINER  */
-#ifndef USE_XML_INSERT
-    if (changed)
-	xml_sort(x0p, h);
-#endif
     retval = 1;
  done:
-#ifdef USE_XML_INSERT
     /* Remove dangling added objects */
     if (changed && x0 && xml_parent(x0)==NULL)
 	xml_purge(x0);
-#endif
     if (x0vec)
 	free(x0vec);
     return retval;
@@ -488,7 +466,7 @@ text_modify_top(clicon_handle       h,
 	 I.e., curl -u andy:bar -sS -X DELETE http://localhost/restconf/data
       */
 	    case OP_DELETE:
-		if (netconf_data_missing(cbret, "Data does not exist; cannot delete resource") < 0)
+		if (netconf_data_missing(cbret, NULL, "Data does not exist; cannot delete resource") < 0)
 		    goto done;
 		goto fail;
 		break;

lib/src/clixon_netconf_lib.c

@@ -32,6 +32,8 @@
   ***** END LICENSE BLOCK *****
 
  * Netconf library functions. See RFC6241
+ * Functions to generate a netconf error message come in two forms: xml-tree and 
+ * cbuf. XML tree is preferred.
  */
 
 #ifdef HAVE_CONFIG_H
@@ -63,6 +65,8 @@
 #include "clixon_options.h"
 #include "clixon_data.h"
 #include "clixon_xml_map.h"
+#include "clixon_xpath_ctx.h"
+#include "clixon_xpath.h"
 #include "clixon_netconf_lib.h"
 
 /*! Create Netconf in-use error XML tree according to RFC 6241 Appendix A
@@ -70,7 +74,7 @@
  * The request requires a resource that already is in use.
  * @param[out] cb       CLIgen buf. Error XML is written in this buffer
  * @param[in]  type     Error type: "application" or "protocol"
- * @param[in]  message  Error message
+ * @param[in]  message  Error message (will be XML encoded)
  */
 int
 netconf_in_use(cbuf *cb,
@@ -109,7 +113,7 @@ netconf_in_use(cbuf *cb,
  * The request specifies an unacceptable value for one or more parameters.
  * @param[out] cb      CLIgen buf. Error XML is written in this buffer
  * @param[in]  type    Error type: "application" or "protocol"
- * @param[in]  message Error message
+ * @param[in]  message Error message (will be XML encoded)
  */
 int
 netconf_invalid_value(cbuf *cb,
@@ -149,7 +153,7 @@ netconf_invalid_value(cbuf *cb,
  * too large for the implementation to handle.
  * @param[out] cb      CLIgen buf. Error XML is written in this buffer
  * @param[in]  type    Error type: "transport", "rpc", "application", "protocol"
- * @param[in]  message Error message
+ * @param[in]  message Error message (will be XML encoded)
  */
 int
 netconf_too_big(cbuf *cb,
@@ -189,7 +193,7 @@ netconf_too_big(cbuf *cb,
  * @param[out] cb      CLIgen buf. Error XML is written in this buffer
  * @param[in]  type    Error type: "rpc", "application" or "protocol"
  * @param[in]  info    bad-attribute or bad-element xml
- * @param[in]  message Error message
+ * @param[in]  message Error message (will be XML encoded)
  */
 int
 netconf_missing_attribute(cbuf *cb,
@@ -230,7 +234,7 @@ netconf_missing_attribute(cbuf *cb,
  * @param[out] cb      CLIgen buf. Error XML is written in this buffer
  * @param[in]  type    Error type: "rpc", "application" or "protocol"
  * @param[in]  info    bad-attribute or bad-element xml
- * @param[in]  message Error message
+ * @param[in]  message Error message (will be XML encoded)
  */
 int
 netconf_bad_attribute(cbuf *cb,
@@ -313,7 +317,7 @@ netconf_unknown_attribute(cbuf *cb,
  * @param[in]  type    Error type: "application" or "protocol"
  * @param[in]  tag     Error tag
  * @param[in]  element bad-element xml
- * @param[in]  message Error message
+ * @param[in]  message Error message (will be XML encoded)
  */
 static int
 netconf_common_xml(cxobj **xret,
@@ -323,8 +327,9 @@ netconf_common_xml(cxobj **xret,
 		   char   *element,
 		   char   *message)
 {
-    int   retval =-1;
+    int    retval =-1;
     cxobj *xerr;
+    char  *encstr = NULL;
     
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, NULL)) == NULL)
@@ -340,11 +345,17 @@ netconf_common_xml(cxobj **xret,
 		     "<error-severity>error</error-severity>",
 		     type, tag, infotag, element, infotag) < 0)
 	goto done;
-    if (message && xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+    if (message){
-			 message) < 0)
+	if (xml_chardata_encode(&encstr, "%s", message) < 0)
-	goto done;
+	    goto done;
+	if (xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+			 encstr) < 0)
+	    goto done;
+    }
     retval = 0;
  done:
+    if (encstr)
+	free(encstr);
     return retval;
 }    
 
@@ -551,7 +562,7 @@ netconf_access_denied(cbuf *cb,
  * authorization failed.
  * @param[out] xret    Error XML tree. Free with xml_free after use
  * @param[in]  type    Error type: "application" or "protocol"
- * @param[in]  message Error message
+ * @param[in]  message Error message  (will be XML encoded)
  * @code
  *  cxobj *xret = NULL;
  *  if (netconf_access_denied_xml(&xret, "protocol", "Unauthorized") < 0)
@@ -565,9 +576,10 @@ netconf_access_denied_xml(cxobj **xret,
 			  char   *type,
 			  char   *message)
 {
-    int   retval =-1;
+    int    retval =-1;
     cxobj *xerr;
-    
+    char  *encstr = NULL;
+
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, NULL)) == NULL)
 	    goto done;
@@ -580,11 +592,17 @@ netconf_access_denied_xml(cxobj **xret,
 		     "<error-tag>access-denied</error-tag>"
 		     "<error-severity>error</error-severity>", type) < 0)
 	goto done;
-    if (message && xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+    if (message){
-			 message) < 0)
+	if (xml_chardata_encode(&encstr, "%s", message) < 0)
-	goto done;
+	    goto done;
+	if (xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+			 encstr) < 0)
+	    goto done;
+    }
     retval = 0;
  done:
+    if (encstr)
+	free(encstr);
     return retval;
 }
 
@@ -752,38 +770,81 @@ netconf_data_exists(cbuf      *cb,
  * does not exist.  For example, a "delete" operation was attempted on
  * data that does not exist.
  * @param[out] cb      CLIgen buf. Error XML is written in this buffer
+ * @param[in]  missing_choice  If set, see RFC7950: 15.6 violates mandatiry choice
  * @param[in]  message Error message
  */
 int
 netconf_data_missing(cbuf *cb,
+		     char *missing_choice,
 		     char *message)
 {
     int   retval = -1;
-    char *encstr = NULL;
+    cxobj *xret = NULL;
 
-    if (cprintf(cb, "<rpc-reply><rpc-error>"
+    if (netconf_data_missing_xml(&xret, missing_choice, message) < 0)
-		"<error-type>application</error-type>"
+	goto done;
-		"<error-tag>data-missing</error-tag>"
+    if (clicon_xml2cbuf(cb, xret, 0, 0) < 0)
-		"<error-severity>error</error-severity>") <0)
+	goto done;
-	goto err;
+    retval = 0;
+ done:
+    if (xret)
+	xml_free(xret);
+    return retval;
+}
+
+/*! Create Netconf data-missing error XML tree according to RFC 6241 App A
+ *
+ * Request could not be completed because the relevant data model content 
+ * does not exist.  For example, a "delete" operation was attempted on
+ * data that does not exist.
+ * @param[out] xret    Error XML tree. Free with xml_free after use
+ * @param[in]  missing_choice  If set, see RFC7950: 15.6 violates mandatiry choice
+ * @param[in]  message Error message
+ */
+int
+netconf_data_missing_xml(cxobj **xret,
+			 char   *missing_choice,
+			 char   *message)
+{
+    int   retval = -1;
+    char *encstr = NULL;
+    cxobj *xerr;
+
+    if (*xret == NULL){
+	if ((*xret = xml_new("rpc-reply", NULL, NULL)) == NULL)
+	    goto done;
+    }
+    else if (xml_name_set(*xret, "rpc-reply") < 0)
+	goto done;
+    if ((xerr = xml_new("rpc-error", *xret, NULL)) == NULL)
+	goto done;
+    if (xml_parse_va(&xerr, NULL, 
+		     "<error-type>application</error-type>"
+		     "<error-tag>data-missing</error-tag>") < 0)
+	goto done;
+    if (missing_choice) /* NYI: RFC7950: 15.6 <error-path> */
+	if (xml_parse_va(&xerr, NULL, 
+			 "<error-app-tag>missing-choice</error-app-tag>"
+			 "<error-info><missing-choice>%s</missing-choice></error-info>",
+			 missing_choice) < 0)
+	    goto done;
+    if (xml_parse_va(&xerr, NULL, 
+		"<error-severity>error</error-severity>") < 0)
+	goto done;
     if (message){
 	if (xml_chardata_encode(&encstr, "%s", message) < 0)
 	    goto done;
-	if (cprintf(cb, "<error-message>%s</error-message>", encstr) < 0)
+	if (xml_parse_va(&xerr, NULL,
-	    goto err;
+			 "<error-message>%s</error-message>", encstr) < 0)
+	    goto done;
     }
-    if (cprintf(cb, "</rpc-error></rpc-reply>") <0)
-	goto err;
     retval = 0;
  done:
     if (encstr)
 	free(encstr);
     return retval;
- err:
-    clicon_err(OE_XML, errno, "cprintf");
-    goto done;
 }
-
+    
 /*! Create Netconf operation-not-supported error XML according to RFC 6241 App A
  *
  * Request could not be completed because the requested operation is not
@@ -858,7 +919,7 @@ netconf_operation_failed(cbuf  *cb,
  * some reason not covered by any other error condition.
  * @param[out] xret    Error XML tree 
  * @param[in]  type    Error type: "rpc", "application" or "protocol"
- * @param[in]  message Error message
+ * @param[in]  message Error message (will be XML encoded)
  * @code
  *  cxobj *xret = NULL;
  *  if (netconf_operation_failed_xml(&xret, "protocol", "Unauthorized") < 0)
@@ -874,6 +935,7 @@ netconf_operation_failed_xml(cxobj **xret,
 {
     int   retval =-1;
     cxobj *xerr;
+    char  *encstr = NULL;
     
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, NULL)) == NULL)
@@ -887,11 +949,17 @@ netconf_operation_failed_xml(cxobj **xret,
 		     "<error-tag>operation-failed</error-tag>"
 		     "<error-severity>error</error-severity>", type) < 0)
 	goto done;
-    if (message && xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+    if (message){
-			 message) < 0)
+	 if (xml_chardata_encode(&encstr, "%s", message) < 0)
+	     goto done;
+	 if (xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+			 encstr) < 0)
 	goto done;
+    }
     retval = 0;
  done:
+    if (encstr)
+	free(encstr);
     return retval;
 }
 
@@ -929,7 +997,7 @@ netconf_malformed_message(cbuf  *cb,
  * For example, the message is not well-formed XML or it uses an
  * invalid character set.
  * @param[out] xret    Error XML tree 
- * @param[in]  message Error message
+ * @param[in]  message Error message (will be XML encoded)
  * @note New in :base:1.1
  * @code
  *  cxobj *xret = NULL;
@@ -943,8 +1011,9 @@ int
 netconf_malformed_message_xml(cxobj **xret,
 			      char   *message)
 {
-    int   retval =-1;
+    int    retval =-1;
     cxobj *xerr;
+    char  *encstr = NULL;
     
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, NULL)) == NULL)
@@ -958,11 +1027,17 @@ netconf_malformed_message_xml(cxobj **xret,
 		     "<error-tag>malformed-message</error-tag>"
 		     "<error-severity>error</error-severity>") < 0)
 	goto done;
-    if (message && xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+     if (message){
-			 message) < 0)
+	 if (xml_chardata_encode(&encstr, "%s", message) < 0)
-	goto done;
+	     goto done;
+	 if (xml_parse_va(&xerr, NULL, "<error-message>%s</error-message>",
+			  encstr) < 0)
+	     goto done;
+     }
     retval = 0;
  done:
+    if (encstr)
+	free(encstr);
     return retval;
 }
 
@@ -970,79 +1045,97 @@ netconf_malformed_message_xml(cxobj **xret,
  *
  * A NETCONF operation would result in configuration data where a
  *   "unique" constraint is invalidated.
- * @param[out]  cb       CLIgen buf. Error XML is written in this buffer
+ * @param[out]  xret   Error XML tree. Free with xml_free after use
- * @param[in]   x        List element containing duplicate
+ * @param[in]   x      List element containing duplicate
- * @param[in]   cvk      List of comonents in x that are non-unique
+ * @param[in]   cvk    List of comonents in x that are non-unique
  * @see RFC7950 Sec 15.1
  */
 int
-netconf_data_not_unique(cbuf *cb,
+netconf_data_not_unique_xml(cxobj **xret,
-			cxobj *x,
+			    cxobj  *x,
-			cvec  *cvk)
+			    cvec   *cvk)
 {
     int     retval = -1;
     cg_var *cvi = NULL; 
     cxobj  *xi;
+    cxobj  *xerr;
+    cxobj  *xinfo;
+    cbuf   *cb = NULL;
     
-    if (cprintf(cb, "<rpc-reply><rpc-error>"
+    if (*xret == NULL){
-		"<error-type>protocol</error-type>"
+	if ((*xret = xml_new("rpc-reply", NULL, NULL)) == NULL)
-		"<error-tag>operation-failed</error-tag>"
+	    goto done;
-		"<error-app-tag>data-not-unique</error-app-tag>"
+    }
-		"<error-severity>error</error-severity>"
+    else if (xml_name_set(*xret, "rpc-reply") < 0)
-		"<error-info>") < 0)
+	goto done;
-	goto err;
+    if ((xerr = xml_new("rpc-error", *xret, NULL)) == NULL)
-    while ((cvi = cvec_each(cvk, cvi)) != NULL){
+	goto done;
-	if ((xi = xml_find(x, cv_string_get(cvi))) == NULL)
+    if (xml_parse_va(&xerr, NULL, "<error-type>protocol</error-type>"
-	    continue; /* ignore, shouldnt happen */
+		     "<error-tag>operation-failed</error-tag>"
-	cprintf(cb, "<non-unique>");
+     		     "<error-app-tag>data-not-unique</error-app-tag>"
-	clicon_xml2cbuf(cb, xi, 0, 0);	
+		     "<error-severity>error</error-severity>") < 0)
-	cprintf(cb, "</non-unique>");
+	goto done;
+    if (cvec_len(cvk)){
+	if ((xinfo = xml_new("error-info", xerr, NULL)) == NULL)
+	    goto done;
+	if ((cb = cbuf_new()) == NULL){
+	    clicon_err(OE_UNIX, errno, "cbuf_new");
+	    goto done;
+	}
+	while ((cvi = cvec_each(cvk, cvi)) != NULL){
+	    if ((xi = xml_find(x, cv_string_get(cvi))) == NULL)
+		continue; /* ignore, shouldnt happen */
+	    clicon_xml2cbuf(cb, xi, 0, 0);	
+	    if (xml_parse_va(&xinfo, NULL, "<non-unique>%s</non-unique>", cbuf_get(cb)) < 0)
+		goto done;
+	    cbuf_reset(cb);
+	}
     }
-    if (cprintf(cb, "</error-info></rpc-error></rpc-reply>") <0)
-	goto err;
     retval = 0;
  done:
+    if (cb)
+	cbuf_free(cb);
     return retval;
- err:
-    clicon_err(OE_XML, errno, "cprintf");
-    goto done;
 }
 
 /*! Create Netconf too-many/few-elements err msg according to RFC 7950 15.2/15.3
  *
  * A NETCONF operation would result in configuration data where a
    list or a leaf-list would have too many entries, the following error
- * @param[out]  cb       CLIgen buf. Error XML is written in this buffer
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @param[in]   x        List element containing duplicate
  * @param[in]   max      If set, return too-many, otherwise too-few
  * @see RFC7950 Sec 15.1
  */
 int
-netconf_minmax_elements(cbuf *cb,
+netconf_minmax_elements_xml(cxobj **xret,
-			cxobj *x,
+	                    cxobj  *x,
-			int    max)
+	                    int     max)
 {
-    int     retval = -1;
+    int    retval = -1;
+    cxobj *xerr;
     
-    if (cprintf(cb, "<rpc-reply><rpc-error>"
+    if (*xret == NULL){
-		"<error-type>protocol</error-type>"
+	if ((*xret = xml_new("rpc-reply", NULL, NULL)) == NULL)
-		"<error-tag>operation-failed</error-tag>"
+	    goto done;
-		"<error-app-tag>too-%s-elements</error-app-tag>"
+    }
-		"<error-severity>error</error-severity>"
+    else if (xml_name_set(*xret, "rpc-reply") < 0)
-		"<error-path>%s</error-path>"
+	goto done;
-		"</rpc-error></rpc-reply>",
+    if ((xerr = xml_new("rpc-error", *xret, NULL)) == NULL)
-		max?"many":"few",
+	goto done;
-		xml_name(x)) < 0) /* XXX should be xml2xpath */
+    if (xml_parse_va(&xerr, NULL, "<error-type>protocol</error-type>"
-	goto err;
+		     "<error-tag>operation-failed</error-tag>"
+		     "<error-app-tag>too-%s-elements</error-app-tag>"
+		     "<error-severity>error</error-severity>"
+		     "<error-path>%s</error-path>",
+		     max?"many":"few",
+		     xml_name(x)) < 0) /* XXX should be xml2xpath */
+	goto done;
     retval = 0;
  done:
     return retval;
- err:
-    clicon_err(OE_XML, errno, "cprintf");
-    goto done;
 }
 
-
 /*! Help function: merge - check yang - if error make netconf errmsg 
  * @param[in]     x       XML tree
  * @param[in]     yspec   Yang spec
@@ -1156,3 +1249,41 @@ netconf_db_find(cxobj *xn,
     return db;
 }
 
+/*! Generate netconf error msg to cbuf to use in string printout or logs
+ * @param[in]  xerr    Netconf error message on the level: <rpc-reply><rpc-error>
+ * @param[out] cberr   Translation from netconf err to cbuf. Free with cbuf_free.
+ * @retval     0       OK, with cberr set
+ * @retval    -1       Error
+ * @code
+ *     cbuf  *cb = NULL;
+ *     if (netconf_err2cb(xerr, &cb) < 0)
+ *        err;
+ *     printf("%s", cbuf_get(cb));
+ * @endcode
+ * @see clicon_rpc_generate_error
+ */
+int
+netconf_err2cb(cxobj *xerr,
+	       cbuf **cberr)
+{
+    int    retval = -1;
+    cbuf  *cb = NULL;
+    cxobj *x;
+
+    if ((cb = cbuf_new()) ==NULL){
+	clicon_err(OE_XML, errno, "cbuf_new");
+	goto done;
+    }
+    if ((x=xpath_first(xerr, "error-type"))!=NULL)
+	cprintf(cb, "%s ", xml_body(x));
+    if ((x=xpath_first(xerr, "error-tag"))!=NULL)
+	cprintf(cb, "%s ", xml_body(x));
+    if ((x=xpath_first(xerr, "error-message"))!=NULL)
+	cprintf(cb, "%s ", xml_body(x));
+    if ((x=xpath_first(xerr, "error-info"))!=NULL)
+	clicon_xml2cbuf(cb, xml_child_i(x,0), 0, 0);
+    *cberr = cb;
+    retval = 0;
+ done:
+    return retval;
+}

lib/src/clixon_options.c

@@ -72,6 +72,7 @@
 #include "clixon_data.h"
 #include "clixon_xpath_ctx.h"
 #include "clixon_xpath.h"
+#include "clixon_netconf_lib.h"
 #include "clixon_xml_map.h"
 
 /* Mapping between Clicon startup modes string <--> constants, 
@@ -89,6 +90,7 @@ static const map_str2int startup_mode_map[] = {
  * @param[in] dbglevel Debug level
  * @retval    0        OK
  * @retval   -1        Error
+ * @note CLICON_FEATURE and CLICON_YANG_DIR are treated specially since they are lists
  */
 int
 clicon_option_dump(clicon_handle h, 
@@ -101,6 +103,7 @@ clicon_option_dump(clicon_handle h,
     void          *val;
     size_t         klen;
     size_t         vlen;
+    cxobj         *x = NULL;
     
     if (clicon_hash_keys(hash, &keys, &klen) < 0)
 	goto done;
@@ -115,7 +118,22 @@ clicon_option_dump(clicon_handle h,
 	else
 	    clicon_debug(dbglevel, "%s = NULL", keys[i]);
     }
-    retval = 0;
+    /* Next print CLICON_FEATURE and CLICON_YANG_DIR from config tree
+     * Since they are lists they are placed in the config tree.
+     */
+    x = NULL;
+    while ((x = xml_child_each(clicon_conf_xml(h), x, CX_ELMNT)) != NULL) {
+	if (strcmp(xml_name(x), "CLICON_YANG_DIR") != 0)
+	    continue;
+	clicon_debug(dbglevel, "%s =\t \"%s\"", xml_name(x), xml_body(x));
+    }
+    x = NULL;
+    while ((x = xml_child_each(clicon_conf_xml(h), x, CX_ELMNT)) != NULL) {
+	if (strcmp(xml_name(x), "CLICON_FEATURE") != 0)
+	    continue;
+	clicon_debug(dbglevel, "%s =\t \"%s\"", xml_name(x), xml_body(x));
+    }
+   retval = 0;
  done:
     if (keys)
 	free(keys);
@@ -145,6 +163,7 @@ parse_configfile(clicon_handle  h,
     char       *body;
     clicon_hash_t *copt = clicon_options(h);
     cbuf       *cbret = NULL;
+    cxobj      *xret = NULL;
     int         ret;
 
     if (filename == NULL || !strlen(filename)){
@@ -194,13 +213,11 @@ parse_configfile(clicon_handle  h,
     }
     if (xml_apply0(xc, CX_ELMNT, xml_default, h) < 0)
 	goto done;	
-    if ((cbret = cbuf_new()) == NULL){
+    if ((ret = xml_yang_validate_add(h, xc, &xret)) < 0)
-	clicon_err(OE_XML, errno, "cbuf_new");
-	goto done;	
-    }
-    if ((ret = xml_yang_validate_add(h, xc, cbret)) < 0)
 	goto done;
     if (ret == 0){
+	if (netconf_err2cb(xret, &cbret) < 0)
+	    goto done;
 	clicon_err(OE_CFG, 0, "Config file validation: %s", cbuf_get(cbret));
 	goto done;
     }
@@ -234,6 +251,8 @@ parse_configfile(clicon_handle  h,
   done:
     if (cbret)
 	cbuf_free(cbret);
+    if (xret)
+	xml_free(xret);
     if (xt)
 	xml_free(xt);
     if (f)
@@ -243,6 +262,7 @@ parse_configfile(clicon_handle  h,
 
 /*! Add configuration option overriding file setting
  * Add to clicon_options hash, and to clicon_conf_xml tree
+ * Assumes clicon_conf_xml_set has been called
  * @param[in]  h      Clicon handle
  * @param[in]  name   Name of configuration option (see clixon-config.yang)
  * @param[in]  value  String value

lib/src/clixon_proto_client.c

@@ -71,6 +71,7 @@
 #include "clixon_proto.h"
 #include "clixon_err.h"
 #include "clixon_err_string.h"
+#include "clixon_netconf_lib.h"
 #include "clixon_proto_client.h"
 
 /*! Send internal netconf rpc from client to backend
@@ -224,29 +225,22 @@ clicon_rpc_netconf_xml(clicon_handle  h,
 }
 
 /*! Generate and log clicon error function call from Netconf error message
+ * @param[in]  prefix  Print this string (if given) before: "<prefix>: <error>"
  * @param[in]  xerr    Netconf error message on the level: <rpc-reply><rpc-error>
  */
 int
-clicon_rpc_generate_error(char  *format, 
+clicon_rpc_generate_error(char  *prefix, 
 			  cxobj *xerr)
 {
     int    retval = -1;
     cbuf  *cb = NULL;
-    cxobj *x;
 
-    if ((cb = cbuf_new()) ==NULL){
+    if (netconf_err2cb(xerr, &cb) < 0)
-	clicon_err(OE_XML, errno, "cbuf_new");
 	goto done;
-    }
+    if (prefix)
-    if ((x=xpath_first(xerr, "error-type"))!=NULL)
+	clicon_log(LOG_ERR, "%s: %s", prefix, cbuf_get(cb));
-	cprintf(cb, "%s ", xml_body(x));
+    else
-    if ((x=xpath_first(xerr, "error-tag"))!=NULL)
+	clicon_log(LOG_ERR, "%s", cbuf_get(cb));
-	cprintf(cb, "%s ", xml_body(x));
-    if ((x=xpath_first(xerr, "error-message"))!=NULL)
-	cprintf(cb, "%s ", xml_body(x));
-    if ((x=xpath_first(xerr, "error-info"))!=NULL)
-	clicon_xml2cbuf(cb, xml_child_i(x,0), 0, 0);
-    clicon_log(LOG_ERR, "%s: %s", format, cbuf_get(cb));
     retval = 0;
  done:
     if (cb)

lib/src/clixon_xml_changelog.c

@@ -67,6 +67,7 @@
 #include "clixon_options.h"
 #include "clixon_data.h"
 #include "clixon_yang_module.h"
+#include "clixon_netconf_lib.h"
 #include "clixon_xml_map.h"
 #include "clixon_xml_changelog.h"
 #include "clixon_xpath_ctx.h"
@@ -424,8 +425,9 @@ clixon_xml_changelog_init(clicon_handle h)
     int        fd = -1;
     cxobj     *xt = NULL;
     yang_stmt *yspec;
-    cbuf      *cbret = NULL;
     int        ret;
+    cxobj     *xret = NULL;
+    cbuf      *cbret = NULL;
 
     yspec = clicon_dbspec_yang(h);
     if ((filename = clicon_option_str(h, "CLICON_XML_CHANGELOG_FILE")) != NULL){
@@ -437,15 +439,13 @@ clixon_xml_changelog_init(clicon_handle h)
 	    goto done;
 	if (xml_rootchild(xt, 0, &xt) < 0)
 	    goto done;
-	if ((cbret = cbuf_new()) == NULL){
+	if ((ret = xml_yang_validate_all(h, xt, &xret)) < 0)
-	    clicon_err(OE_UNIX, errno, "cbuf_new");
 	    goto done;
-	}
+	if (ret==1 && (ret = xml_yang_validate_add(h, xt, &xret)) < 0)
-	if ((ret = xml_yang_validate_all(h, xt, cbret)) < 0)
-	    goto done;
-	if (ret==1 && (ret = xml_yang_validate_add(h, xt, cbret)) < 0)
 	    goto done;
 	if (ret == 0){ /* validation failed */
+	    if (netconf_err2cb(xret, &cbret) < 0)
+		goto done;
 	    clicon_err(OE_YANG, 0, "validation failed: %s", cbuf_get(cbret));
 	    goto done;
 	}   
@@ -455,12 +455,14 @@ clixon_xml_changelog_init(clicon_handle h)
     }
     retval = 0;
  done:
+    if (cbret)
+	cbuf_free(cbret);
+    if (xret)
+	xml_free(xret);
     if (fd != -1)
 	close(fd);
     if (xt)
 	xml_free(xt);
-    if (cbret)
-	cbuf_free(cbret);
     return retval;
 }
 

lib/src/clixon_xml_map.c

@@ -248,7 +248,7 @@ xml2cli(FILE              *f,
 /*! Validate xml node of type leafref, ensure the value is one of that path's reference
  * @param[in]  xt    XML leaf node of type leafref
  * @param[in]  ytype Yang type statement belonging to the XML node
- * @param[out] cbret Error buffer
+ * @param[out] xret  Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed
  * @retval    -1     Error
@@ -256,7 +256,7 @@ xml2cli(FILE              *f,
 static int
 validate_leafref(cxobj     *xt,
 		 yang_stmt *ytype,
-		 cbuf      *cbret)
+		 cxobj    **xret)
 {
     int          retval = -1;
     yang_stmt   *ypath;
@@ -270,7 +270,7 @@ validate_leafref(cxobj     *xt,
     if ((leafrefbody = xml_body(xt)) == NULL)
 	goto ok;
     if ((ypath = yang_find(ytype, Y_PATH, NULL)) == NULL){
-	if (netconf_missing_element(cbret, "application", yang_argument_get(ytype), "Leafref requires path statement") < 0)
+	if (netconf_missing_element_xml(xret, "application", yang_argument_get(ytype), "Leafref requires path statement") < 0)
 	    goto done;
 	goto fail;
     }
@@ -284,12 +284,12 @@ validate_leafref(cxobj     *xt,
 	    break;
     }
     if (i==xlen){
-	if (netconf_bad_element(cbret, "application", leafrefbody, "Leafref validation failed: No such leaf") < 0)
+	if (netconf_bad_element_xml(xret, "application", leafrefbody, "Leafref validation failed: No such leaf") < 0)
 	    goto done;
 	goto fail;
     }
  ok:
-    retval = 0;
+    retval = 1;
  done:
     if (xvec)
 	free(xvec);
@@ -312,7 +312,7 @@ validate_leafref(cxobj     *xt,
  * @param[in]  xt    XML leaf node of type identityref
  * @param[in]  ys    Yang spec of leaf
  * @param[in]  ytype Yang type field of type identityref
- * @param[out] cbret Error buffer
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed
  * @retval    -1     Error
@@ -324,7 +324,8 @@ static int
 validate_identityref(cxobj     *xt,
 		     yang_stmt *ys,
 		     yang_stmt *ytype,
-		     cbuf      *cbret)
+		     cxobj    **xret)
+
 {
     int         retval = -1;
     char       *node;
@@ -350,13 +351,13 @@ validate_identityref(cxobj     *xt,
     }
     /* This is the type's base reference */
     if ((ybaseref = yang_find(ytype, Y_BASE, NULL)) == NULL){
-	if (netconf_missing_element(cbret, "application", yang_argument_get(ytype), "Identityref validation failed, no base") < 0)
+	if (netconf_missing_element_xml(xret, "application", yang_argument_get(ytype), "Identityref validation failed, no base") < 0)
 	    goto done;
 	goto fail;
     }
     /* This is the actual base identity */
     if ((ybaseid = yang_find_identity(ybaseref, yang_argument_get(ybaseref))) == NULL){
-	if (netconf_missing_element(cbret, "application", yang_argument_get(ybaseref), "Identityref validation failed, no base identity") < 0)
+	if (netconf_missing_element_xml(xret, "application", yang_argument_get(ybaseref), "Identityref validation failed, no base identity") < 0)
 	    goto done;
 	goto fail;
     }
@@ -367,7 +368,7 @@ validate_identityref(cxobj     *xt,
 	cbuf_reset(cb);
 	cprintf(cb, "Identityref validation failed, %s not derived from %s", 
 		node, yang_argument_get(ybaseid));
-	if (netconf_operation_failed(cbret, "application", cbuf_get(cb)) < 0)
+	if (netconf_operation_failed_xml(xret, "application", cbuf_get(cb)) < 0)
 	    goto done;
 	goto fail;
     }
@@ -413,10 +414,12 @@ xml_yang_root(cxobj  *x,
 }
 
 /*! Validate an RPC node
- * @param[in]  xt  XML node to be validated
+ * @param[in]  h     Clicon handle
- * @retval     1   Validation OK
+ * @param[in]  xrpc  XML node to be validated
- * @retval     0   Validation failed
+ * @param[out] xret  Error XML tree. Free with xml_free after use
- * @retval    -1   Error
+ * @retval     1     Validation OK
+ * @retval     0     Validation failed
+ * @retval    -1     Error
  * rfc7950
  * 7.14.2
  * If a leaf in the input tree has a "mandatory" statement with the
@@ -454,8 +457,8 @@ xml_yang_root(cxobj  *x,
  */
 int
 xml_yang_validate_rpc(clicon_handle h,
-		      cxobj *xrpc,
+		      cxobj        *xrpc,
-		      cbuf  *cbret)
+		      cxobj       **xret)
 {
     int        retval = -1;
     yang_stmt *yn=NULL;  /* rpc name */
@@ -469,15 +472,15 @@ xml_yang_validate_rpc(clicon_handle h,
     /* xn is name of rpc, ie <rcp><xn/></rpc> */
     while ((xn = xml_child_each(xrpc, xn, CX_ELMNT)) != NULL) {
 	if ((yn = xml_spec(xn)) == NULL){
-	    if (netconf_unknown_element(cbret, "application", xml_name(xn), NULL) < 0)
+	    if (netconf_unknown_element_xml(xret, "application", xml_name(xn), NULL) < 0)
 		goto done;
 	    goto fail;
 	}
-	if ((retval = xml_yang_validate_all(h, xn, cbret)) < 1) 
+	if ((retval = xml_yang_validate_all(h, xn, xret)) < 1) 
 	    goto done; /* error or validation fail */
-	if ((retval = xml_yang_validate_add(h, xn, cbret)) < 1)
+	if ((retval = xml_yang_validate_add(h, xn, xret)) < 1)
 	    goto done; /* error or validation fail */
-	if (xml_apply0(xn, CX_ELMNT, xml_default, NULL) < 0)
+	if (xml_apply0(xn, CX_ELMNT, xml_default, h) < 0)
 	    goto done;
     }
     // ok: /* pass validation */
@@ -492,7 +495,7 @@ xml_yang_validate_rpc(clicon_handle h,
 /*! Check if an xml node is a part of a choice and have >1 siblings 
  * @param[in]  xt    XML node to be validated
  * @param[in]  yt    xt:s yang statement
- * @param[out] cbret Error buffer (set w netconf error if retval == 0)
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed (cbret set)
  * @retval    -1     Error
@@ -501,7 +504,7 @@ xml_yang_validate_rpc(clicon_handle h,
 static int
 check_choice(cxobj     *xt, 
 	     yang_stmt *yt,
-	     cbuf      *cbret)
+	     cxobj    **xret)
 {
     int        retval = -1;
     yang_stmt *y;
@@ -552,7 +555,7 @@ check_choice(cxobj     *xt,
 	    continue; /* not choice */
 	    break;
 	}
-	if (netconf_bad_element(cbret, "application", xml_name(x), "Element in choice statement already exists") < 0)
+	if (netconf_bad_element_xml(xret, "application", xml_name(x), "Element in choice statement already exists") < 0)
 	    goto done;
 	goto fail;
     } /* while */
@@ -569,7 +572,7 @@ check_choice(cxobj     *xt,
 /*! Check if an xml node lacks mandatory children
  * @param[in]  xt    XML node to be validated
  * @param[in]  yt    xt:s yang statement
- * @param[out] cbret Error buffer (set w netconf error if retval == 0)
+ * @param[out] xret  Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed (cbret set)
  * @retval    -1     Error
@@ -577,7 +580,8 @@ check_choice(cxobj     *xt,
 static int
 check_mandatory(cxobj     *xt, 
 		yang_stmt *yt,
-		cbuf      *cbret)
+		cxobj    **xret)
+
 {
     int        retval = -1;
     int        i;
@@ -600,7 +604,7 @@ check_mandatory(cxobj     *xt,
 	    while ((cvi = cvec_each(cvk, cvi)) != NULL) {
 		keyname = cv_string_get(cvi);	    
 		if (xml_find_type(xt, NULL, keyname, CX_ELMNT) == NULL){
-		    if (netconf_missing_element(cbret, "application", keyname, "Mandatory key") < 0)
+		    if (netconf_missing_element_xml(xret, "application", keyname, "Mandatory key") < 0)
 			goto done;
 		    goto fail;
 		}
@@ -623,7 +627,7 @@ check_mandatory(cxobj     *xt,
 		    break; /* got it */
 	    }
 	    if (x == NULL){
-		if (netconf_missing_element(cbret, "application", yc->ys_argument, "Mandatory variable") < 0)
+		if (netconf_missing_element_xml(xret, "application", yc->ys_argument, "Mandatory variable") < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -640,17 +644,7 @@ check_mandatory(cxobj     *xt,
 	    if (x == NULL){
 		/* @see RFC7950: 15.6 Error Message for Data That Violates 
 		 * a Mandatory "choice" Statement */
-		if (cprintf(cbret, "<rpc-reply><rpc-error>"
+		if (netconf_data_missing_xml(xret, yc->ys_argument, NULL) < 0)
-			    "<error-type>application</error-type>"
-			    "<error-tag>data-missing</error-tag>"
-			    "<error-app-tag>missing-choice</error-app-tag>"
-#ifdef NYI
-			    //  "<error-path></error-path>"
-#endif
-			    "<error-info><missing-choice>%s</missing-choice></error-info>"
-			    "<error-severity>error</error-severity>"
-			    "</rpc-error></rpc-reply>",
-			    yc->ys_argument) <0)
 		    goto done;
 		goto fail;
 	    }
@@ -667,10 +661,14 @@ check_mandatory(cxobj     *xt,
     goto done;
 }
 
+/*!
+ * @param[out] xret    Error XML tree. Free with xml_free after use
+ */
 static int
 check_list_key(cxobj     *xt, 
 	       yang_stmt *yt,
-	       cbuf      *cbret)
+	       cxobj    **xret)
+
 {
     int        retval = -1;
     int        i;
@@ -690,7 +688,7 @@ check_list_key(cxobj     *xt,
 	    while ((cvi = cvec_each(cvk, cvi)) != NULL) {
 		keyname = cv_string_get(cvi);	    
 		if (xml_find_type(xt, NULL, keyname, CX_ELMNT) == NULL){
-		    if (netconf_missing_element(cbret, "application", keyname, "Mandatory key") < 0)
+		    if (netconf_missing_element_xml(xret, "application", keyname, "Mandatory key") < 0)
 			goto done;
 		    goto fail;
 		}
@@ -739,7 +737,7 @@ check_insert_duplicate(char **vec,
  * @param[in]  xt    The parent of x
  * @param[in]  y     Its yang spec (Y_LIST)
  * @param[in]  yu    A yang unique spec (Y_UNIQUE)
- * @param[out] cbret Error buffer (set w netconf error if retval == 0)
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed (cbret set)
  * @retval    -1     Error
@@ -750,7 +748,8 @@ check_unique_list(cxobj     *x,
 		  cxobj     *xt, 
 		  yang_stmt *y,
 		  yang_stmt *yu,
-		  cbuf      *cbret)
+		  cxobj    **xret)
+
 {
     int       retval = -1;
     cvec      *cvk; /* unique vector */
@@ -784,7 +783,7 @@ check_unique_list(cxobj     *x,
 	if (cvi==NULL){
 	    /* Last element (i) is newly inserted, see if it is already there */
 	    if (check_insert_duplicate(vec, i, vlen) < 0){
-		if (netconf_data_not_unique(cbret, x, cvk) < 0)
+		if (netconf_data_not_unique_xml(xret, x, cvk) < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -807,7 +806,7 @@ check_unique_list(cxobj     *x,
  * @param[in]  x  One x (the last) of a specific lis
  * @param[in]  y  Yang spec of x
  * @param[in]  nr Number of elements (like x) in thlist
- * @param[out] cbret Error buffer (set w netconf error if retval == 0)
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed (cbret set)
  * @retval    -1     Error
@@ -817,7 +816,7 @@ static int
 check_min_max(cxobj     *x,
 	      yang_stmt *y,
 	      int        nr,
-	      cbuf      *cbret)
+	      cxobj     **xret)
 {
     int         retval = -1;
     yang_stmt  *ymin; /* yang min */
@@ -827,7 +826,7 @@ check_min_max(cxobj     *x,
     if ((ymin = yang_find(y, Y_MIN_ELEMENTS, NULL)) != NULL){
 	cv = yang_cv_get(ymin);
 	if (nr < cv_uint32_get(cv)){
-	    if (netconf_minmax_elements(cbret, x, 0) < 0)
+	    if (netconf_minmax_elements_xml(xret, x, 0) < 0)
 		goto done;
 	    goto fail;
 	}
@@ -836,7 +835,7 @@ check_min_max(cxobj     *x,
 	cv = yang_cv_get(ymax);
 	if (cv_uint32_get(cv) > 0 && /* 0 means unbounded */
 	    nr > cv_uint32_get(cv)){
-	    if (netconf_minmax_elements(cbret, x, 1) < 0)
+	    if (netconf_minmax_elements_xml(xret, x, 1) < 0)
 		goto done;
 	    goto fail;
 	}
@@ -851,9 +850,9 @@ check_min_max(cxobj     *x,
 
 /*! Detect unique constraint for duplicates from parent node and minmax
  * @param[in]  xt    XML parent (may have lists w unique constraints as child)
- * @param[out] cbret Error buffer (set w netconf error if retval == 0)
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
- * @retval     0     Validation failed (cbret set)
+ * @retval     0     Validation failed (xret set)
  * @retval    -1     Error
  * Assume xt:s children are sorted and yang populated.
  * The function does two different things of the children of an XML node:
@@ -889,8 +888,8 @@ check_min_max(cxobj     *x,
  * are not allowed.
  */
 static int
-check_list_unique_minmax(cxobj *xt,
+check_list_unique_minmax(cxobj  *xt,
-			 cbuf  *cbret)
+			 cxobj **xret)
 {
     int         retval = -1;
     cxobj      *x = NULL;
@@ -932,7 +931,7 @@ check_list_unique_minmax(cxobj *xt,
 	    }
 	    else {
 		/* Check if the list length violates min/max */
-		if ((ret = check_min_max(xp, yp, nr, cbret)) < 0)
+		if ((ret = check_min_max(xp, yp, nr, xret)) < 0)
 		    goto done;
 		if (ret == 0)
 		    goto fail;
@@ -953,7 +952,7 @@ check_list_unique_minmax(cxobj *xt,
 		do {
 		    if (yang_keyword_get(ye) == Y_LIST || yang_keyword_get(ye) == Y_LEAF_LIST){
 			/* Check if the list length violates min/max */
-			if ((ret = check_min_max(xt, ye, 0, cbret)) < 0)
+			if ((ret = check_min_max(xt, ye, 0, xret)) < 0)
 			    goto done;
 			if (ret == 0)
 			    goto fail;
@@ -973,7 +972,7 @@ check_list_unique_minmax(cxobj *xt,
 	     * its first element x, its yang spec y, its parent xt, and 
 	     * a unique yang spec yu,
 	     */
-	    if ((ret = check_unique_list(x, xt, y, yu, cbret)) < 0)
+	    if ((ret = check_unique_list(x, xt, y, yu, xret)) < 0)
 		goto done;
 	    if (ret == 0)
 		goto fail;
@@ -984,7 +983,7 @@ check_list_unique_minmax(cxobj *xt,
      */
     if (yp){  
 	/* Check if the list length violates min/max */
-	if ((ret = check_min_max(xp, yp, nr, cbret)) < 0)
+	if ((ret = check_min_max(xp, yp, nr, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -996,7 +995,7 @@ check_list_unique_minmax(cxobj *xt,
 	do {
 	    if (yang_keyword_get(ye) == Y_LIST || yang_keyword_get(ye) == Y_LEAF_LIST){
 		/* Check if the list length violates min/max */
-		if ((ret = check_min_max(xt, ye, 0, cbret)) < 0)
+		if ((ret = check_min_max(xt, ye, 0, xret)) < 0)
 		    goto done;
 		if (ret == 0)
 		    goto fail;
@@ -1014,14 +1013,14 @@ check_list_unique_minmax(cxobj *xt,
  * 1. Check if mandatory leafs present as subs.
  * 2. Check leaf values, eg int ranges and string regexps.
  * @param[in]  xt    XML node to be validated
- * @param[out] cbret Error buffer (set w netconf error if retval == 0)
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed (cbret set)
  * @retval    -1     Error
  * @code
  *   cxobj *x;
- *   cbuf *cbret = cbuf_new();
+ *   cbuf *xret = NULL;
- *   if ((ret = xml_yang_validate_add(h, x, cbret)) < 0)
+ *   if ((ret = xml_yang_validate_add(h, x, &xret)) < 0)
  *      err;
  *   if (ret == 0)
  *      fail;
@@ -1032,8 +1031,8 @@ check_list_unique_minmax(cxobj *xt,
  */
 int
 xml_yang_validate_add(clicon_handle h,
-		      cxobj   *xt, 
+		      cxobj        *xt, 
-		      cbuf    *cbret)
+		      cxobj       **xret)
 {
     int        retval = -1;
     cg_var    *cv = NULL;
@@ -1047,11 +1046,11 @@ xml_yang_validate_add(clicon_handle h,
     /* if not given by argument (overide) use default link 
        and !Node has a config sub-statement and it is false */
     if ((yt = xml_spec(xt)) != NULL && yang_config(yt) != 0){
-	if ((ret = check_choice(xt, yt, cbret)) < 0)
+	if ((ret = check_choice(xt, yt, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
-	if ((ret = check_mandatory(xt, yt, cbret)) < 0)
+	if ((ret = check_mandatory(xt, yt, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -1073,21 +1072,21 @@ xml_yang_validate_add(clicon_handle h,
 		 * are considered as "" */
 		cvtype = cv_type_get(cv);
 		if (cv_isint(cvtype) || cvtype == CGV_BOOL || cvtype == CGV_DEC64){
-		    if (netconf_bad_element(cbret, "application",  yt->ys_argument, "Invalid NULL value") < 0)
+		    if (netconf_bad_element_xml(xret, "application",  yt->ys_argument, "Invalid NULL value") < 0)
 			goto done;
 		    goto fail;
 		}
 	    }
 	    else{
 		if (cv_parse1(body, cv, &reason) != 1){
-		    if (netconf_bad_element(cbret, "application",  yt->ys_argument, reason) < 0)
+		    if (netconf_bad_element_xml(xret, "application",  yt->ys_argument, reason) < 0)
 			goto done;
 		    goto fail;
 		}
 	    }
 
 	    if ((ys_cv_validate(h, cv, yt, &reason)) != 1){
-		if (netconf_bad_element(cbret, "application",  yt->ys_argument, reason) < 0)
+		if (netconf_bad_element_xml(xret, "application",  yt->ys_argument, reason) < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -1098,7 +1097,7 @@ xml_yang_validate_add(clicon_handle h,
     }
     x = NULL;
     while ((x = xml_child_each(xt, x, CX_ELMNT)) != NULL) {
-	if ((ret = xml_yang_validate_add(h, x, cbret)) < 0)
+	if ((ret = xml_yang_validate_add(h, x, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -1116,11 +1115,12 @@ xml_yang_validate_add(clicon_handle h,
 }
 
 /*! Some checks done only at edit_config, eg keys in lists
+ * @param[out] xret    Error XML tree. Free with xml_free after use
  */
 int
 xml_yang_validate_list_key_only(clicon_handle h,
-				cxobj   *xt, 
+				cxobj        *xt, 
-				cbuf    *cbret)
+				cxobj       **xret)
 {
     int        retval = -1;
     yang_stmt *yt;   /* yang spec of xt going in */
@@ -1130,14 +1130,14 @@ xml_yang_validate_list_key_only(clicon_handle h,
     /* if not given by argument (overide) use default link 
        and !Node has a config sub-statement and it is false */
     if ((yt = xml_spec(xt)) != NULL && yang_config(yt) != 0){
-	if ((ret = check_list_key(xt, yt, cbret)) < 0)
+	if ((ret = check_list_key(xt, yt, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
     }
     x = NULL;
     while ((x = xml_child_each(xt, x, CX_ELMNT)) != NULL) {
-	if ((ret = xml_yang_validate_list_key_only(h, x, cbret)) < 0)
+	if ((ret = xml_yang_validate_list_key_only(h, x, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -1154,17 +1154,18 @@ xml_yang_validate_list_key_only(clicon_handle h,
 /*! Validate a single XML node with yang specification for all (not only added) entries
  * 1. Check leafrefs. Eg you delete a leaf and a leafref references it.
  * @param[in]  xt  XML node to be validated
- * @param[out] cbret Error buffer (set w netconf error if retval == 0)
+ * @param[out] xret  Error XML tree (if retval=0). Free with xml_free after use
  * @retval     1     Validation OK
  * @retval     0     Validation failed (cbret set)
  * @retval    -1     Error
  * @code
  *   cxobj *x;
- *   cbuf *cbret = cbuf_new();
+ *   cbuf *xret = NULL;
- *   if ((ret = xml_yang_validate_all(x, cbret)) < 0)
+ *   if ((ret = xml_yang_validate_all(h, x, &xret)) < 0)
  *      err;
  *   if (ret == 0)
  *      fail;
+ *   xml_free(xret);
  * @endcode
  * @see xml_yang_validate_add
  * @see xml_yang_validate_rpc
@@ -1172,8 +1173,8 @@ xml_yang_validate_list_key_only(clicon_handle h,
  */
 int
 xml_yang_validate_all(clicon_handle h,
-		      cxobj   *xt, 
+		      cxobj        *xt, 
-		      cbuf    *cbret)
+		      cxobj       **xret)
 {
     int        retval = -1;
     yang_stmt *ys;  /* yang node */
@@ -1188,7 +1189,7 @@ xml_yang_validate_all(clicon_handle h,
        and !Node has a config sub-statement and it is false */
     ys=xml_spec(xt);
     if (ys==NULL){
-	if (netconf_unknown_element(cbret, "application", xml_name(xt), NULL) < 0)
+	if (netconf_unknown_element_xml(xret, "application", xml_name(xt), NULL) < 0)
 	    goto done;
 	goto fail;
     }
@@ -1207,12 +1208,16 @@ xml_yang_validate_all(clicon_handle h,
  	    */
 	    if ((yc = yang_find(ys, Y_TYPE, NULL)) != NULL){
 		if (strcmp(yc->ys_argument, "leafref") == 0){
-		    if (validate_leafref(xt, yc, cbret) < 0)
+		    if ((ret = validate_leafref(xt, yc, xret)) < 0)
 			goto done;
+		    if (ret == 0)
+			goto fail;
 		}
 		else if (strcmp(yc->ys_argument, "identityref") == 0){
-		    if (validate_identityref(xt, ys, yc, cbret) < 0)
+		    if ((ret = validate_identityref(xt, ys, yc, xret)) < 0)
 			goto done;
+		    if (ret == 0)
+			goto fail;
 		}
 	    }
 	    break;
@@ -1230,7 +1235,7 @@ xml_yang_validate_all(clicon_handle h,
 		goto done;
 	    if (!nr){
 		ye = yang_find(yc, Y_ERROR_MESSAGE, NULL);
-		if (netconf_operation_failed(cbret, "application",
+		if (netconf_operation_failed_xml(xret, "application",
 					     ye?ye->ys_argument:"must xpath validation failed") < 0)
 		    goto done;
 		goto fail;
@@ -1242,7 +1247,7 @@ xml_yang_validate_all(clicon_handle h,
 	    if ((nr = xpath_vec_bool(xt, "%s", xpath)) < 0)
 		goto done;
 	    if (!nr){
-		if (netconf_operation_failed(cbret, "application",
+		if (netconf_operation_failed_xml(xret, "application",
 					     "when xpath validation failed") < 0)
 		    goto done;
 		goto fail;
@@ -1251,7 +1256,7 @@ xml_yang_validate_all(clicon_handle h,
     }
     x = NULL;
     while ((x = xml_child_each(xt, x, CX_ELMNT)) != NULL) {
-	if ((ret = xml_yang_validate_all(h, x, cbret)) < 0)
+	if ((ret = xml_yang_validate_all(h, x, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -1259,7 +1264,7 @@ xml_yang_validate_all(clicon_handle h,
     /* Check unique and min-max after choice test for example*/
     if (yang_config(ys) != 0){
 	/* Checks if next level contains any unique list constraints */
-	if ((ret = check_list_unique_minmax(xt, cbret)) < 0)
+	if ((ret = check_list_unique_minmax(xt, xret)) < 0)
 	    goto done;
 	if (ret == 0)
 	    goto fail;
@@ -1274,24 +1279,25 @@ xml_yang_validate_all(clicon_handle h,
 }
 
 /*! Translate a single xml node to a cligen variable vector. Note not recursive 
+ * @param[out] xret    Error XML tree (if ret == 0). Free with xml_free after use
  * @retval     1     Validation OK
- * @retval     0     Validation failed (cbret set)
+ * @retval     0     Validation failed (xret set)
  * @retval    -1     Error
  */
 int
 xml_yang_validate_all_top(clicon_handle h,
-			  cxobj   *xt, 
+			  cxobj        *xt, 
-			  cbuf    *cbret)
+			  cxobj       **xret)
 {
     int    ret;
     cxobj *x;
 
     x = NULL;
     while ((x = xml_child_each(xt, x, CX_ELMNT)) != NULL) {
-	if ((ret = xml_yang_validate_all(h, x, cbret)) < 1)
+	if ((ret = xml_yang_validate_all(h, x, xret)) < 1)
 	    return ret;
     }
-    if ((ret = check_list_unique_minmax(xt, cbret)) < 1)
+    if ((ret = check_list_unique_minmax(xt, xret)) < 1)
 	return ret;
     return 1;
 }
@@ -2053,7 +2059,7 @@ xml_tree_prune_flagged(cxobj *xt,
  */
 int
 xml_default(cxobj *xt, 
-	   void   *arg)
+	    void   *arg)
 {
     int        retval = -1;
     yang_stmt *ys;
@@ -2079,13 +2085,8 @@ xml_default(cxobj *xt,
 	    if (!cv_flag(y->ys_cv, V_UNSET)){  /* Default value exists */
 		if (!xml_find(xt, y->ys_argument)){
 
-#ifdef USE_XML_INSERT
 		    if ((xc = xml_new(y->ys_argument, NULL, y)) == NULL)
 			goto done;
-#else
-		    if ((xc = xml_new(y->ys_argument, xt, y)) == NULL)
-			goto done;
-#endif
 		    xml_flag_set(xc, XML_FLAG_DEFAULT);
 		    if ((xb = xml_new("body", xc, NULL)) == NULL)
 			goto done;
@@ -2098,18 +2099,12 @@ xml_default(cxobj *xt,
 			goto done;
 		    free(str);
 		    added++;
-#ifdef USE_XML_INSERT
 		    if (xml_insert(xt, xc) < 0)
 			goto done;
-#endif
 		}
 	    }
 	}
     }
-#ifndef USE_XML_INSERT
-    if (added)
-	xml_sort(xt, NULL);
-#endif
     retval = 0;
  done:
     return retval;
@@ -2206,7 +2201,9 @@ xml_spec_populate_rpc(clicon_handle h,
 	if (yrpc){
 	    xml_spec_set(x, yrpc);
 	    if ((yi = yang_find(yrpc, Y_INPUT, NULL)) != NULL){
-		/* kludge rpc -> input XXX THIS HIDES AN ERROR IN xml_spec_populate */
+		/* xml_spec_populate need to have parent with yang spec for
+		 * recursive population to work. Therefore, assign input yang
+		 * to rpc level although not 100% intuitive */
 		xml_spec_set(x, yi); 
 		if (xml_apply(x, CX_ELMNT, xml_spec_populate, yspec) < 0)
 		    goto done;
@@ -2497,7 +2494,7 @@ api_path2xml_vec(char             **vec,
 	else{
 	    if ((valvec = clicon_strsep(restval, ",", &nvalvec)) == NULL)
 		goto done;
-	    if (nvalvec > cvec_len(cvk)){ 	    
+	    if (nvalvec != cvec_len(cvk)){ 	    
 		clicon_err(OE_XML, EINVAL, "List key %s length mismatch", name);
 		goto fail;
 	    }

lib/src/clixon_yang.c

@@ -301,7 +301,38 @@ ys_free1(yang_stmt *ys)
     return 0;
 }
 
-/*! Free a tree of yang statements recursively */
+/*! Remove child i from parent yp (dont free) 
+ * @param[in]  yp   Parent node
+ * @param[in]  i    Order of child to remove
+ * @retval     NULL No such node, nothing done
+ * @retval     yc   returned orphaned yang node
+ * @see ys_free Deallocate yang node
+ * @note Do not call this in a loop of yang children (unless you know what you are doing)
+ */
+static yang_stmt *
+ys_prune(yang_stmt *yp,
+	 int        i)
+{
+    size_t     size;
+    yang_stmt *yc = NULL;
+    
+    if (i >= yp->ys_len)
+	goto done;
+    size = (yp->ys_len - i - 1)*sizeof(struct yang_stmt *);
+    yc = yp->ys_stmt[i];
+    memmove(&yp->ys_stmt[i],
+	    &yp->ys_stmt[i+1],
+	    size);
+    yc = yp->ys_stmt[yp->ys_len--] = NULL;;
+ done:
+    return yc;
+}
+
+/*! Free a yang statement tree recursively 
+ * @param[in]  ys   Yang node to remove and all its children recursively
+ * @note does not remove yang node from tree
+ * @see ys_prune  Remove from parent
+ */
 int 
 ys_free(yang_stmt *ys)
 {
@@ -1363,7 +1394,6 @@ ys_populate_leaf(clicon_handle h,
 	/* 3b. If not default value, indicate empty cv. */
 	cv_flag_set(cv, V_UNSET); /* no value (no default) */
     }
-
     /* 4. Check if leaf is part of list, if key exists mark leaf as key/unique */
     if (yparent && yparent->ys_keyword == Y_LIST){
 	if ((ret = yang_key_match(yparent, ys->ys_argument)) < 0)
@@ -1822,23 +1852,25 @@ ys_populate_unknown(clicon_handle h,
 
 /*! Populate with cligen-variables, default values, etc. Sanity checks on complete tree.
  *
- * We do this in 2nd pass after complete parsing to be sure to have a complete parse-tree
+ * @param[in]  ys  Yang statement
- * See ys_parse_sub for first pass and what can be assumed
+ * @param[in]  h   Clicon handle
+ * Preferably run this command using yang_apply
+ * Done in 2nd pass after complete parsing to be sure to have a complete
+ * parse-tree
+
  * After this pass, cv:s are set for LEAFs and LEAF-LISTs
+ * @see ys_parse_sub   for first pass and what can be assumed
+ * @see ys_populate2   for after grouping expand and augment
+ * (there may be more functions (all?) that may be moved to ys_populate2)
  */
 int
 ys_populate(yang_stmt    *ys, 
 	    void         *arg)
 {
-    int retval = -1;
+    int           retval = -1;
     clicon_handle h = (clicon_handle)arg;
     
     switch(ys->ys_keyword){
-    case Y_LEAF:
-    case Y_LEAF_LIST:
-	if (ys_populate_leaf(h, ys) < 0)
-	    goto done;
-	break;
     case Y_LIST:
 	if (ys_populate_list(h, ys) < 0)
 	    goto done;
@@ -1851,11 +1883,6 @@ ys_populate(yang_stmt    *ys,
 	if (ys_populate_length(h, ys) < 0)
 	    goto done;
 	break;
-    case Y_MANDATORY: /* call yang_mandatory() to check if set */
-    case Y_CONFIG:
-	if (ys_parse(ys, CGV_BOOL) == NULL) 
-	    goto done;
-	break;
     case Y_TYPE:
 	if (ys_populate_type(h, ys) < 0)
 	    goto done;
@@ -1880,9 +1907,42 @@ ys_populate(yang_stmt    *ys,
     return retval;
 }
 
+/*! Run after grouping expand and augment 
+ * @see ys_populate   run before grouping expand and augment
+ */
+static int
+ys_populate2(yang_stmt    *ys, 
+	    void         *arg)
+{
+    int           retval = -1;
+    clicon_handle h = (clicon_handle)arg;
+    
+    switch(ys->ys_keyword){
+    case Y_LEAF:
+    case Y_LEAF_LIST:
+	if (ys_populate_leaf(h, ys) < 0)
+	    goto done;
+	break;
+    case Y_MANDATORY: /* call yang_mandatory() to check if set */
+    case Y_CONFIG:
+	if (ys_parse(ys, CGV_BOOL) == NULL) 
+	    goto done;
+	break;
+    default:
+	break;
+    }
+    retval = 0;
+  done:
+    return retval;
+}
+
 /*! Resolve a grouping name from a point in the yang tree 
- * @retval  0  OK, but ygrouping determines if a grouping was resolved or not
+ * @param[in]  ys         Yang statement of "uses" statement doing the lookup
- * @retval -1  Error, with clicon_err called
+ * @param[in]  prefix     Prefix of grouping to look for
+ * @param[in]  name       Name of grouping to look for
+ * @param[out] ygrouping0 A found grouping yang structure as result
+ * @retval     0          OK, ygrouping may be NULL
+ * @retval    -1          Error, with clicon_err called
  */
 static int
 ys_grouping_resolve(yang_stmt  *ys, 
@@ -1999,23 +2059,94 @@ yang_augment_spec(yang_stmt *ysp,
     return retval;
 }
 
-/*! Macro expansion of grouping/uses done in step 2 of yang parsing 
+/*! Given a refine node, perform the refinement action on the target refine node
-  NOTE
+ * The RFC is somewhat complicate in the rules for refine.
-  RFC6020 says this:
+ * Most nodes will be replaced, but some are added
-    Identifiers appearing inside the grouping are resolved relative to the scope in which the 
+ * @param[in]  yr   Refine node
-    grouping is defined, not where it is used.  Prefix mappings, type names, grouping
+ * @param[in]  yt   Refine target node (will be modified)
-    names, and extension usage are evaluated in the hierarchy where the
+ * @see RFC7950 Sec 7.13.2
-    "grouping" statement appears. 
+ * There may be some missed cornercases
-  But it will be very difficult to generate keys etc with this semantics. So for now I
+ */
-  macro-expand them
-*/
 static int
-yang_expand(yang_stmt *yn)
+ys_do_refine(yang_stmt *yr,
+	     yang_stmt *yt)
+{
+    int           retval = -1;
+    yang_stmt    *yrc; /* refine child */
+    yang_stmt    *yrc1;
+    yang_stmt    *ytc; /* target child */
+    enum rfc_6020 keyw;
+    int           i;
+    
+    /* Loop through refine node children. First if remove do that first 
+     * In some cases remove a set of nodes.
+     */
+    yrc = NULL;
+    while ((yrc = yn_each(yr, yrc)) != NULL) {
+	keyw = yang_keyword_get(yrc);
+	switch (keyw){
+	case Y_DEFAULT: /* remove old, add new */
+	case Y_DESCRIPTION:
+	case Y_REFERENCE:
+	case Y_CONFIG:
+	case Y_MANDATORY:
+	case Y_PRESENCE:
+	case Y_MIN_ELEMENTS:
+	case Y_MAX_ELEMENTS:
+	case Y_EXTENSION:
+	    /* Remove old matching, dont increment due to prune in loop */
+	    for (i=0; i<yt->ys_len; ){
+		ytc = yt->ys_stmt[i];
+		if (keyw != yang_keyword_get(ytc)){
+		    i++;
+		    continue;
+		}
+		ys_prune(yt, i);
+		ys_free(ytc);
+	    }
+	    /* fall through and add if not found */
+	case Y_MUST:   /* keep old, add new */
+	case Y_IF_FEATURE:  
+	    break;
+	default:
+	    break;
+	}
+    }
+    /* Second, add the node(s) */
+    yrc = NULL;
+    while ((yrc = yn_each(yr, yrc)) != NULL) {
+	keyw = yang_keyword_get(yrc);
+	/* Make copy */
+	if ((yrc1 = ys_dup(yrc)) == NULL)
+	    goto done;
+	if (yn_insert(yt, yrc1) < 0)
+	    goto done;
+    }
+    retval = 0;
+ done:
+    return retval;
+}
+
+/*! Macro expansion of grouping/uses done in step 2 of yang parsing 
+ * RFC7950:
+ * Identifiers appearing inside the grouping are resolved
+ * relative to the scope in which the  grouping is defined, not where it is
+ * used.  Prefix mappings, type names, grouping  names, and extension usage are
+ * evaluated in the hierarchy where the "grouping" statement appears. 
+ *   The identifiers defined in the grouping are not bound to a namespace
+ * until the contents of the grouping are added to the schema tree via a
+ * "uses" statement that does not appear inside a "grouping" statement,
+ * at which point they are bound to the namespace of the current module.
+ */
+static int
+yang_expand_grouping(yang_stmt *yn)
 {
     int        retval = -1;
     yang_stmt *ys = NULL;
-    yang_stmt *ygrouping;
+    yang_stmt *ygrouping;  /* grouping original */
-    yang_stmt *yg;
+    yang_stmt *ygrouping2; /* grouping copy */
+    yang_stmt *yg;         /* grouping child */
+    yang_stmt *yr;         /* refinement */
     int        glen;
     int        i;
     int        j;
@@ -2034,27 +2165,32 @@ yang_expand(yang_stmt *yn)
 	    prefix  = yarg_prefix(ys); /* And this its prefix */
 	    if (ys_grouping_resolve(ys, prefix, name, &ygrouping) < 0)
 		goto done;
-
+	    if (prefix){
+		free(prefix); 
+		prefix = NULL;
+	    }
 	    if (ygrouping == NULL){
 		clicon_log(LOG_NOTICE, "%s: Yang error : grouping \"%s\" not found in module \"%s\"", 
 			   __FUNCTION__, ys->ys_argument, ys_module(ys)->ys_argument);
 		goto done;
 		break;
 	    }
-	    if (prefix)
-		free(prefix); /* XXX move up */
 	    /* Check mark flag to see if this grouping (itself) has been expanded
 	       If not, this needs to be done before we can insert it into
 	       the 'uses' place */
 	    if ((ygrouping->ys_flags & YANG_FLAG_MARK) == 0){ 
-		if (yang_expand(ygrouping) < 0)
+		if (yang_expand_grouping(ygrouping) < 0)
 		    goto done;
 		ygrouping->ys_flags |= YANG_FLAG_MARK; /* Mark as expanded */
 	    }
+	    /* Make a copy of the grouping, then make refinements to this copy
+	     */
+	    if ((ygrouping2 = ys_dup(ygrouping)) == NULL)
+		goto done;
 	    /* Replace ys with ygrouping,... 
 	     * First enlarge parent vector 
 	     */
-	    glen = ygrouping->ys_len;
+	    glen = ygrouping2->ys_len;
 	    /* 
 	     * yn is parent: the children of ygrouping replaces ys.
 	     * Is there a case when glen == 0?  YES AND THIS BREAKS
@@ -2064,7 +2200,7 @@ yang_expand(yang_stmt *yn)
 		yn->ys_len += glen - 1;
 		if (glen && (yn->ys_stmt = realloc(yn->ys_stmt, (yn->ys_len)*sizeof(yang_stmt *))) == 0){
 		    clicon_err(OE_YANG, errno, "realloc");
-		    return -1;
+		    goto done;
 		}
 		/* Then move all existing elements up from i+1 (not uses-stmt) */
 		if (size)
@@ -2072,16 +2208,42 @@ yang_expand(yang_stmt *yn)
 			    &yn->ys_stmt[i+1],
 			    size);
 	    }
+
+	    /* Iterate through refinements and modify grouping copy 
+	     * See RFC 7950 7.13.2 yrt is the refine target node
+	     */
+	    yr = NULL;
+	    while ((yr = yn_each(ys, yr)) != NULL) {
+		yang_stmt *yrt; /* refine target node */
+		if (yang_keyword_get(yr) != Y_REFINE)
+		    continue;
+		/* Find a node */
+		if (yang_desc_schema_nodeid(ygrouping2,
+					    yang_argument_get(yr),
+					    -1,
+					    &yrt) < 0)
+		    goto done;
+		/* Not found, try next */
+		if (yrt == NULL) 		
+		    continue;
+		/* Do the actual refinement */
+		if (ys_do_refine(yr, yrt) < 0)
+		    goto done;
+		/* RFC: The argument is a string that identifies a node in the 
+		 * grouping.  I interpret that as only one node --> break */
+		break;
+	    }
 	    /* Then copy and insert each child element */
 	    for (j=0; j<glen; j++){
-		if ((yg = ys_dup(ygrouping->ys_stmt[j])) == NULL)
+		yg = ygrouping2->ys_stmt[j]; /* Child of refined copy */
-		    goto done;
 		yn->ys_stmt[i+j] = yg;
 		yg->ys_parent = yn;
 	    }
-	    /* XXX: refine */
 	    /* Remove 'uses' node */
 	    ys_free(ys); 
+	    /* Remove the grouping copy */
+	    ygrouping2->ys_len = 0;
+	    ys_free(ygrouping2);
 	    break; /* Note same child is re-iterated since it may be changed */
 	default:
 	    i++;
@@ -2091,7 +2253,7 @@ yang_expand(yang_stmt *yn)
     /* Second pass since length may have changed */
     for (i=0; i<yn->ys_len; i++){
 	ys = yn->ys_stmt[i];
-	if (yang_expand(ys) < 0)
+	if (yang_expand_grouping(ys) < 0)
 	    goto done;
     }
     retval = 0;
@@ -2441,8 +2603,8 @@ ys_schemanode_check(yang_stmt *ys,
 	if (yang_desc_schema_nodeid(yp, ys->ys_argument, -1, &yres) < 0)
 	    goto done;
 	if (yres == NULL){
-	    clicon_err(OE_YANG, 0, "schemanode sanity check of %d %s", 
+	    clicon_err(OE_YANG, 0, "schemanode sanity check of %s %s", 
-		       ys->ys_keyword,
+		       yang_key2str(ys->ys_keyword),
 		       ys->ys_argument);
 	    goto done;
 	}
@@ -2609,15 +2771,20 @@ yang_parse_post(clicon_handle h,
 
     /* 6: Macro expansion of all grouping/uses pairs. Expansion needs marking */
     for (i=modnr; i<yspec->ys_len; i++){
-	if (yang_expand(yspec->ys_stmt[i]) < 0)
+	if (yang_expand_grouping(yspec->ys_stmt[i]) < 0)
 	    goto done;
 	yang_apply(yspec->ys_stmt[i], -1, ys_flag_reset, (void*)YANG_FLAG_MARK);
     }
 
-    /* 7: Top-level augmentation of all modules XXX: only new modules? */
+    /* 7: Top-level augmentation of all modules. (Augment also in uses) */
     if (yang_augment_spec(yspec, modnr) < 0)
 	goto done;
 
+    /* 4: Go through parse tree and do 2nd step populate (eg default) */
+    for (i=modnr; i<yspec->ys_len; i++)
+	if (yang_apply(yspec->ys_stmt[i], -1, ys_populate2, (void*)h) < 0)
+	    goto done;
+
     /* 8: sanity check of schemanode references, need more here */
     for (i=modnr; i<yspec->ys_len; i++)
 	if (yang_apply(yspec->ys_stmt[i], -1, ys_schemanode_check, NULL) < 0)
@@ -2829,19 +2996,13 @@ yang_spec_load_dir(clicon_handle h,
 	 * This is a failsafe in case anything else fails
 	 */
 	if (revm && rev0){
-	    int size;
 	    if (revm > rev0) /* Loaded module is older or eq -> remove ym */
 		ym = ym0;
 	    for (j=0; j<yspec->ys_len; j++)
 		if (yspec->ys_stmt[j] == ym)
 		    break;
-	    size = (yspec->ys_len - j - 1)*sizeof(struct yang_stmt *);
+	    ys_prune(yspec, j);
-	    memmove(&yspec->ys_stmt[j],
-		    &yspec->ys_stmt[j+1],
-		    size);
 	    ys_free(ym);
-	    yspec->ys_len--;
-	    yspec->ys_stmt[yspec->ys_len] = NULL;
 	}
     }
     if (yang_parse_post(h, yspec, modnr) < 0)
@@ -3101,7 +3262,7 @@ yang_abs_schema_nodeid(yang_stmt    *yspec,
  * @param[out] yres          First yang node matching schema nodeid
  * @retval     0             OK
  * @retval    -1             Error, with clicon_err called
- * @see yang_schema_nodeid
+ * @see yang_abs_schema_nodeid
  * Used in yang: unique, refine, uses augment
  */
 int
@@ -3429,6 +3590,7 @@ yang_arg2cvec(yang_stmt *ys,
 
 /*! Check if yang node yn has key-stmt as child which matches name
  *
+ * The function looks at the LIST argument string (not actual children)
  * @param[in]  yn   Yang node with sub-statements (look for a key child)
  * @param[in]  name Check if this name (eg "b") is a key in the yang key statement
  *

test/all.sh

@@ -2,6 +2,9 @@
 # Run, eg as:
 # ./all.sh 2>&1 | tee test.log # break on first test
 
+# Pattern to run tests, default is all, but you may want to narrow it down
+: ${pattern:=test_*.sh}
+
 if [ $# -gt 0 ]; then 
     echo "usage: $0 # detailed logs and stopon first error"
     exit -1
@@ -9,7 +12,7 @@ fi
 
 err=0
 testnr=0
-for test in test_*.sh; do
+for test in $pattern; do
     if [ $testnr != 0 ]; then echo; fi
     testfile=$test
     . ./$test 

test/mem.sh

@@ -2,6 +2,9 @@
 # Run valgrind leak test for cli, restconf, netconf or background.
 # Stop on first error
     
+# Pattern to run tests, default is all, but you may want to narrow it down
+: ${pattern:=test_*.sh}
+
 # Run valgrindtest once, args:
 # what: (cli|netconf|restconf|backend)* # no args means all
 memonce(){
@@ -45,7 +48,7 @@ memonce(){
     esac
 
     err=0
-    for test in test_*.sh; do
+    for test in $pattern; do
 	if [ $testnr != 0 ]; then echo; fi
 	testfile=$test
 	. ./$test 

test/test_augment.sh

@@ -22,6 +22,7 @@ cat <<EOF > $cfg
   <CLICON_FEATURE>a:test</CLICON_FEATURE>
   <CLICON_YANG_DIR>$dir</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
   <CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
   <CLICON_CLI_MODE>$APPNAME</CLICON_CLI_MODE>
@@ -66,14 +67,14 @@ module ietf-interfaces {
     }
   }
   grouping endpoint {
-       description "A reusable endpoint group.";
+       description "A reusable endpoint group. From rf7950 Sec 7.12.2";
-       leaf mip {
+       leaf ip {
          type string;
        }
-       leaf mport {
+       leaf port {
          type uint16;
        }
-  }
+   }
 }
 EOF
 
@@ -98,12 +99,14 @@ module example-augment {
        identity you {
           base my-type;
        }
-       grouping mypoint {
+       grouping localgroup {
-         description "A reusable endpoint group.";
+         description "Local grouping defining lid and lport";
-         leaf ip {
+         leaf lid {
-           type string;
+            description "this will be kept as-is";
+            type string;
          }
-         leaf port {
+         leaf lport {
+           description "this will be refined";
            type uint16;
          }
        }
@@ -124,12 +127,14 @@ module example-augment {
               }
           }
           uses if:endpoint {
+            description "Use an external grouping defining ip and port";
             refine port {
               default 80;
             }
           }
-          uses mypoint {
+          uses localgroup {
-            refine mport {
+            description "Use a local grouping defining lip and lport";
+            refine lport {
               default 8080;
             }
           }
@@ -137,37 +142,35 @@ module example-augment {
 }
 EOF
 
-new "test params: -f $cfg -y $fyang"
+new "test params: -f $cfg"
 
 if [ $BE -ne 0 ]; then
     new "kill old backend"
-    sudo clixon_backend -zf $cfg -y $fyang
+    sudo clixon_backend -zf $cfg
     if [ $? -ne 0 ]; then
 	err
     fi
-    new "start backend -s init -f $cfg -y $fyang"
+    new "start backend -s init -f $cfg"
-    start_backend -s init -f $cfg -y $fyang
+    start_backend -s init -f $cfg
 
     new "waiting"
     sleep $RCWAIT
 fi
 
 # mandatory-leaf See RFC7950 Sec 7.17
-# Error1: the xml should have xmlns for "mymod"
-#                XMLNS_YANG_ONLY must be undeffed
 new "netconf set interface with augmented type and mandatory leaf"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
   <interface xmlns:mymod="urn:example:augment">
     <name>e1</name>
     <type>mymod:some-new-iftype</type>
     <mymod:mandatory-leaf>true</mymod:mandatory-leaf>
   </interface></interfaces></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
-new "netconf validate ok"
+new "netconf verify get with refined ports"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces"><interface xmlns:mymod="urn:example:augment"><name>e1</name><type>mymod:some-new-iftype</type><mandatory-leaf>true</mandatory-leaf><port>80</port><lport>8080</lport></interface></interfaces></data></rpc-reply>]]>]]>$'
 
 new "netconf set identity defined in other"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
   <interface xmlns:mymod="urn:example:augment">
     <name>e2</name>
     <type>fddi</type>
@@ -176,10 +179,10 @@ expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><can
   </interface></interfaces></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate ok"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
 
 new "netconf set identity defined in main"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
 <interface xmlns:mymod="urn:example:augment">
    <name>e3</name>
    <type>fddi</type>
@@ -188,9 +191,11 @@ expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><can
  </interface></interfaces></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate ok"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]>$'
+
+new "discard"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 if [ $BE -eq 0 ]; then
     exit # BE

test/test_cli.sh

@@ -46,7 +46,7 @@ if [ $BE -ne 0 ]; then
     start_backend -s init -f $cfg
 
     new "waiting"
-    sleep $RCWAIT
+    wait_backend
 fi
 
 new "cli configure top"

test/test_identity.sh

@@ -15,6 +15,7 @@ cat <<EOF > $cfg
   <CLICON_YANG_DIR>$dir</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
   <CLICON_BACKEND_DIR>/usr/local/lib/$APPNAME/backend</CLICON_BACKEND_DIR>
   <CLICON_BACKEND_REGEXP>example_backend.so$</CLICON_BACKEND_REGEXP>
@@ -108,80 +109,82 @@ cat <<EOF > $fyang
      }
 EOF
 
-new "test params: -f $cfg -y $fyang"
+new "test params: -f $cfg"
 if [ $BE -ne 0 ]; then
     new "kill old backend"
     sudo clixon_backend -zf $cfg
     if [ $? -ne 0 ]; then
 	err
     fi
-    new "start backend -s init -f $cfg -y $fyang"
+    new "start backend -s init -f $cfg"
-    start_backend -s init -f $cfg -y $fyang
+    start_backend -s init -f $cfg
 
     new "waiting"
     sleep $RCWAIT
 fi
 
 new "Set crypto to aes"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">aes</crypto></config></edit-config></rpc>]]>]]>' '^<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><ok/></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">aes</crypto></config></edit-config></rpc>]]>]]>' '^<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><ok/></rpc-reply>]]>]]>$'
 
 new "netconf validate "
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "Set crypto to mc:aes"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">mc:aes</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">mc:aes</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "Set crypto to des:des3"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">des:des3</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">des:des3</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "Set crypto to mc:foo"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">mc:foo</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">mc:foo</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "Set crypto to des:des3 using xmlns"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto" xmlns:des="urn:example:des">des:des3</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto" xmlns:des="urn:example:des">des:des3</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
+if false; then
 # XXX this is not supported
-#new "Set crypto to x:des3 using xmlns"
+new "Set crypto to x:des3 using xmlns"
-#expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto" xmlns:x="urn:example:des">x:des3</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto" xmlns:x="urn:example:des">x:des3</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+fi # not supported
 
 new "Set crypto to foo:bar"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">foo:bar</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><crypto xmlns="urn:example:my-crypto">foo:bar</crypto></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
-new "netconf validate"
+new "netconf validate (expect fail)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>Identityref validation failed, foo:bar not derived from crypto-alg</error-message></rpc-error></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>Identityref validation failed, foo:bar not derived from crypto-alg</error-message></rpc-error></rpc-reply>]]>]]>$"
 
 new "cli set crypto to mc:aes"
-expectfn "$clixon_cli -1 -f $cfg -y $fyang -l o set crypto mc:aes" 0 "^$"
+expectfn "$clixon_cli -1 -f $cfg -l o set crypto mc:aes" 0 "^$"
 
 new "cli validate"
-expectfn "$clixon_cli -1 -f $cfg -y $fyang -l o validate" 0 "^$"
+expectfn "$clixon_cli -1 -f $cfg -l o validate" 0 "^$"
 
 new "cli set crypto to aes"
-expectfn "$clixon_cli -1 -f $cfg -y $fyang -l o set crypto aes" 0 "^$"
+expectfn "$clixon_cli -1 -f $cfg -l o set crypto aes" 0 "^$"
 
 new "cli validate"
-expectfn "$clixon_cli -1 -f $cfg -y $fyang -l o validate" 0 "^$"
+expectfn "$clixon_cli -1 -f $cfg -l o validate" 0 "^$"
 
 new "cli set crypto to des:des3"
-expectfn "$clixon_cli -1 -f $cfg -y $fyang -l o set crypto des:des3" 0 "^$"
+expectfn "$clixon_cli -1 -f $cfg -l o set crypto des:des3" 0 "^$"
 
 new "cli validate"
-expectfn "$clixon_cli -1 -f $cfg -y $fyang -l o validate" 0 "^$"
+expectfn "$clixon_cli -1 -f $cfg -l o validate" 0 "^$"
 
 if [ $BE -eq 0 ]; then
     exit # BE

test/test_nacm_ext.sh

@@ -149,7 +149,8 @@ new "start restconf daemon (-a is enable http basic auth)"
 start_restconf -f $cfg -- -a
 
 new "waiting"
-sleep $RCWAIT
+wait_backend
+wait_restconf
 
 new "auth get"
 expecteq "$(curl -u andy:bar -sS -X GET http://localhost/restconf/data)" 0 '{"data": {"clixon-example:state": {"op": ["42","41","43"]}}}
@@ -183,10 +184,10 @@ new "admin edit nacm"
 expecteq "$(curl -u andy:bar -sS -X PUT -d '{"nacm-example:x": 1}' http://localhost/restconf/data/nacm-example:x)" 0 ""
 
 new "limited edit nacm"
-expecteq "$(curl -u wilma:bar -sS -X PUT -d '{"x": 2}' http://localhost/restconf/data/nacm-example:x)" 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "access-denied","error-severity": "error","error-message": "default deny"}}}
'
+expecteq "$(curl -u wilma:bar -sS -X PUT -d '{"nacm-example:x": 2}' http://localhost/restconf/data/nacm-example:x)" 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "access-denied","error-severity": "error","error-message": "default deny"}}}
'
 
 new "guest edit nacm"
-expecteq "$(curl -u guest:bar -sS -X PUT -d '{"x": 3}' http://localhost/restconf/data/nacm-example:x)" 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "access-denied","error-severity": "error","error-message": "access denied"}}}
'
+expecteq "$(curl -u guest:bar -sS -X PUT -d '{"nacm-example:x": 3}' http://localhost/restconf/data/nacm-example:x)" 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "access-denied","error-severity": "error","error-message": "access denied"}}}
'
 
 new "cli show conf as admin"
 expectfn "$clixon_cli -1 -U andy -l o -f $cfg show conf" 0 "^x 1;$"

test/test_nacm_module_read.sh

@@ -153,7 +153,7 @@ new "commit it"
 expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "enable nacm"
-expecteq "$(curl -u andy:bar -sS -X PUT -d '{"enable-nacm": true}' http://localhost/restconf/data/ietf-netconf-acm:nacm/enable-nacm)" 0 ""
+expecteq "$(curl -u andy:bar -sS -X PUT -d '{"ietf-netconf-acm:enable-nacm": true}' http://localhost/restconf/data/ietf-netconf-acm:nacm/enable-nacm)" 0 ""
 
 #--------------- nacm enabled
 
@@ -256,7 +256,7 @@ expecteof "$clixon_netconf -U guest -qf $cfg" 0 '<rpc xmlns="urn:ietf:params:xml
 #------------------ Set read-default permit
 
 new "admin set read-default permit"
-expecteq "$(curl -u andy:bar -sS -X PUT -d '{"read-default": "permit"}' http://localhost/restconf/data/ietf-netconf-acm:nacm/read-default)" 0 ""
+expecteq "$(curl -u andy:bar -sS -X PUT -d '{"ietf-netconf-acm:read-default": "permit"}' http://localhost/restconf/data/ietf-netconf-acm:nacm/read-default)" 0 ""
 
 new "limit read ok"
 expecteq "$(curl -u wilma:bar -sS -X GET http://localhost/restconf/data/clixon-example:translate)" 0 '{"clixon-example:translate": [{"k": "key42","value": "val42"},{ "k": "key43","value": "val43"}]}

test/test_nacm_module_write.sh

@@ -164,7 +164,7 @@ nacm(){
     expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
     new "enable nacm"
-    expecteq "$(curl -u andy:bar -sS -X PUT -d '{"enable-nacm": true}' http://localhost/restconf/data/ietf-netconf-acm:nacm/enable-nacm)" 0 ""
+    expecteq "$(curl -u andy:bar -sS -X PUT -d '{"ietf-netconf-acm:enable-nacm": true}' http://localhost/restconf/data/ietf-netconf-acm:nacm/enable-nacm)" 0 ""
 }
 
 #--------------- enable nacm

test/test_nacm_protocol.sh

@@ -162,7 +162,7 @@ new "commit it"
 expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "enable nacm"
-expecteq "$(curl -u andy:bar -sS -X PUT -d '{"enable-nacm": true}' http://localhost/restconf/data/ietf-netconf-acm:nacm/enable-nacm)" 0 ""
+expecteq "$(curl -u andy:bar -sS -X PUT -d '{"ietf-netconf-acm:enable-nacm": true}' http://localhost/restconf/data/ietf-netconf-acm:nacm/enable-nacm)" 0 ""
 
 #--------------- nacm enabled
 

test/test_netconf.sh

@@ -46,7 +46,7 @@ if [ $BE -ne 0 ]; then
     start_backend -s init -f $cfg -- -s
 
     new "waiting"
-    sleep $RCWAIT
+    wait_backend
 fi
 
 new "netconf hello"

test/test_order.sh

@@ -33,6 +33,7 @@ cat <<EOF > $cfg
   <CLICON_CONFIGFILE>/tmp/conf_yang.xml</CLICON_CONFIGFILE>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
   <CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
   <CLICON_CLI_MODE>$APPNAME</CLICON_CLI_MODE>
@@ -149,16 +150,16 @@ cat <<EOF > $dbdir/running_db
 </config>
 EOF
 
-new "test params: -s running -f $cfg -y $fyang -- -s"
+new "test params: -s running -f $cfg -- -s"
 
 if [ $BE -ne 0 ]; then
     new "kill old backend"
-    sudo clixon_backend -zf $cfg -y $fyang
+    sudo clixon_backend -zf $cfg
     if [ $? -ne 0 ]; then
 	err
     fi
     new "start backend"
-    start_backend -s running -f $cfg -y $fyang -- -s
+    start_backend -s running -f $cfg -- -s
 
     new "waiting"
     sleep $RCWAIT
@@ -169,75 +170,75 @@ new "state data (should be unordered: 42,41,43)"
 cat <<EOF > $tmp
 <rpc><get><filter type="xpath" select="state"/></get></rpc>]]>]]>
 EOF
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "$(cat $tmp)" '<rpc-reply><data><state xmlns="urn:example:clixon"><op>42</op><op>41</op><op>43</op></state></data></rpc-reply>]]>]]>'
+expecteof "$clixon_netconf -qf $cfg" 0 "$(cat $tmp)" '<rpc-reply><data><state xmlns="urn:example:clixon"><op>42</op><op>41</op><op>43</op></state></data></rpc-reply>]]>]]>'
 
 # Check as file
 new "verify running from start, should be: c,l,y0,y1,y2,y3; y1 and y3 sorted."
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><running/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><c xmlns="urn:example:order"><d>hej</d></c><l xmlns="urn:example:order">hopp</l><y0 xmlns="urn:example:order">d</y0><y0 xmlns="urn:example:order">b</y0><y0 xmlns="urn:example:order">c</y0><y0 xmlns="urn:example:order">a</y0><y1 xmlns="urn:example:order">a</y1><y1 xmlns="urn:example:order">b</y1><y1 xmlns="urn:example:order">c</y1><y1 xmlns="urn:example:order">d</y1><y2 xmlns="urn:example:order"><k>d</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>a</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>c</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>bar</a></y2><y3 xmlns="urn:example:order"><k>a</k><a>bar</a></y3><y3 xmlns="urn:example:order"><k>b</k><a>bar</a></y3><y3 xmlns="urn:example:order"><k>c</k><a>bar</a></y3><y3 xmlns="urn:example:order"><k>d</k><a>bar</a></y3></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><running/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><c xmlns="urn:example:order"><d>hej</d></c><l xmlns="urn:example:order">hopp</l><y0 xmlns="urn:example:order">d</y0><y0 xmlns="urn:example:order">b</y0><y0 xmlns="urn:example:order">c</y0><y0 xmlns="urn:example:order">a</y0><y1 xmlns="urn:example:order">a</y1><y1 xmlns="urn:example:order">b</y1><y1 xmlns="urn:example:order">c</y1><y1 xmlns="urn:example:order">d</y1><y2 xmlns="urn:example:order"><k>d</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>a</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>c</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>bar</a></y2><y3 xmlns="urn:example:order"><k>a</k><a>bar</a></y3><y3 xmlns="urn:example:order"><k>b</k><a>bar</a></y3><y3 xmlns="urn:example:order"><k>c</k><a>bar</a></y3><y3 xmlns="urn:example:order"><k>d</k><a>bar</a></y3></data></rpc-reply>]]>]]>$'
 
 new "get each ordered-by user leaf-list"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y2[k='a']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>a</k><a>bar</a></y2></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y2[k='a']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>a</k><a>bar</a></y2></data></rpc-reply>]]>]]>$'
 
 new "get each ordered-by user leaf-list"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y3[k='a']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y3 xmlns="urn:example:order"><k>a</k><a>bar</a></y3></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y3[k='a']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y3 xmlns="urn:example:order"><k>a</k><a>bar</a></y3></data></rpc-reply>]]>]]>$'
 
 new "get each ordered-by user leaf-list"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y2[k='b']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>b</k><a>bar</a></y2></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y2[k='b']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>b</k><a>bar</a></y2></data></rpc-reply>]]>]]>$'
 
 new "get each ordered-by user leaf-list"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y3[k='b']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y3 xmlns="urn:example:order"><k>b</k><a>bar</a></y3></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><running/></source><filter type=\"xpath\" select=\"/y3[k='b']\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><y3 xmlns="urn:example:order"><k>b</k><a>bar</a></y3></data></rpc-reply>]]>]]>$'
 
 new "delete candidate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><default-operation>none</default-operation><config operation="delete"/></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><default-operation>none</default-operation><config operation="delete"/></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 # LEAF_LISTS
 
 new "add two entries (c,b) to leaf-list user order"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y0 xmlns="urn:example:order">c</y0><y0 xmlns="urn:example:order">b</y0></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y0 xmlns="urn:example:order">c</y0><y0 xmlns="urn:example:order">b</y0></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "add one entry (a) to leaf-list user order"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y0 xmlns="urn:example:order">a</y0></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y0 xmlns="urn:example:order">a</y0></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf commit"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "add one entry (0) to leaf-list user order after commit"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y0 xmlns="urn:example:order">0</y0></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y0 xmlns="urn:example:order">0</y0></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf commit"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "verify leaf-list user order in running (as entered: c,b,a,0)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><running/></source><filter type="xpath" select="/y0"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><y0 xmlns="urn:example:order">c</y0><y0 xmlns="urn:example:order">b</y0><y0 xmlns="urn:example:order">a</y0><y0 xmlns="urn:example:order">0</y0></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><running/></source><filter type="xpath" select="/y0"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><y0 xmlns="urn:example:order">c</y0><y0 xmlns="urn:example:order">b</y0><y0 xmlns="urn:example:order">a</y0><y0 xmlns="urn:example:order">0</y0></data></rpc-reply>]]>]]>$'
 
 # LISTS
 
 new "add two entries to list user order"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order"><k>c</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>foo</a></y2></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order"><k>c</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>foo</a></y2></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "add one entry to list user order"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order"><k>a</k><a>fie</a></y2></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order"><k>a</k><a>fie</a></y2></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "verify list user order (as entered)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/y2"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>c</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>foo</a></y2><y2 xmlns="urn:example:order"><k>a</k><a>fie</a></y2></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/y2"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>c</k><a>bar</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>foo</a></y2><y2 xmlns="urn:example:order"><k>a</k><a>fie</a></y2></data></rpc-reply>]]>]]>$'
 
 new "Overwrite existing ordered-by user y2->c"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order">
 <k>c</k><a>newc</a>
 </y2></config></edit-config></rpc>]]>]]>'
 
 new "Overwrite existing ordered-by user y2->b"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order">
 <k>b</k><a>newb</a>
 </y2></config></edit-config></rpc>]]>]]>'
 
 new "Overwrite existing ordered-by user y2->a"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><y2 xmlns="urn:example:order">
 <k>a</k><a>newa</a>
 </y2></config></edit-config></rpc>]]>]]>'
 
 new "Tests for no duplicates."
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/y2"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>c</k><a>newc</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>newb</a></y2><y2 xmlns="urn:example:order"><k>a</k><a>newa</a></y2></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/y2"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><y2 xmlns="urn:example:order"><k>c</k><a>newc</a></y2><y2 xmlns="urn:example:order"><k>b</k><a>newb</a></y2><y2 xmlns="urn:example:order"><k>a</k><a>newa</a></y2></data></rpc-reply>]]>]]>$'
 
 #-- order by type rather than strings.
 # there are three leaf-lists:strings, ints, and decimal64, and two lists:
@@ -246,44 +247,44 @@ expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><cand
 # The check is to write the entries as: 10,2,1, and then expect them to
 # get back as 1,2,10 (if typed).
 new "put strings (10,2,1)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
 <strings>10</strings><strings>2</strings><strings>1</strings>
 </types></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "check string order (1,10,2)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/strings"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><strings>1</strings><strings>10</strings><strings>2</strings></types></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/strings"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><strings>1</strings><strings>10</strings><strings>2</strings></types></data></rpc-reply>]]>]]>$'
 
 new "put leaf-list int (10,2,1)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
 <ints>10</ints><ints>2</ints><ints>1</ints>
 </types></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "check leaf-list int order (1,2,10)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/ints"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><ints>1</ints><ints>2</ints><ints>10</ints></types></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/ints"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><ints>1</ints><ints>2</ints><ints>10</ints></types></data></rpc-reply>]]>]]>$'
 
 new "put list int (10,2,1)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
 <listints><a>10</a></listints><listints><a>2</a></listints><listints><a>1</a></listints>
 </types></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "check list int order (1,2,10)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/listints"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><listints><a>1</a></listints><listints><a>2</a></listints><listints><a>10</a></listints></types></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/listints"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><listints><a>1</a></listints><listints><a>2</a></listints><listints><a>10</a></listints></types></data></rpc-reply>]]>]]>$'
 
 new "put leaf-list decimal64 (10,2,1)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
 <decs>10.0</decs><decs>2.0</decs><decs>1.0</decs>
 </types></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "check leaf-list decimal64 order (1,2,10)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/decs"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><decs>1.0</decs><decs>2.0</decs><decs>10.0</decs></types></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/decs"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><decs>1.0</decs><decs>2.0</decs><decs>10.0</decs></types></data></rpc-reply>]]>]]>$'
 
 new "put list decimal64 (10,2,1)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><types xmlns="urn:example:order">
 <listdecs><a>10.0</a></listdecs><listdecs><a>2.0</a></listdecs><listdecs><a>1.0</a></listdecs>
 </types></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "check list decimal64 order (1,2,10)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/listdecs"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><listdecs><a>1.0</a></listdecs><listdecs><a>2.0</a></listdecs><listdecs><a>10.0</a></listdecs></types></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/types/listdecs"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><types xmlns="urn:example:order"><listdecs><a>1.0</a></listdecs><listdecs><a>2.0</a></listdecs><listdecs><a>10.0</a></listdecs></types></data></rpc-reply>]]>]]>$'
 
 if [ $BE -eq 0 ]; then
     exit # BE

test/test_restconf.sh

@@ -43,6 +43,7 @@ if [ $BE -ne 0 ]; then
     if [ $? -ne 0 ]; then
 	err
     fi
+    sudo pkill clixon_backend # to be sure
     new "start backend -s init -f $cfg -- -s"
     start_backend -s init -f $cfg -- -s
 fi
@@ -57,8 +58,6 @@ new "waiting"
 wait_backend
 wait_restconf
 
-new "restconf tests"
-
 new "restconf root discovery. RFC 8040 3.1 (xml+xrd)"
 expecteq "$(curl -s -X GET http://localhost/.well-known/host-meta)" 0 "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
    <Link rel='restconf' href='/restconf'/>

test/test_restconf2.sh

@@ -14,9 +14,9 @@ fyang=$dir/restconf.yang
 cat <<EOF > $cfg
 <clixon-config xmlns="http://clicon.org/config">
   <CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
-  <CLICON_YANG_DIR>/usr/local/var</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_RESTCONF_PRETTY>false</CLICON_RESTCONF_PRETTY>
   <CLICON_SOCK>/usr/local/var/$APPNAME/$APPNAME.sock</CLICON_SOCK>
   <CLICON_BACKEND_PIDFILE>$dir/restconf.pidfile</CLICON_BACKEND_PIDFILE>
@@ -66,7 +66,7 @@ module example{
 }
 EOF
 
-new "test params: -f $cfg -y $fyang"
+new "test params: -f $cfg"
 
 if [ $BE -ne 0 ]; then
     new "kill old backend"
@@ -74,28 +74,30 @@ if [ $BE -ne 0 ]; then
     if [ $? -ne 0 ]; then
 	err
     fi
-    new "start backend -s init -f $cfg -y $fyang"
+    sudo pkill clixon_backend # to be sure
-    start_backend -s init -f $cfg -y $fyang
+    new "start backend -s init -f $cfg"
+    start_backend -s init -f $cfg
 fi
 
 new "kill old restconf daemon"
 sudo pkill -u www-data -f "/www-data/clixon_restconf"
 
 new "start restconf daemon"
-start_restconf -f $cfg -y $fyang
+start_restconf -f $cfg
 
 new "waiting"
 wait_backend
 wait_restconf
 
-new "restconf tests"
-
 new "restconf POST tree without key"
 expectfn 'curl -s -X POST -d {"example:cont1":{"interface":{"type":"regular"}}} http://localhost/restconf/data' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "missing-element","error-info": {"bad-element": "name"},"error-severity": "error","error-message": "Mandatory key"}}}
'
 
 new "restconf POST initial tree"
 expectfn 'curl -s -X POST -d {"example:cont1":{"interface":{"name":"local0","type":"regular"}}} http://localhost/restconf/data' 0 ""
 
+new "restconf POST top without namespace"
+expectfn 'curl -s -X POST -d {"cont1":{"interface":{"name":"local0","type":"regular"}}} http://localhost/restconf/data' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "unknown-element","error-info": {"bad-element": "cont1"},"error-severity": "error","error-message": "Unassigned yang spec"}}}'
+
 new "restconf GET datastore initial"
 expectfn "curl -s -X GET http://localhost/restconf/data/example:cont1" 0 '{"example:cont1": {"interface": \[{"name": "local0","type": "regular"}\]}}'
 
@@ -114,17 +116,19 @@ new "restconf GET if-type"
 expectfn "curl -s -X GET http://localhost/restconf/data/example:cont1/interface=local0/type" 0 '{"example:type": "regular"}'
 
 new "restconf POST interface without mandatory type"
-expectfn 'curl -s -X POST http://localhost/restconf/data/example:cont1 -d {"interface":{"name":"TEST"}} http://localhost/restconf/data/example:cont1' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "missing-element","error-info": {"bad-element": "type"},"error-severity": "error","error-message": "Mandatory variable"}}}
'
+expectfn 'curl -s -X POST http://localhost/restconf/data/example:cont1 -d {"example:interface":{"name":"TEST"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "missing-element","error-info": {"bad-element": "type"},"error-severity": "error","error-message": "Mandatory variable"}}}
'
 
 new "restconf POST interface without mandatory key"
-expectfn 'curl -s -X POST http://localhost/restconf/data/example:cont1 -d {"interface":{"type":"regular"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "missing-element","error-info": {"bad-element": "name"},"error-severity": "error","error-message": "Mandatory key"}}}
'
+expectfn 'curl -s -X POST http://localhost/restconf/data/example:cont1 -d {"example:interface":{"type":"regular"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "missing-element","error-info": {"bad-element": "name"},"error-severity": "error","error-message": "Mandatory key"}}}
'
 
 new "restconf POST interface"
 expectfn 'curl -s -X POST -d {"example:interface":{"name":"TEST","type":"eth0"}} http://localhost/restconf/data/example:cont1' 0 ""
 
-# XXX should it be example:interface?
+new "restconf POST interface without namespace"
+expectfn 'curl -s -X POST -d {"interface":{"name":"TEST2","type":"eth0"}} http://localhost/restconf/data/example:cont1' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "rpc","error-tag": "malformed-message","error-severity": "error","error-message": "Data is not prefixed with matching namespace"}}}'
+
 new "restconf POST again"
-expecteq "$(curl -s -X POST -d '{"interface":{"name":"TEST","type":"eth0"}}' http://localhost/restconf/data/example:cont1)" 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "data-exists","error-severity": "error","error-message": "Data already exists; cannot create new resource"}}}
'
+expecteq "$(curl -s -X POST -d '{"example:interface":{"name":"TEST","type":"eth0"}}' http://localhost/restconf/data/example:cont1)" 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "data-exists","error-severity": "error","error-message": "Data already exists; cannot create new resource"}}}
'
 
 new "restconf POST from top"
 expecteq "$(curl -s -X POST -d '{"example:cont1":{"interface":{"name":"TEST","type":"eth0"}}}' http://localhost/restconf/data)" 0 '{"ietf-restconf:errors" : {"error": {"error-type": "application","error-tag": "data-exists","error-severity": "error","error-message": "Data already exists; cannot create new resource"}}}
'
@@ -163,16 +167,22 @@ new "restconf PUT initial datastore again"
 expectfn 'curl -s -X PUT -d {"data":{"example:cont1":{"interface":{"name":"local0","type":"regular"}}}} http://localhost/restconf/data' 0 ""
 
 new "restconf PUT change interface"
-expectfn 'curl -s -X PUT -d {"interface":{"name":"local0","type":"atm0"}} http://localhost/restconf/data/example:cont1/interface=local0' 0 ""
+expectfn 'curl -s -X PUT -d {"example:interface":{"name":"local0","type":"atm0"}} http://localhost/restconf/data/example:cont1/interface=local0' 0 ""
 
 new "restconf GET datastore atm"
 expectfn "curl -s -X GET http://localhost/restconf/data/example:cont1" 0 '{"example:cont1": {"interface": \[{"name": "local0","type": "atm0"}\]}}'
 
 new "restconf PUT add interface"
-expectfn 'curl -s -X PUT -d {"interface":{"name":"TEST","type":"eth0"}} http://localhost/restconf/data/example:cont1/interface=TEST' 0 ""
+expectfn 'curl -s -X PUT -d {"example:interface":{"name":"TEST","type":"eth0"}} http://localhost/restconf/data/example:cont1/interface=TEST' 0 ""
 
 new "restconf PUT change key error"
-expectfn 'curl -is -X PUT -d {"interface":{"name":"ALPHA","type":"eth0"}} http://localhost/restconf/data/example:cont1/interface=TEST' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
+expectfn 'curl -is -X PUT -d {"example:interface":{"name":"ALPHA","type":"eth0"}} http://localhost/restconf/data/example:cont1/interface=TEST' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
+
+new "restconf PUT change type to eth0 (non-key sub-element to list)"
+expectfn 'curl -s -X PUT -d {"example:type":"eth0"} http://localhost/restconf/data/example:cont1/interface=local0/type' 0 ""
+
+new "restconf GET datastore eth"
+expectfn "curl -s -X GET http://localhost/restconf/data/example:cont1/interface=local0" 0 '{"example:interface": \[{"name": "local0","type": "eth0"}\]}'
 
 #--------------- json type tests
 new "restconf POST type x3"

test/test_restconf_listkey.sh

@@ -14,7 +14,6 @@ fyang=$dir/list.yang
 cat <<EOF > $cfg
 <clixon-config xmlns="http://clicon.org/config">
   <CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
-  <CLICON_YANG_DIR>/usr/local/var</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
   <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
@@ -32,13 +31,30 @@ module list{
    prefix ex;
    container c{
       list a{
-         key b;
+         key "b c";
          leaf b{
             type string;
          }
          leaf c{
             type string;
          }
+         leaf nonkey{
+            description "non-key element";
+            type string;
+         }
+         list e{
+           description "A list in a list";
+           key "f";
+           leaf f{
+             type string;
+           }
+           leaf nonkey{
+              type string;
+           }
+         }
+         leaf-list f{
+           type string;
+         }
       }
       leaf-list d{
         type string;
@@ -55,6 +71,8 @@ if [ $BE -ne 0 ]; then
     if [ $? -ne 0 ]; then
 	err
     fi
+    sudo pkill clixon_backend # to be sure
+    
     new "start backend -s init -f $cfg"
     start_backend -s init -f $cfg
 fi
@@ -69,17 +87,36 @@ new "waiting"
 wait_backend
 wait_restconf
 
-new "restconf PUT add list entry"
+new "restconf PUT add whole list entry"
-expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x -d {"list:a":{"b":"x","c":"0"}}' 0 ''
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y -d {"list:a":{"b":"x","c":"y","nonkey":"0"}}' 0 ''
 
-new "restconf PUT change regular list entry"
+new "restconf PUT add whole list entry XML"
-expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x -d {"list:a":{"b":"x","c":"z"}}' 0 ''
+expecteq "$(curl -s -X PUT -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+xml' -d '<a xmlns="urn:example:clixon"><b>xx</b><c>xy</c><nonkey>0</nonkey></a>' http://localhost/restconf/data/list:c/a=xx,xy)" 0 ''
 
-new "restconf PUT change list key entry (expect fail)"
-expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x -d {"list:a":{"b":"y"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
 
-new "restconf PUT change actual list key entry (expect fail)"
+new "restconf PUT change whole list entry (same keys)"
-expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x/b -d {"b":"y"}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y -d {"list:a":{"b":"x","c":"y","nonkey":"z"}}' 0 ''
+
+new "restconf PUT change whole list entry (no namespace)(expect fail)"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y -d {"a":{"b":"x","c":"y","nonkey":"z"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "rpc","error-tag": "malformed-message","error-severity": "error","error-message": "Data is not prefixed with matching namespace"}}}'
+
+new "restconf PUT change list entry (wrong keys)(expect fail)"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y -d {"list:a":{"b":"y","c":"x"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
+
+new "restconf PUT change list entry (wrong keys)(expect fail) XML"
+expecteq "$(curl -s -X PUT -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+xml' -d '<a xmlns="urn:example:clixon"><b>xy</b><c>xz</c><nonkey>0</nonkey></a>' http://localhost/restconf/data/list:c/a=xx,xy)" 0 '<errors xmlns="urn:ietf:params:xml:ns:yang:ietf-restconf"><error><error-type>protocol</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>api-path keys do not match data keys</error-message></error></errors>
'
+
+new "restconf PUT change list entry (just one key)(expect fail)"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x -d {"list:a":{"b":"x"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "rpc","error-tag": "malformed-message","error-severity": "error","error-message": "List key a length mismatch"}}}
'
+
+new "restconf PUT sub non-key"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/nonkey -d {"list:nonkey":"u"}' 0 ''
+
+new "restconf PUT sub key same value"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/b -d {"list:b":"x"}' 0 ''
+
+new "restconf PUT just key other value (should fail)ZX"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x/b -d {"b":"y"}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "rpc","error-tag": "malformed-message","error-severity": "error","error-message": "List key a length mismatch"}}}'
 
 new "restconf PUT add leaf-list entry"
 expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/d=x -d {"list:d":"x"}' 0 ''
@@ -87,6 +124,31 @@ expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/d=x -d {"list:d":
 new "restconf PUT change leaf-list entry (expect fail)"
 expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/d=x -d {"list:d":"y"}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
 
+new "restconf PUT list-list"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/e=z -d {"list:e":{"f":"z","nonkey":"0"}}' 0 ''
+
+new "restconf PUT change list-lst entry (wrong keys)(expect fail)"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/e=z -d {"list:e":{"f":"wrong","nonley":"0"}}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
+
+new "restconf PUT list-list sub non-key"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/e=z/nonkey -d {"list:nonkey":"u"}' 0 ''
+
+new "restconf PUT list-list single first key"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x/e=z/f -d {"f":"z"}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "rpc","error-tag": "malformed-message","error-severity": "error","error-message": "List key a length mismatch"}}}'
+
+new "restconf PUT list-list just key ok"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/e=z/f -d {"list:f":"z"}' 0 ''
+
+new "restconf PUT list-list just key just key wrong value (should fail)"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/e=z/f -d {"list:f":"wrong"}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
+
+new "restconf PUT add list+leaf-list entry"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/f=u -d {"list:f":"u"}' 0 ''
+
+new "restconf PUT change list+leaf-list entry (expect fail)"
+expectfn 'curl -s -X PUT http://localhost/restconf/data/list:c/a=x,y/f=u -d {"list:f":"w"}' 0 '{"ietf-restconf:errors" : {"error": {"error-type": "protocol","error-tag": "operation-failed","error-severity": "error","error-message": "api-path keys do not match data keys"}}}'
+
+
 new "Kill restconf daemon"
 stop_restconf 
 

test/test_rpc.sh

@@ -77,11 +77,11 @@ expecteq "$(curl -s -X POST -d '{"clixon-example:input":{"x":"0","y":"99"}}' htt
 
 new "restconf example rpc json/json"
 # XXX example:input example:output
-expecteq "$(curl -s -X POST -H 'Content-Type: application/yang-data+json' -H 'Content-Type: application/yang-data+json' -d '{"clixon-example:input":{"x":"0"}}' http://localhost/restconf/operations/clixon-example:example)" 0 '{"clixon-example:output": {"x": "0","y": "42"}}
+expecteq "$(curl -s -X POST -H 'Content-Type: application/yang-data+json' -H 'Accept: application/yang-data+json' -d '{"clixon-example:input":{"x":"0"}}' http://localhost/restconf/operations/clixon-example:example)" 0 '{"clixon-example:output": {"x": "0","y": "42"}}
 
'
 
 new "restconf example rpc xml/json"
-expecteq "$(curl -s -X POST -H 'Content-Type: application/yang-data+xml' -H 'Content-Type: application/yang-data+json' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' http://localhost/restconf/operations/clixon-example:example)"  0 '{"clixon-example:output": {"x": "0","y": "42"}}
+expecteq "$(curl -s -X POST -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+json' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' http://localhost/restconf/operations/clixon-example:example)"  0 '{"clixon-example:output": {"x": "0","y": "42"}}
 
'
 
 new "restconf example rpc json/xml"
@@ -92,6 +92,13 @@ new "restconf example rpc xml/xml"
 expecteq "$(curl -s -X POST -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+xml' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' http://localhost/restconf/operations/clixon-example:example)" 0 '<output xmlns="urn:example:clixon"><x>0</x><y>42</y></output>
 
'
 
+new "restconf example rpc xml in w json encoding (expect fail)"
+expecteq "$(curl -s -X POST -H 'Content-Type: application/yang-data+json' -H 'Accept: application/yang-data+xml' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' http://localhost/restconf/operations/clixon-example:example)" 0 "<errors xmlns=\"urn:ietf:params:xml:ns:yang:ietf-restconf\"><error><error-type>rpc</error-type><error-tag>malformed-message</error-tag><error-severity>error</error-severity><error-message> on line 1: syntax error at or before: '&lt;'</error-message></error></errors>
"
+
+
+new "restconf example rpc json in xml encoding (expect fail)"
+expecteq "$(curl -s -X POST -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+xml' -d '{"clixon-example:input":{"x":"0"}}' http://localhost/restconf/operations/clixon-example:example)" 0 '<errors xmlns="urn:ietf:params:xml:ns:yang:ietf-restconf"><error><error-type>rpc</error-type><error-tag>malformed-message</error-tag><error-severity>error</error-severity><error-message>xml_parse: line 0: syntax error: at or before: "</error-message></error></errors>
'
+
 new "netconf example rpc"
 expecteof "$clixon_netconf -qf $cfg" 0 '<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><example xmlns="urn:example:clixon"><x>0</x></example></rpc>]]>]]>' '^<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><x xmlns="urn:example:clixon">0</x><y xmlns="urn:example:clixon">42</y></rpc-reply>]]>]]>$'
 

test/test_stream.sh

@@ -39,6 +39,7 @@ cat <<EOF > $cfg
   <CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_RESTCONF_PRETTY>false</CLICON_RESTCONF_PRETTY>
   <CLICON_SOCK>/usr/local/var/$APPNAME/$APPNAME.sock</CLICON_SOCK>
   <CLICON_BACKEND_DIR>/usr/local/lib/$APPNAME/backend</CLICON_BACKEND_DIR>
@@ -101,7 +102,7 @@ cat <<EOF > $fyang
    }
 EOF
 
-new "test params: -f $cfg -y $fyang"
+new "test params: -f $cfg"
 
 if [ $BE -ne 0 ]; then
     new "kill old backend"
@@ -109,15 +110,15 @@ if [ $BE -ne 0 ]; then
     if [ $? -ne 0 ]; then
 	err
     fi
-    new "start backend -s init -f $cfg -y $fyang"
+    new "start backend -s init -f $cfg"
-    start_backend -s init -f $cfg -y $fyang
+    start_backend -s init -f $cfg
 fi
 
 new "kill old restconf daemon"
 sudo pkill -u www-data -f "/www-data/clixon_restconf"
       
 new "start restconf daemon"
-start_restconf -f $cfg -y $fyang
+start_restconf -f $cfg
 
 new "waiting"
 wait_backend
@@ -128,35 +129,35 @@ wait_restconf
 new "1. Netconf RFC5277 stream testing"
 # 1.1 Stream discovery
 new "netconf event stream discovery RFC5277 Sec 3.2.5"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get><filter type="xpath" select="netconf/streams" xmlns="urn:ietf:params:xml:ns:netmod:notification"/></get></rpc>]]>]]>' '<rpc-reply><data><netconf xmlns="urn:ietf:params:xml:ns:netmod:notification"><streams><stream><name>EXAMPLE</name><description>Example event stream</description><replay-support>true</replay-support></stream></streams></netconf></data></rpc-reply>]]>]]>'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get><filter type="xpath" select="netconf/streams" xmlns="urn:ietf:params:xml:ns:netmod:notification"/></get></rpc>]]>]]>' '<rpc-reply><data><netconf xmlns="urn:ietf:params:xml:ns:netmod:notification"><streams><stream><name>EXAMPLE</name><description>Example event stream</description><replay-support>true</replay-support></stream></streams></netconf></data></rpc-reply>]]>]]>'
 
 new "netconf event stream discovery RFC8040 Sec 6.2"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get><filter type="xpath" select="restconf-state/streams"/></get></rpc>]]>]]>' '<rpc-reply><data><restconf-state xmlns="urn:ietf:params:xml:ns:yang:ietf-restconf-monitoring"><streams><stream><name>EXAMPLE</name><description>Example event stream</description><replay-support>true</replay-support><access><encoding>xml</encoding><location>https://localhost/streams/EXAMPLE</location></access></stream></streams></restconf-state></data></rpc-reply>]]>]]>'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get><filter type="xpath" select="restconf-state/streams"/></get></rpc>]]>]]>' '<rpc-reply><data><restconf-state xmlns="urn:ietf:params:xml:ns:yang:ietf-restconf-monitoring"><streams><stream><name>EXAMPLE</name><description>Example event stream</description><replay-support>true</replay-support><access><encoding>xml</encoding><location>https://localhost/streams/EXAMPLE</location></access></stream></streams></restconf-state></data></rpc-reply>]]>]]>'
 
 #
 # 1.2 Netconf stream subscription
 new "netconf EXAMPLE subscription"
-expectwait "$clixon_netconf -qf $cfg -y $fyang" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream></create-subscription></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' $NCWAIT
+expectwait "$clixon_netconf -qf $cfg" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream></create-subscription></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' $NCWAIT
 
 new "netconf subscription with empty startTime"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream><startTime/></create-subscription></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>bad-element</error-tag><error-info><bad-element>startTime</bad-element></error-info><error-severity>error</error-severity><error-message>regexp match fail:'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream><startTime/></create-subscription></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>bad-element</error-tag><error-info><bad-element>startTime</bad-element></error-info><error-severity>error</error-severity><error-message>regexp match fail:'
 
 new "netconf EXAMPLE subscription with simple filter"
-expectwait "$clixon_netconf -qf $cfg -y $fyang" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream><filter type="xpath" select="event"/></create-subscription></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' $NCWAIT
+expectwait "$clixon_netconf -qf $cfg" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream><filter type="xpath" select="event"/></create-subscription></rpc>]]>]]>' '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' $NCWAIT
 
 new "netconf EXAMPLE subscription with filter classifier"
-expectwait "$clixon_netconf -qf $cfg -y $fyang" "<rpc><create-subscription xmlns=\"urn:ietf:params:xml:ns:netmod:notification\"><stream>EXAMPLE</stream><filter type=\"xpath\" select=\"event[event-class='fault']\"/></create-subscription></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' $NCWAIT
+expectwait "$clixon_netconf -qf $cfg" "<rpc><create-subscription xmlns=\"urn:ietf:params:xml:ns:netmod:notification\"><stream>EXAMPLE</stream><filter type=\"xpath\" select=\"event[event-class='fault']\"/></create-subscription></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' $NCWAIT
 
 new "netconf NONEXIST subscription"
-expectwait "$clixon_netconf -qf $cfg -y $fyang" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>NONEXIST</stream></create-subscription></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>invalid-value</error-tag><error-severity>error</error-severity><error-message>No such stream</error-message></rpc-error></rpc-reply>]]>]]>$' $NCWAIT
+expectwait "$clixon_netconf -qf $cfg" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>NONEXIST</stream></create-subscription></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>invalid-value</error-tag><error-severity>error</error-severity><error-message>No such stream</error-message></rpc-error></rpc-reply>]]>]]>$' $NCWAIT
 
 new "netconf EXAMPLE subscription with wrong date"
-expectwait "$clixon_netconf -qf $cfg -y $fyang" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream><startTime>kallekaka</startTime></create-subscription></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>bad-element</error-tag><error-info><bad-element>startTime</bad-element></error-info><error-severity>error</error-severity><error-message>regexp match fail:' 0
+expectwait "$clixon_netconf -qf $cfg" '<rpc><create-subscription xmlns="urn:ietf:params:xml:ns:netmod:notification"><stream>EXAMPLE</stream><startTime>kallekaka</startTime></create-subscription></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>bad-element</error-tag><error-info><bad-element>startTime</bad-element></error-info><error-severity>error</error-severity><error-message>regexp match fail:' 0
 
 #new "netconf EXAMPLE subscription with replay"
 #NOW=$(date +"%Y-%m-%dT%H:%M:%S")
 #sleep 10
-#expectwait "$clixon_netconf -qf $cfg -y $fyang" "<rpc><create-subscription xmlns=\"urn:ietf:params:xml:ns:netmod:notification\"><stream>EXAMPLE</stream><startTime>$NOW</startTime></create-subscription></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' 10
+#expectwait "$clixon_netconf -qf $cfg" "<rpc><create-subscription xmlns=\"urn:ietf:params:xml:ns:netmod:notification\"><stream>EXAMPLE</stream><startTime>$NOW</startTime></create-subscription></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]><notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0"><eventTime>20' 10
 sleep 2
 
 #

test/test_transaction.sh

@@ -87,6 +87,7 @@ checklog(){
     s=$1 # statement
     l0=$2 # linenr
     new "Check $s in log"
+#    echo "grep \"transaction_log $s\"  $flog"
     t=$(grep -n "transaction_log $s" $flog)
     if [ -z "$t" ]; then
 	echo -e "\e[31m\nError in Test$testnr [$testname]:"
@@ -124,7 +125,7 @@ if [ $BE -ne 0 ]; then
     sleep $RCWAIT
 fi
 
-let nr=1
+let nr=0
 
 new "Basic transaction to add top-level x"
 expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><edit-config><target><candidate/></target><config><x xmlns='urn:example:clixon'><y><a>$nr</a></y></x></config></edit-config></rpc>]]>]]>" '^<rpc-reply><ok/></rpc-reply>]]>]]>$'

test/test_union.sh

@@ -18,6 +18,7 @@ cat <<EOF > $cfg
   <CLICON_YANG_DIR>$dir</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
   <CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
   <CLICON_CLI_MODE>$APPNAME</CLICON_CLI_MODE>
@@ -76,7 +77,7 @@ module example{
 }
 EOF
 
-new "test params: -f $cfg -y $fyang"
+new "test params: -f $cfg"
 
 if [ $BE -ne 0 ]; then
     new "kill old backend"
@@ -84,21 +85,21 @@ if [ $BE -ne 0 ]; then
     if [ $? -ne 0 ]; then
 	err
     fi
-    new "start backend -s init -f $cfg -y $fyang"
+    new "start backend -s init -f $cfg"
-    start_backend -s init -f $cfg -y $fyang
+    start_backend -s init -f $cfg
 
     new "waiting"
     sleep $RCWAIT
 fi
 
 new "cli set transitive string"
-expectfn "$clixon_cli -1f $cfg -l o -y $fyang set c talle x" 0 "^$"
+expectfn "$clixon_cli -1f $cfg -l o set c talle x" 0 "^$"
 
 new "cli set transitive union"
-expectfn "$clixon_cli -1f $cfg -l o -y $fyang set c ulle 33" 0 "^$"
+expectfn "$clixon_cli -1f $cfg -l o set c ulle 33" 0 "^$"
 
 new "cli set transitive union error"
-expectfn "$clixon_cli -1f $cfg -l o -y $fyang set c ulle kalle" 255 '^CLI syntax error: "set c ulle kalle": Unknown command$'
+expectfn "$clixon_cli -1f $cfg -l o set c ulle kalle" 255 '^CLI syntax error: "set c ulle kalle": Unknown command$'
 
 if [ $BE -eq 0 ]; then
     exit # BE

test/test_when_must.sh

@@ -15,6 +15,7 @@ cat <<EOF > $cfg
   <CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
   <CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
   <CLICON_CLI_MODE>$APPNAME</CLICON_CLI_MODE>
@@ -89,56 +90,56 @@ module $APPNAME{
 }
 EOF
 
-new "test params: -f $cfg -y $fyang"
+new "test params: -f $cfg"
 
 if [ $BE -ne 0 ]; then
     new "kill old backend"
-    sudo clixon_backend -zf $cfg -y $fyang
+    sudo clixon_backend -zf $cfg
     if [ $? -ne 0 ]; then
 	err
     fi
-    new "start backend -s init -f $cfg -y $fyang"
+    new "start backend -s init -f $cfg"
-    start_backend -s init -f $cfg -y $fyang
+    start_backend -s init -f $cfg
 
     new "waiting"
     sleep $RCWAIT
 fi
 
 new "when: add static route"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><whenex xmlns="urn:example:clixon"><type>static</type><name>r1</name><static-routes/></whenex></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><whenex xmlns="urn:example:clixon"><type>static</type><name>r1</name><static-routes/></whenex></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "when: validate ok"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "when: add direct route"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><whenex xmlns="urn:example:clixon"><type>direct</type><name>r2</name><static-routes/></whenex></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><whenex xmlns="urn:example:clixon"><type>direct</type><name>r2</name><static-routes/></whenex></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "when get config"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>" '^<rpc-reply><data><whenex xmlns="urn:example:clixon"><type>direct</type><name>r2</name><static-routes/></whenex><whenex xmlns="urn:example:clixon"><type>static</type><name>r1</name><static-routes/></whenex></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>" '^<rpc-reply><data><whenex xmlns="urn:example:clixon"><type>direct</type><name>r2</name><static-routes/></whenex><whenex xmlns="urn:example:clixon"><type>static</type><name>r1</name><static-routes/></whenex></data></rpc-reply>]]>]]>$'
 
 new "when: validate fail"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>when xpath validation failed</error-message></rpc-error></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>when xpath validation failed</error-message></rpc-error></rpc-reply>]]>]]>$"
 
 new "when: discard-changes"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "must: add interface"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><interface xmlns="urn:example:clixon"><ifType>ethernet</ifType><ifMTU>1500</ifMTU></interface></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><interface xmlns="urn:example:clixon"><ifType>ethernet</ifType><ifMTU>1500</ifMTU></interface></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "must: validate ok"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "must: add atm interface"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><interface xmlns="urn:example:clixon"><ifType>atm</ifType><ifMTU>32</ifMTU></interface></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><interface xmlns="urn:example:clixon"><ifType>atm</ifType><ifMTU>32</ifMTU></interface></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "must: atm validate fail"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>An ATM MTU must be 64 .. 17966</error-message></rpc-error></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>An ATM MTU must be 64 .. 17966</error-message></rpc-error></rpc-reply>]]>]]>$"
 
 new "must: add eth interface"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><interface xmlns="urn:example:clixon"><ifType>ethernet</ifType><ifMTU>989</ifMTU></interface></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><interface xmlns="urn:example:clixon"><ifType>ethernet</ifType><ifMTU>989</ifMTU></interface></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "must: eth validate fail"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>An Ethernet MTU must be 1500</error-message></rpc-error></rpc-reply>]]>]]>"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><rpc-error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-severity>error</error-severity><error-message>An Ethernet MTU must be 1500</error-message></rpc-error></rpc-reply>]]>]]>"
 
 if [ $BE -eq 0 ]; then
     exit # BE

test/test_xml.sh

@@ -6,7 +6,7 @@
 # Magic line must be first in script (see README.md)
 s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
 
-: ${clixon_util_xml:=clixon_util_xml}
+: ${clixon_util_xml:=clixon_util_xml -o} # -o is output
 
 new "xml parse"
 expecteof "$clixon_util_xml" 0 "<a><b/></a>" "^<a><b/></a>$"

test/test_yang.sh

@@ -17,6 +17,7 @@ cat <<EOF > $cfg
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$dir</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>
   <CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
   <CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
   <CLICON_CLI_MODE>$APPNAME</CLICON_CLI_MODE>
@@ -141,35 +142,35 @@ module $APPNAME{
 }
 EOF
 
-new "test params: -f $cfg -y $fyang"
+new "test params: -f $cfg"
 
 if [ $BE -ne 0 ]; then
     new "kill old backend"
-    sudo clixon_backend -zf $cfg -y $fyang
+    sudo clixon_backend -zf $cfg
     if [ $? -ne 0 ]; then
 	err
     fi
-    new "start backend -s init -f $cfg -y $fyang"
+    new "start backend -s init -f $cfg"
-    start_backend -s init -f $cfg -y $fyang
+    start_backend -s init -f $cfg
 
     new "waiting"
     wait_backend
 fi
 
 new "cli defined extension"
-expectfn "$clixon_cli -1f $cfg -y $fyang show version" 0 "3."
+expectfn "$clixon_cli -1f $cfg show version" 0 "3."
 
 new "empty values in leaf-list"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><f><e>a</e></f></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><f><e>a</e></f></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "empty values in leaf-list2"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><f><e/></f></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><f><e/></f></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf get config"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>" '^<rpc-reply><data><x xmlns="urn:example:clixon"><f><e/><e>a</e></f></x></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>" '^<rpc-reply><data><x xmlns="urn:example:clixon"><f><e/><e>a</e></f></x></data></rpc-reply>]]>]]>$'
 
 new "netconf discard-changes"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 
 #new "cli not defined extension"
@@ -179,106 +180,106 @@ expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><discard-changes/></rpc>]
 #expectfn "$clixon_cli -1f $cfg -y $fyangerr show version" 0 "Yang error: Extension ex:not-defined not found"
 
 new "netconf schema resource, RFC 7895"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get><filter type="xpath" select="modules-state/module" xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library"/></get></rpc>]]>]]>' '<module><name>ietf-yang-types</name><revision>2013-07-15</revision><namespace>urn:ietf:params:xml:ns:yang:ietf-yang-types</namespace><conformance-type>implement</conformance-type></module>'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get><filter type="xpath" select="modules-state/module" xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library"/></get></rpc>]]>]]>' '<module><name>ietf-yang-types</name><revision>2013-07-15</revision><namespace>urn:ietf:params:xml:ns:yang:ietf-yang-types</namespace><conformance-type>implement</conformance-type></module>'
 
 new "netconf edit config"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>5</c><val>one</val></y><d/></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>5</c><val>one</val></y><d/></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf commit"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 # text empty type in running
 new "netconf commit 2nd"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf get config xpath"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><candidate/></source><filter type=\"xpath\" select=\"/x/y[a=1][b=2][c=5]\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>5</c><val>one</val></y></x></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><candidate/></source><filter type=\"xpath\" select=\"/x/y[a=1][b=2][c=5]\"/></get-config></rpc>]]>]]>" '^<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>5</c><val>one</val></y></x></data></rpc-reply>]]>]]>$'
 
 new "netconf edit leaf-list"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><f><e>hej</e><e>hopp</e></f></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><f><e>hej</e><e>hopp</e></f></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf get leaf-list"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/x/f/e"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><x xmlns="urn:example:clixon"><f><e>hej</e><e>hopp</e></f></x></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/x/f/e"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><x xmlns="urn:example:clixon"><f><e>hej</e><e>hopp</e></f></x></data></rpc-reply>]]>]]>$'
 
 new "netconf get leaf-list path"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get-config><source><candidate/></source><filter type=\"xpath\" select=\"/x/f[e='hej']\"/></get-config></rpc>]]>]]>" "^<rpc-reply><data><x xmlns=\"urn:example:clixon\"><f><e>hej</e><e>hopp</e></f></x></data></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get-config><source><candidate/></source><filter type=\"xpath\" select=\"/x/f[e='hej']\"/></get-config></rpc>]]>]]>" "^<rpc-reply><data><x xmlns=\"urn:example:clixon\"><f><e>hej</e><e>hopp</e></f></x></data></rpc-reply>]]>]]>$"
 
 new "netconf get (should be some)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><get><filter type=\"xpath\" select=\"/\"/></get></rpc>]]>]]>" '^<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>5</c><val>one</val></y><d/></x>'
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><get><filter type=\"xpath\" select=\"/\"/></get></rpc>]]>]]>" '^<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>5</c><val>one</val></y><d/></x>'
 
 new "cli set leaf-list"
-expectfn "$clixon_cli -1f $cfg -y $fyang set x f e foo" 0 ""
+expectfn "$clixon_cli -1f $cfg set x f e foo" 0 ""
 
 new "cli show leaf-list"
-expectfn "$clixon_cli -1f $cfg -y $fyang show xpath /x/f/e" 0 "<e>foo</e>"
+expectfn "$clixon_cli -1f $cfg show xpath /x/f/e" 0 "<e>foo</e>"
 
 new "netconf set state data (not allowed)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><state xmlns="urn:example:clixon"><op>42</op></state></config></edit-config></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>protocol</error-type><error-tag>invalid-value</error-tag><error-severity>error</error-severity><error-message>State data not allowed</error-message></rpc-error></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><state xmlns="urn:example:clixon"><op>42</op></state></config></edit-config></rpc>]]>]]>' '^<rpc-reply><rpc-error><error-type>protocol</error-type><error-tag>invalid-value</error-tag><error-severity>error</error-severity><error-message>State data not allowed</error-message></rpc-error></rpc-reply>]]>]]>$'
 
 new "netconf set presence and not present"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><nopresence/><presence/></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><nopresence/><presence/></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf get presence only"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/x/presence"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><x xmlns="urn:example:clixon"><presence/></x></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/x/presence"/></get-config></rpc>]]>]]>' '^<rpc-reply><data><x xmlns="urn:example:clixon"><presence/></x></data></rpc-reply>]]>]]>$'
 
 new "netconf get presence only"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/x/nopresence"/></get-config></rpc>]]>]]>' "^<rpc-reply><data/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source><filter type="xpath" select="/x/nopresence"/></get-config></rpc>]]>]]>' "^<rpc-reply><data/></rpc-reply>]]>]]>$"
 
 new "netconf discard-changes"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf anyxml"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><any><foo><bar a="nisse"/></foo></any></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><any><foo><bar a="nisse"/></foo></any></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf validate anyxml"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf delete candidate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><default-operation>none</default-operation><config operation="delete"/></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><default-operation>none</default-operation><config operation="delete"/></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 # Check 3-keys
 new "netconf add one 3-key entry"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>one</val></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>one</val></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf check add one 3-key"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>one</val></y></x></data></rpc-reply>]]>]]>'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>one</val></y></x></data></rpc-reply>]]>]]>'
 
 new "netconf add another (with same 1st key)"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf check add another"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>one</val></y><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></data></rpc-reply>]]>]]>'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>one</val></y><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></data></rpc-reply>]]>]]>'
 
 new "netconf replace first"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>replace</val></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>replace</val></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf check replace"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>replace</val></y><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></data></rpc-reply>]]>]]>'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>1</b><c>1</c><val>replace</val></y><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></data></rpc-reply>]]>]]>'
 
 new "netconf delete first"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y operation="remove"><a>1</a><b>1</b><c>1</c></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><x xmlns="urn:example:clixon"><y operation="remove"><a>1</a><b>1</b><c>1</c></y></x></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf check delete"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></data></rpc-reply>]]>]]>'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '<rpc-reply><data><x xmlns="urn:example:clixon"><y><a>1</a><b>2</b><c>1</c><val>two</val></y></x></data></rpc-reply>]]>]]>'
 
 # clear db for next test
 new "netconf delete candidate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><default-operation>none</default-operation><config operation="delete"/></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><default-operation>none</default-operation><config operation="delete"/></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf commit empty candidate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><commit/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconfig config submodule"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><edit-config><target><candidate/></target><config><mylist xmlns="urn:example:clixon"><x>a</x><subm-container><subm-leaf>foo</subm-leaf></subm-container></mylist></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><edit-config><target><candidate/></target><config><mylist xmlns="urn:example:clixon"><x>a</x><subm-container><subm-leaf>foo</subm-leaf></subm-container></mylist></config></edit-config></rpc>]]>]]>' "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf submodule get config"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><mylist xmlns="urn:example:clixon"><x>a</x><subm-container><subm-leaf>foo</subm-leaf></subm-container></mylist></data></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 '<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>' '^<rpc-reply><data><mylist xmlns="urn:example:clixon"><x>a</x><subm-container><subm-leaf>foo</subm-leaf></subm-container></mylist></data></rpc-reply>]]>]]>$'
 
 new "netconf submodule validate"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 new "netconf submodule discard-changes"
-expecteof "$clixon_netconf -qf $cfg -y $fyang" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
+expecteof "$clixon_netconf -qf $cfg" 0 "<rpc><discard-changes/></rpc>]]>]]>" "^<rpc-reply><ok/></rpc-reply>]]>]]>$"
 
 if [ $BE -eq 0 ]; then
     exit # BE

util/clixon_util_json.c

@@ -59,7 +59,7 @@
 #include "clixon/clixon.h"
 
 /*
- * Turn this on to get a json parse and pretty print test program
+ * JSON parse and pretty print test program
  * Usage: xpath
  * read json from input
  * Example compile:

util/clixon_util_xml.c

@@ -34,6 +34,11 @@
  * XML support functions.
  * @see https://www.w3.org/TR/2008/REC-xml-20081126
  *      https://www.w3.org/TR/2009/REC-xml-names-20091208
+ * The function can do yang validation, process xml and json, etc.
+ * On success, nothing is printed and exitcode 0
+ * On failure, an error is printed on stderr and exitcode != 0
+ * Failure error prints are different, it would be nice to make them more
+ * uniform. (see clicon_rpc_generate_error)
  */
 
 #ifdef HAVE_CONFIG_H
@@ -49,7 +54,9 @@
 #include <stdint.h>
 #include <syslog.h>
 #include <assert.h>
+#include <fcntl.h>
 #include <sys/time.h>
+#include <sys/stat.h>
 
 /* cligen */
 #include <cligen/cligen.h>
@@ -69,12 +76,20 @@
 static int
 usage(char *argv0)
 {
-    fprintf(stderr, "usage:%s [options]\n"
+    fprintf(stderr, "usage:%s [options] with xml on stdin\n"
 	    "where options are\n"
             "\t-h \t\tHelp\n"
     	    "\t-D <level> \tDebug\n"
+	    "\t-f <file>\tXML input file (overrides stdin)\n"
+	    "\t-J \t\tInput as JSON\n"
 	    "\t-j \t\tOutput as JSON\n"
-	    "\t-l <s|e|o> \tLog on (s)yslog, std(e)rr, std(o)ut (stderr is default)\n",
+	    "\t-l <s|e|o> \tLog on (s)yslog, std(e)rr, std(o)ut (stderr is default)\n"
+	    "\t-o \t\tOutput the file\n"
+	    "\t-v \t\tValidate the result in terms of Yang model (requires -y)\n"
+	    "\t-p \t\tPretty-print output\n"
+	    "\t-y <filename> \tYang filename or dir (load all files)\n"
+    	    "\t-Y <dir> \tYang dirs (can be several)\n"
+	    ,
 	    argv0);
     exit(0);
 }
@@ -83,17 +98,39 @@ int
 main(int    argc,
      char **argv)
 {
-    int   retval = -1;
+    int           retval = -1;
-    cxobj *xt = NULL;
+    cxobj        *xt = NULL;
-    cxobj *xc;
+    cxobj        *xc;
-    cbuf  *cb = cbuf_new();
+    cbuf         *cb = cbuf_new();
-    int   c;
+    int           c;
-    int   logdst = CLICON_LOG_STDERR;
+    int           logdst = CLICON_LOG_STDERR;
-    int   json = 0;
+    int           jsonin = 0;
+    int           jsonout = 0;
+    char         *input_filename = NULL;
+    char         *yang_file_dir = NULL;
+    yang_stmt    *yspec = NULL;
+    cxobj        *xerr = NULL; /* malloced must be freed */
+    int           ret;
+    int           pretty = 0;
+    int           validate = 0;
+    int           output = 0;
+    clicon_handle h;
+    struct stat   st;
+    int           fd = 0; /* stdin */
+    cxobj        *xcfg = NULL;
+    cbuf         *cbret = NULL;
 
+    /* In the startup, logs to stderr & debug flag set later */
+    clicon_log_init(__FILE__, LOG_INFO, CLICON_LOG_STDERR); 
+
+    if ((h = clicon_handle_init()) == NULL)
+	goto done;
+
+    xcfg = xml_new("clixon-config", NULL, NULL);
+    clicon_conf_xml_set(h, xcfg);
     optind = 1;
     opterr = 0;
-    while ((c = getopt(argc, argv, "hD:jl:")) != -1)
+    while ((c = getopt(argc, argv, "hD:f:Jjl:pvoy:Y:")) != -1)
 	switch (c) {
 	case 'h':
 	    usage(argv[0]);
@@ -102,37 +139,131 @@ main(int    argc,
 	    if (sscanf(optarg, "%d", &debug) != 1)
 		usage(argv[0]);
 	    break;
+	case 'f':
+	    input_filename = optarg;
+	    break;
+	case 'J':
+	    jsonin++;
+	    break;
 	case 'j':
-	    json++;
+	    jsonout++;
 	    break;
 	case 'l': /* Log destination: s|e|o|f */
 	    if ((logdst = clicon_log_opt(optarg[0])) < 0)
 		usage(argv[0]);
 	    break;
+	case 'o':
+	    output++;
+	    break;
+	case 'v':
+	    validate++;
+	    break;
+	case 'p':
+	    pretty++;
+	    break;
+	case 'y':
+	    yang_file_dir = optarg;
+	    break;
+	case 'Y':
+	    if (clicon_option_add(h, "CLICON_YANG_DIR", optarg) < 0)
+		goto done;
+	    break;
 	default:
 	    usage(argv[0]);
 	    break;
 	}
-    clicon_log_init(__FILE__, debug?LOG_DEBUG:LOG_INFO, logdst);
+    if (validate && !yang_file_dir){
-    if (xml_parse_file(0, "</config>", NULL, &xt) < 0){
+	fprintf(stderr, "-v requires -y\n");
-	fprintf(stderr, "xml parse error %s\n", clicon_err_reason);
+	usage(argv[0]);
-	goto done;
+    }
+    clicon_log_init(__FILE__, debug?LOG_DEBUG:LOG_INFO, logdst);
+    /* 1. Parse yang */
+    if (yang_file_dir){
+	if ((yspec = yspec_new()) == NULL)
+	    goto done;
+	if (stat(yang_file_dir, &st) < 0){
+	    clicon_err(OE_YANG, errno, "%s not found", yang_file_dir);
+	    goto done;
+	}
+	if (S_ISDIR(st.st_mode)){
+	    if (yang_spec_load_dir(h, yang_file_dir, yspec) < 0)
+		goto done;
+	}
+	else{
+	    if (yang_spec_parse_file(h, yang_file_dir, yspec) < 0)
+		goto done;
+	}
+    }
+    if (input_filename){
+	if ((fd = open(input_filename, O_RDONLY)) < 0){
+	    clicon_err(OE_YANG, errno, "open(%s)", input_filename);	
+	    goto done;
+	}
+    }
+    /* 2. Parse data (xml/json) */
+    if (jsonin){
+	if ((ret = json_parse_file(fd, yspec, &xt, &xerr)) < 0)
+	    goto done;
+	if (ret == 0){
+	    clicon_rpc_generate_error("util_xml", xerr);
+	    goto done;
+	}
+    }
+    else{
+	if (xml_parse_file(fd, "</config>", NULL, &xt) < 0){
+	    fprintf(stderr, "xml parse error: %s\n", clicon_err_reason);
+	    goto done;
+	}
+    }
+
+    /* Dump data structures (for debug) */
+    if (debug){
+	cbuf_reset(cb);
+	xmltree2cbuf(cb, xt, 0);       
+	fprintf(stderr, "%s\n", cbuf_get(cb));
+	cbuf_reset(cb);
+    }
+
+    /* 3. Validate data (if yspec) */
+    if (validate){
+	xc = xml_child_i(xt, 0);
+	/* Populate */
+	if (xml_apply0(xc, CX_ELMNT, xml_spec_populate, yspec) < 0)
+	    goto done;
+	/* Sort */
+	if (xml_apply0(xc, CX_ELMNT, xml_sort, h) < 0)
+	    goto done;
+	/* Add default values */
+	if (xml_apply(xc, CX_ELMNT, xml_default, h) < 0)
+	    goto done;
+	if (xml_apply0(xc, -1, xml_sort_verify, h) < 0)
+	    clicon_log(LOG_NOTICE, "%s: sort verify failed", __FUNCTION__);
+	if ((ret = xml_yang_validate_all_top(h, xc, &xerr)) < 0) 
+	    goto done;
+	if (ret > 0 && (ret = xml_yang_validate_add(h, xc, &xerr)) < 0)
+	    goto done;
+	if (ret == 0){
+	    if (netconf_err2cb(xerr, &cbret) < 0)
+		goto done;
+	    fprintf(stderr, "xml validation error: %s\n", cbuf_get(cbret));
+	    goto done;
+	}
+    }
+    /* 4. Output data (xml/json) */
+    if (output){
+	xc = NULL;
+	while ((xc = xml_child_each(xt, xc, -1)) != NULL) 
+	    if (jsonout)
+		xml2json_cbuf(cb, xc, pretty); /* print xml */
+	    else
+		clicon_xml2cbuf(cb, xc, 0, pretty); /* print xml */
+	fprintf(stdout, "%s", cbuf_get(cb));
+	fflush(stdout);
     }
-    xc = NULL;
-    while ((xc = xml_child_each(xt, xc, -1)) != NULL) 
-	if (json)
-	    xml2json_cbuf(cb, xc, 0); /* print xml */
-	else
-	    clicon_xml2cbuf(cb, xc, 0, 0); /* print xml */
-    fprintf(stdout, "%s", cbuf_get(cb));
-    fflush(stdout);
-#if 0
-    cbuf_reset(cb);
-    xmltree2cbuf(cb, xt, 0);       /* dump data structures */
-    fprintf(stderr, "%s\n", cbuf_get(cb));
-#endif
     retval = 0;
  done:
+    if (cbret)
+	cbuf_free(cbret);
     if (xt)
 	xml_free(xt);
     if (cb)

yang/Makefile.in

@@ -39,8 +39,6 @@ includedir	= @includedir@
 datarootdir     = @datarootdir@
 enable_stdyangs	= @enable_stdyangs@
 
-CLIXON_DATADIR   = @CLIXON_DATADIR@
-
 SUBDIRS  = clixon
 # See configure.ac
 ifeq ($(enable_stdyangs),yes)

yang/clixon/Makefile.in

@@ -38,10 +38,11 @@ bindir 		= @bindir@
 includedir	= @includedir@
 datarootdir     = @datarootdir@
 
-CLIXON_DATADIR   = @CLIXON_DATADIR@
+# See also STD_YANG_INSTALLDIR for the standard yang files
+YANG_INSTALLDIR   = @YANG_INSTALLDIR@
 
-YANGSPECS	 = clixon-config@2019-03-05.yang
+YANGSPECS	 = clixon-config@2019-06-05.yang
-YANGSPECS	+= clixon-lib@2019-01-02.yang
+YANGSPECS	+= clixon-lib@2019-06-05.yang
 YANGSPECS	+= clixon-rfc5277@2008-07-01.yang
 YANGSPECS	+= clixon-xml-changelog@2019-03-21.yang
 
@@ -55,11 +56,11 @@ distclean: clean
 	rm -f Makefile *~ .depend
 
 install: $(YANGSPECS) 
-	install -d -m 0755 $(DESTDIR)$(CLIXON_DATADIR)
+	install -d -m 0755 $(DESTDIR)$(YANG_INSTALLDIR)
-	install -m 0644 $(YANGSPECS) $(DESTDIR)$(CLIXON_DATADIR)
+	install -m 0644 $(YANGSPECS) $(DESTDIR)$(YANG_INSTALLDIR)
 
 uninstall:
-	(cd $(DESTDIR)$(CLIXON_DATADIR); rm -rf *.yang)
+	(cd $(DESTDIR)$(YANG_INSTALLDIR); rm -rf *.yang)
 
 install-include:	
 

yang/clixon/clixon-config@2019-06-05.yang

@@ -39,7 +39,15 @@ module clixon-config {
 
        ***** END LICENSE BLOCK *****";
 
-    revision 2019-03-05 {
+    revision 2019-06-05 {
+	description
+	    "Added: CLICON_YANG_REGEXP, CLICON_CLI_TAB_MODE, 
+                    CLICON_CLI_HIST_FILE, CLICON_CLI_HIST_SIZE, 
+                    CLICON_XML_CHANGELOG, CLICON_XML_CHANGELOG_FILE;
+             Renamed CLICON_XMLDB_CACHE to CLICON_DATASTORE_CACHE (changed type)
+             Deleted: CLICON_XMLDB_PLUGIN, CLICON_USE_STARTUP_CONFIG";
+    }
+    revision 2019-03-05{ 
 	description
 	    "Changed URN. Changed top-level symbol to clixon-config.
              Released in Clixon 3.10";
@@ -197,7 +205,7 @@ module clixon-config {
 		"Yang directory path for finding module and submodule files. 
                  A list of these options should be in the configuration. 
                  When loading a Yang module, Clixon searches this list in the order
-                 they appear. Ensure that CLIXON_DATADIR(default 
+                 they appear. Ensure that YANG_INSTALLDIR(default 
                  /usr/local/share/clixon) is present in the path";
 	}
 	leaf CLICON_YANG_MAIN_FILE {
@@ -427,14 +435,6 @@ module clixon-config {
 	    description
 		"Directory where \"running\", \"candidate\" and \"startup\" are placed.";
 	}
-	leaf CLICON_XMLDB_PLUGIN {
-	    type string;
-	    status obsolete;
-	    description
-		"XMLDB datastore plugin filename 
-	         (see datastore/ and clixon_xml_db.[ch])
-	         Obsolete: Merged with libclixon in 3.10";
-	}
 	leaf CLICON_DATASTORE_CACHE {
 	    type datastore_cache;
 	    default cache;

yang/clixon/clixon-lib@2019-06-05.yang

@@ -40,6 +40,10 @@ module clixon-lib {
 
        ***** END LICENSE BLOCK *****";
 
+    revision 2019-06-05 {
+	description
+	    "ping rpc added for liveness";
+    }
     revision 2019-01-02 {
 	description
 	    "Released in Clixon 3.9";
@@ -53,6 +57,6 @@ module clixon-lib {
 	}
     }
     rpc ping {
-	description "Check aliveness of backend daemon.";
+        description "Check aliveness of backend daemon.";
     }
 }

yang/standard/Makefile.in

@@ -38,8 +38,8 @@ bindir 		= @bindir@
 includedir	= @includedir@
 datarootdir     = @datarootdir@
 
-# Could place them in separate standards dir?
+# See also YANG_INSTALLDIR for the clixon-specific yang files
-CLIXON_DATADIR   = @CLIXON_DATADIR@
+STD_YANG_INSTALLDIR   = @STD_YANG_INSTALLDIR@
 
 YANGSPECS  = iana-if-type@2014-05-08.yang
 YANGSPECS += ietf-interfaces@2018-02-20.yang
@@ -61,11 +61,11 @@ distclean: clean
 	rm -f Makefile *~ .depend
 
 install: $(YANGSPECS) 
-	install -d -m 0755 $(DESTDIR)$(CLIXON_DATADIR)
+	install -d -m 0755 $(DESTDIR)$(STD_YANG_INSTALLDIR)
-	install -m 0644 $(YANGSPECS) $(DESTDIR)$(CLIXON_DATADIR)
+	install -m 0644 $(YANGSPECS) $(DESTDIR)$(STD_YANG_INSTALLDIR)
 
 uninstall:
-	(cd $(DESTDIR)$(CLIXON_DATADIR); rm -rf *.yang)
+	(cd $(DESTDIR)$(STD_YANG_INSTALLDIR); rm -rf *.yang)
 
 install-include: