Merge branch 'master' into filter-and-xml-encoding-fixes

CHANGELOG.md

@@ -1,6 +1,7 @@
 # Clixon Changelog
 
-* [5.2.0](#520) Expected: June 2021
+* [5.3.0](#530) Expected: September 2021
+* [5.2.0](#520) 1 July 2021
 * [5.1.0](#510) 15 April 2021
 * [5.0.0](#500) 27 February 2021
   * [5.0.1](#501) 10 March 2021
@@ -29,46 +30,90 @@
 * [3.3.2](#332) Aug 27 2017
 * [3.3.1](#331) June 7 2017
 
-## 5.2.0
+## 5.3.0
-Expected: June 2021
+Expected: September, 2021
 
 ### New features
 
-* New utility: clixon_util_validate for stand-alone application that validates or commits datastores
+* Restconf YANG PATCH according to RFC 8072
-* Restconf native HTTP/2 support using nghttp2
+  * Experimental: enable by setting YANG_PATCH in include/clixon_custom.h
-  * Enable using: `--with-restconf=native --enable-nghttp2`
+  * Thanks to Alan Yaniger for providing this patch
-  * FCGI/nginx not affected only for `--with-restconf=native`
-  * HTTP/1 co-exists, unless `--disable-evhtp` which results in http/2 only
-  * Upgrade from HTTP/1.1 to HTTP/2
-    * https: ALPN upgrade
-    * http: Upgrade header
-* YANG when statement in conjunction with grouping/uses/augment
-  * Several cases were not implemented fully according to RFC 7950:
-    * Do not extend default values if when statements evaluate to false
-    * Do not allow edit-config of nodes if when statements evaluate to false (Sec 8.3.2)
-    * If a key leaf is defined in a grouping that is used in a list, the "uses" statement MUST NOT have a "when" statement. (See 7.21.5)
-  * See [yang uses's substatement when has no effect #218](https://github.com/clicon/clixon/issues/218)
-* YANG deviation [deviation statement not yet support #211](https://github.com/clicon/clixon/issues/211)
-  * See RFC7950 Sec 5.6.3
 
 ### API changes on existing protocol/config features
 
 Users may have to change how they access the system
 
+* Native Restconf is now default, not fcgi/nginx
+  * That is, to configure with fcgi, you need to explicitly configure: `--with-restconf=fcgi`
+* New clixon-config@2021-07-11.yang revision
+   * Removed default of `CLICON_RESTCONF_INSTALLDIR`
+     * The default behaviour is changed to use the config $(sbindir) to locate `clixon_restconf` when starting restconf internally
+
+### Corrected Bugs
+
+* Fixed: SEGV in clixon_netconf_lib functions from internal errors including validation.
+  * Check xerr argument both before and after call on netconf lib functions
+* Fixed: RFC 8040 yang-data extension allows non-key lists
+  * Added YANG_FLAG_NOKEY as exception to mandatory key lists
+* Fixed: mandatory leaf in a uses statement caused abort
+  * Occurence was in ietf-yang-patch.yang
+* Native RESTCONF fixes for http/1 or http/2 only modes
+  * Memleak in http/1-only
+  * Exit if http/1 request sent to http/2-only (bad client magic)
+  * Hang if http/1 TLS request sent to http/2 only (alpn accepted http/1.1)
+* Fixed: [RESTConf GET for a specific list instance retrieves data from other submodules that have same list name and key value #244](https://github.com/clicon/clixon/issues/244)
+
+## 5.2.0
+1 July 2021
+
+The 5.2 release has YANG support for "deviation", "when" and statement ordering. The native restconf mode also supports http/2 using libnghttp2
+
+### New features
+
+* Restconf native HTTP/2 support using nghttp2
+  * FCGI/nginx not affected only for `--with-restconf=native`
+  * HTTP/1 co-exists, unless `--disable-evhtp` which results in http/2 only
+  * For HTTP/2 only: `--disable-nghttp2`
+  * Upgrade from HTTP/1.1 to HTTP/2
+    * https: ALPN upgrade
+    * http: Upgrade header (using: `HTTP/1.1 101 Switching Protocols`)
+* Full support of YANG `when` statement in conjunction with grouping/uses/augment
+  * The following cases are now supported according to RFC 7950:
+    * Do not extend default values if when statements evaluate to false
+    * Do not allow edit-config of nodes if when statements evaluate to false (Sec 8.3.2)
+    * If a key leaf is defined in a grouping that is used in a list, the "uses" statement MUST NOT have a "when" statement. (See 7.21.5)
+  * See [yang uses's substatement when has no effect #218](https://github.com/clicon/clixon/issues/218)
+* YANG `deviation` support [deviation statement not yet support #211](https://github.com/clicon/clixon/issues/211)
+  * See RFC7950 Sec 5.6.3
+* Added ordering sanity check for YANG modules and sub-modules
+  * If YANG sub-statements are placed in wrong order, clixon fails with error.
+* New utility: clixon_util_validate for stand-alone application that validates or commits datastores
+
+### API changes on existing protocol/config features
+
+Users may have to change how they access the system
+
+* Netconf message-id attribute changed from optional to mandatory
+  * Example:
+    * Correct: `<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="42">`
+    * Wrong: `<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">`
+  * Set `CLICON_NETCONF_MESSAGE_ID_OPTIONAL` to `true` to accept omission of message-id attribute
+  * See also [need make sure message-id exist in rpc validate #240](https://github.com/clicon/clixon/issues/240)
 * Changed config and install options for Restconf
-  * clixon_restconf daemon is installed in /usr/local/sbin (as clixon_backend), instead of /www-data
+  * clixon_restconf daemon is installed in `/usr/local/sbin` (as clixon_backend), instead of /www-data
     * `configure --with-wwwdir=<dir>` remains but only applies to fcgi socket and log
     * New option `CLICON_RESTCONF_INSTALLDIR` is set to where clixon_restconf is installed, with default `/usr/local/sbin/` 
   * Restconf drop privileges user is defined by `CLICON_RESTCONF_USER`
     * `configure --with-wwwuser=<user>` is removed
   * clixon_restconf drop of privileges is defined by `CLICON_RESTCONF_PRIVILEGES` option
-* New clixon-config@2020-05-20.yang revision
+* New clixon-config@2021-05-20.yang revision
   * Added: `CLICON_RESTCONF_USER`
   * Added: `CLICON_RESTCONF_PRIVILEGES`
   * Added: `CLICON_RESTCONF_INSTALLDIR`
-  * Added: `CLICON_RESTCONF_STARTUP_DONTUPDATE`	
+  * Added: `CLICON_RESTCONF_STARTUP_DONTUPDATE`
+  * Added: `CLICON_NETCONF_MESSAGE_ID_OPTIONAL`
 * New clixon-restconf@2020-05-20.yang revision
-  * Added: restconf `log-destination` 
+  * Added: restconf `log-destination` (syslog or file:`/var/log/clixon_restconf.log`)
 * RESTCONF error replies have changed
   * Added Restconf-style xml/json message bodies everywhere
     * Clixon removed the message body from many errors in the 4.6 version since they used html encoding. 
@@ -82,37 +127,42 @@ Users may have to change how they access the system
 
 Developers may need to change their code
 
-* Made backend transaction and commit/validate API available to plugin code.
-  * This enables that RPOC handles can call commit and validate via lib
-  * The commit/validate API is now: `candidate_validate()` and `candidate_commit()`
 * Event exit API changed to a single decrementing counter where 1 means exit.
   * Removed: `clicon_exit_reset()`
   * Changed: `clicon_exit_set()` --> `clixon_exit_set(int nr)`
   * Changed: `clicon_exit_get()` --> `clixon_exit_get()`
+* Made backend transaction and commit/validate API available to plugin code.
+  * This enables RPC plugin code can call commit and validate via lib
+  * The commit/validate API is now: `candidate_validate()` and `candidate_commit()`
 
 ### Minor features
 
-* Changed default CI to be restconf=native instead of fcgi
+* CI testing:
-* Moved CI from travis to github actions
+  * Changed default CI to be Ǹative restconf` instead of fcgi using nginx
-* Added autotool check for getresuid (+ related functions) necessary for lowering of priviliges for backend and restconf
+  * Moved CI from travis to github actions
-  * If getresuid is not available, CLICON_RESTCONF_PRIVILEGES must be set to 'none'
+* Added autotool check for `getresuid` (+ related functions) necessary for lowering of priviliges for backend and restconf
+  * If `getresuid` is not available, `CLICON_RESTCONF_PRIVILEGES` must be 'none'
 * Added new startup-mode: `running-startup`: First try running db, if it is empty try startup db.
   * See [Can startup mode to be extended to support running-startup mode? #234](https://github.com/clicon/clixon/issues/234)
 * Restconf: added inline configuration using `-R <xml>` command line as an alternative to making advanced restconf configuration
-* [Need an option to disable restconf mandatory action of overwriting startup_db #230](https://github.com/clicon/clixon/issues/230)
+* New option `CLICON_RESTCONF_STARTUP_DONTUPDATE` added to disable RFC 8040 mandatory copy of running to startup after commit* 
-  * Configure option `CLICON_RESTCONF_STARTUP_DONTUPDATE` added to disable RFC 8040 mandatory copy of running to startup after commit
+  * See [Need an option to disable restconf mandatory action of overwriting startup_db #230](https://github.com/clicon/clixon/issues/230)
 * Add default network namespace constant: `RESTCONF_NETNS_DEFAULT` with default value "default".
 * CLI: Two new hide variables added (thanks: shmuelnatan)
   * hide-database : specifies that a command is not visible in database. This can be useful for setting passwords and not exposing them to users.
-	* hide-database-auto-completion : specifies that a command is not visible in database and in auto completion. This can be useful for a password that was put in device by super user, not be changed.
+  * hide-database-auto-completion : specifies that a command is not visible in database and in auto completion. This can be useful for a password that was put in device by super user, not be changed.
 	
 ### Corrected Bugs
 
+* Fixed: [uses oc-if:interface-ref error with openconfig #233](https://github.com/clicon/clixon/issues/233)
+* Fixed: [need make sure message-id exist in rpc validate #240](https://github.com/clicon/clixon/issues/240)
+  * Netconf message-id attribute changed from optional to mandatory (see API changes)
+* Fixed: [restconf patch method unable to change value to empty string #229](https://github.com/clicon/clixon/issues/229)
 * Fixed: [restconf patch method adds redundant namespaces #235](https://github.com/clicon/clixon/issues/235)
 * Fixed: Restconf HEAD did not work everywhere GET did, such as well-known and exact root.
 * Fixed: [JSON parsing error for a specific input. #236](https://github.com/clicon/clixon/issues/236)
   * JSON empty list parse problems, eg `a:[]`
-  * May also have fixed: [Json parser not work properly with empry array \[\] #228](https://github.com/clicon/clixon/issues/228)
+  * Also fixed: [Json parser not work properly with empty array \[\] #228](https://github.com/clicon/clixon/issues/228)
 * Fixed: [restconf patch method unable to chage value to empty string #229](https://github.com/clicon/clixon/issues/229)
 * Fixed: [when condition error under augment in restconf #227](https://github.com/clicon/clixon/issues/227)
 * Fixed: [Using YANG union with decimal64 and string leads to regexp match fail #226](https://github.com/clicon/clixon/issues/226)
@@ -122,8 +172,8 @@ Developers may need to change their code
   * See [XPATH issues #219](https://github.com/clicon/clixon/issues/219)
 * Fix Union in xpath [XPATH issues #219](https://github.com/clicon/clixon/issues/219)
 * Fix: XPath:s used in netconf (eg get-config) did not correctly access default values
-* [RESTCONF GET request of single-key list with empty string returns all elements #213](https://github.com/clicon/clixon/issues/213)
+* Fixed: [RESTCONF GET request of single-key list with empty string returns all elements #213](https://github.com/clicon/clixon/issues/213)
-* [RESTCONF GETof lists with empty string keys does not work #214](https://github.com/clicon/clixon/issues/214)
+* Fixed: [RESTCONF GETof lists with empty string keys does not work #214](https://github.com/clicon/clixon/issues/214)
 * Fixed: [Multiple http requests in native restconf yields same reply #212](https://github.com/clicon/clixon/issues/212)
 
 ## 5.1.0

README.md

@@ -8,15 +8,15 @@ Clixon is a YANG-based configuration manager, with interactive CLI,
 NETCONF and RESTCONF interfaces, an embedded database and transaction
 mechanism.
 
-See [documentation](https://clixon-docs.readthedocs.io), [project page](https://www.clicon.org) and [examples](https://github.com/clicon/clixon-examples), [Travis-CI](https://travis-ci.org/clicon/clixon)
+See [documentation](https://clixon-docs.readthedocs.io), [project page](https://www.clicon.org) and [examples](https://github.com/clicon/clixon-examples), [Github actions CI](https://github.com/clicon/clixon/actions/workflows/ci.yml)
   
 Clixon is open-source and dual licensed. Either Apache License, Version 2.0 or GNU
 General Public License Version 2; you choose, see [LICENSE.md](LICENSE.md).
 
-Latest release is 5.1.0. See [CHANGELOG.md](CHANGELOG.md) release history.
+Latest release is 5.2.0 released on July 1st 2021. See [CHANGELOG.md](CHANGELOG.md) release history.
 
 Clixon interaction is best done posting issues, pull requests, or joining the
 [slack channel](https://clixondev.slack.com).
-[Slack invite](https://join.slack.com/t/clixondev/shared_invite/zt-qk66zp47-ahdOlEHKEPmb~5ciVJilPQ) (updated 24/5 2021)
+[Slack invite](https://join.slack.com/t/clixondev/shared_invite/zt-seopvltv-hs~BS7UrMjRdKoDRlCK97w) (updated 30/6 2021)
 
 Clixon is sponsored by [Rubicon Communications LLC(Netgate)](https://www.netgate.com/)

apps/backend/Makefile.in

@@ -152,7 +152,9 @@ install-include: clixon_backend.h clixon_backend_handle.h clixon_backend_transac
 .SUFFIXES: .c .o
 
 .c.o:
-	$(CC) $(INCLUDES) $(CPPFLAGS) -D__PROGRAM__=\"$(APPL)\" $(CFLAGS) -c $<
+	# Note: CLIXON_CONFIG_SBINDIR is where clixon_restconf is believed to be installed, unless
+	# overruled by CLICON_RESTCONF_INSTALLDIR option
+	$(CC) $(INCLUDES) $(CPPFLAGS) -D__PROGRAM__=\"$(APPL)\" -DCLIXON_CONFIG_SBINDIR=\"$(sbindir)\" $(CFLAGS) -c $<
 
 # Just link test programs
 test.c	:

apps/backend/backend_client.c

@@ -238,7 +238,7 @@ client_get_streams(clicon_handle   h,
     cprintf(cb,"</%s>", top);
 
     if (clixon_xml_parse_string(cbuf_get(cb), YB_MODULE, yspec, &x, NULL) < 0){
-	if (netconf_operation_failed_xml(xret, "protocol", clicon_err_reason)< 0)
+	if (xret && netconf_operation_failed_xml(xret, "protocol", clicon_err_reason)< 0)
 	    goto done;
 	goto fail;
     }

apps/backend/backend_commit.c

@@ -132,7 +132,7 @@ generic_validate(clicon_handle       h,
 		cprintf(cb, "Mandatory variable of %s in module %s",
 			xml_parent(x1)?xml_name(xml_parent(x1)):"",
 			yang_argument_get(ys_module(ys)));
-		if (netconf_missing_element_xml(xret, "protocol", xml_name(x1), cbuf_get(cb)) < 0)
+		if (xret && netconf_missing_element_xml(xret, "protocol", xml_name(x1), cbuf_get(cb)) < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -480,9 +480,9 @@ startup_commit(clicon_handle  h,
  * and call application callback validations.
  * @param[in]  h         Clicon handle
  * @param[in]  candidate The candidate database. The wanted backend state
- * @param[out] xret      Error XML tree. Free with xml_free after use
+ * @param[out] xret      Error XML tree, if retval is 0. Free with xml_free after use
  * @retval    -1         Error - or validation failed (but cbret not set)
- * @retval     0         Validation failed (with cbret set)
+ * @retval     0         Validation failed (with xret set)
  * @retval     1         Validation OK       
  * @note Need to differentiate between error and validation fail 
  *       (only done for generic_validate)
@@ -505,16 +505,19 @@ validate_common(clicon_handle       h,
 	goto done;
     }	
     /* This is the state we are going to */
-    if (xmldb_get0(h, db, YB_MODULE, NULL, "/", 0, &td->td_target, NULL, NULL) < 0)
+    if ((ret = xmldb_get0(h, db, YB_MODULE, NULL, "/", 0, &td->td_target, NULL, xret)) < 0)
 	goto done;
-
+    if (ret == 0)
+	goto fail;
     /* Clear flags xpath for get */
     xml_apply0(td->td_target, CX_ELMNT, (xml_applyfn_t*)xml_flag_reset,
 	       (void*)(XML_FLAG_MARK|XML_FLAG_CHANGE));
     /* 2. Parse xml trees 
      * This is the state we are going from */
-    if (xmldb_get0(h, "running", YB_MODULE, NULL, "/", 0, &td->td_src, NULL, NULL) < 0)
+    if ((ret = xmldb_get0(h, "running", YB_MODULE, NULL, "/", 0, &td->td_src, NULL, xret)) < 0)
 	goto done;
+    if (ret == 0)
+	goto fail;
     /* Clear flags xpath for get */
     xml_apply0(td->td_src, CX_ELMNT, (xml_applyfn_t*)xml_flag_reset,
 	       (void*)(XML_FLAG_MARK|XML_FLAG_CHANGE));
@@ -606,26 +609,45 @@ candidate_validate(clicon_handle h,
     if ((td = transaction_new()) == NULL)
 	goto done;
         /* Common steps (with commit) */
-    if ((ret = validate_common(h, db, td, &xret)) < 1){
+    if ((ret = validate_common(h, db, td, &xret)) < 0){
 	/* A little complex due to several sources of validation fails or errors.
 	 * (1) xerr is set -> translate to cbret; (2) cbret set use that; otherwise
-	 * use clicon_err. */
+	 * use clicon_err. 
-	if (xret && clicon_xml2cbuf(cbret, xret, 0, 0, -1) < 0)
+	 * TODO: -1 return should be fatal error, not failed validation
+	 */
+	if (!cbuf_len(cbret) &&
+	    netconf_operation_failed(cbret, "application", clicon_err_reason)< 0)
+	    goto done;
+	goto fail;
+    }
+    if (ret == 0){
+	if (xret == NULL){
+	    clicon_err(OE_CFG, EINVAL, "xret is NULL");
+	    goto done;
+	}
+	if (clicon_xml2cbuf(cbret, xret, 0, 0, -1) < 0)
 	    goto done;
-	plugin_transaction_abort_all(h, td);
 	if (!cbuf_len(cbret) &&
 	    netconf_operation_failed(cbret, "application", clicon_err_reason)< 0)
 	    goto done;
 	goto fail;
     }
     if (xmldb_get0_clear(h, td->td_src) < 0 ||
-	xmldb_get0_clear(h, td->td_target) < 0){
+	xmldb_get0_clear(h, td->td_target) < 0)
-	plugin_transaction_abort_all(h, td);
 	goto done;
-    }
+
     plugin_transaction_end_all(h, td);
     retval = 1;
  done:
+    if (xret)
+	xml_free(xret);
+     if (td){
+	 if (retval < 1)
+	     plugin_transaction_abort_all(h, td);
+	 xmldb_get0_free(h, &td->td_target);
+	 xmldb_get0_free(h, &td->td_src);
+	 transaction_free(td);
+     }
     return retval;
  fail:
     retval = 0;

apps/backend/backend_main.c

@@ -747,11 +747,6 @@ main(int    argc,
 		   clicon_configfile(h));
 	goto done;
     }
-
-    /* Treat unknown XML as anydata */
-    if (clicon_option_bool(h, "CLICON_YANG_UNKNOWN_ANYDATA") == 1)
-	xml_bind_yang_unknown_anydata(1);
-
     /* Publish stream on pubsub channels.
      * CLICON_STREAM_PUB should be set to URL to where streams are published
      * and configure should be run with --enable-publish

apps/backend/backend_plugin_restconf.c

@@ -245,6 +245,7 @@ restconf_pseudo_process_control(clicon_handle h)
     int    i;
     int    nr;
     cbuf  *cb = NULL;
+    char  *dir = NULL;
 
     nr = 10;
     if ((argv = calloc(nr, sizeof(char *))) == NULL){
@@ -256,12 +257,18 @@ restconf_pseudo_process_control(clicon_handle h)
 	clicon_err(OE_UNIX, errno, "cbuf_new");
 	goto done;
     }
-    /* CLICON_RESTCONF_INSTALLDIR is where we think clixon_restconf is installed
+    /* Try to figure out where clixon_restconf is installed
-     * Problem is where to define it? Now in config file, but maybe it should be in configure?
+     * If config option CLICON_RESTCONF_INSTALLDIR is installed, use that.
-     * Tried Makefile but didnt work on Docker since it was moved around.
+     * If not, use the Makefile 
      * Use PATH?
      */
-    cprintf(cb, "%s/clixon_restconf", clicon_option_str(h, "CLICON_RESTCONF_INSTALLDIR"));
+    if ((dir = clicon_option_str(h, "CLICON_RESTCONF_INSTALLDIR")) == NULL){
+	if ((dir = CLIXON_CONFIG_SBINDIR) == NULL){
+	    clicon_err(OE_RESTCONF, EINVAL, "Both option CLICON_RESTCONF_INSTALLDIR and makefile constant CLIXON_CONFIG_SBINDIR are NULL which make sit not possible to know where clixon_restconf is installed(shouldnt happen)");
+	    goto done;
+	}
+    }
+    cprintf(cb, "%s/clixon_restconf", dir);
     argv[i++] = cbuf_get(cb);
     argv[i++] = "-f";
     argv[i++] = clicon_option_str(h, "CLICON_CONFIGFILE");
@@ -270,11 +277,11 @@ restconf_pseudo_process_control(clicon_handle h)
      * see restconf_pseudo_set_log which sets flag when process starts
      */
     argv[i++] = "-D";
-    argv[i++] = strdup("0");
+    argv[i++] = "0";
     argv[i++] = "-l";
-    argv[i++] = strdup("s"); /* There is also log-destination in clixon-restconf.yang */
+    argv[i++] = "s"; /* There is also log-destination in clixon-restconf.yang */
     argv[i++] = "-R";
-    argv[i++] = strdup(""); 
+    argv[i++] = ""; 
     argv[i++] = NULL;
     assert(i==nr);
     if (clixon_process_register(h, RESTCONF_PROCESS,

apps/cli/cli_common.c

@@ -448,7 +448,7 @@ cli_debug_cli(clicon_handle h,
     cg_var *cv;
     int     level;
 
-    if ((cv = cvec_find(vars, "level")) == NULL){
+    if ((cv = cvec_find_var(vars, "level")) == NULL){
 	if (cvec_len(argv) != 1){
 	    clicon_err(OE_PLUGIN, EINVAL, "Requires either label var or single arg: 0|1");
 	    goto done;
@@ -479,7 +479,7 @@ cli_debug_backend(clicon_handle h,
     cg_var *cv;
     int     level;
 
-    if ((cv = cvec_find(vars, "level")) == NULL){
+    if ((cv = cvec_find_var(vars, "level")) == NULL){
 	if (cvec_len(argv) != 1){
 	    clicon_err(OE_PLUGIN, EINVAL, "Requires either label var or single arg: 0|1");
 	    goto done;
@@ -513,7 +513,7 @@ cli_debug_restconf(clicon_handle h,
     cg_var *cv;
     int     level;
 
-    if ((cv = cvec_find(vars, "level")) == NULL){
+    if ((cv = cvec_find_var(vars, "level")) == NULL){
 	if (cvec_len(argv) != 1){
 	    clicon_err(OE_PLUGIN, EINVAL, "Requires either label var or single arg: 0|1");
 	    goto done;

apps/cli/cli_generate.c

@@ -626,18 +626,18 @@ yang2cli_var(clicon_handle h,
 	    completionp = clicon_cli_genmodel_completion(h);
 	if (completionp)
 	    cprintf(cb, "(");
-	if ((retval = yang2cli_var_sub(h, ys, yrestype, helptext, cvtype, 
+	if (yang2cli_var_sub(h, ys, yrestype, helptext, cvtype, 
-				       options, cvv, patterns, fraction_digits, cb)) < 0)
+			     options, cvv, patterns, fraction_digits, cb) < 0)
 	    goto done;
 	if (completionp){
 	    result = cli_expand_var_generate(h, ys, cvtype, 
 					options, fraction_digits,
 					cb);
 		if (result < 0)
-		goto done;
+		    goto done;
 		if (result == 0)
-	    yang2cli_helptext(cb, helptext);
+		    yang2cli_helptext(cb, helptext);
-	    cprintf(cb, ")");
+		cprintf(cb, ")");
 	}
     }
     retval = 0;

apps/cli/cli_main.c

@@ -635,11 +635,6 @@ main(int    argc,
 	goto done;
     /* Set default namespace according to CLICON_NAMESPACE_NETCONF_DEFAULT */
     xml_nsctx_namespace_netconf_default(h);
-    
-    /* Treat unknwon XML as anydata */
-    if (clicon_option_bool(h, "CLICON_YANG_UNKNOWN_ANYDATA") == 1)
-	xml_bind_yang_unknown_anydata(1);
-
     /* Create top-level and store as option */
     if ((yspec = yspec_new()) == NULL)
 	goto done;

apps/cli/cli_show.c

@@ -495,8 +495,8 @@ cli_show_config1(clicon_handle h,
 	    cli_xml2cli(xc, prefix, gt, cligen_output); /* cli syntax */
 	break;
     case FORMAT_NETCONF:
-	cligen_output(stdout, "<rpc xmlns=\"%s\"><edit-config><target><candidate/></target><config>\n",
+	cligen_output(stdout, "<rpc xmlns=\"%s\" %s><edit-config><target><candidate/></target><config>\n",
-		      NETCONF_BASE_NAMESPACE);
+		      NETCONF_BASE_NAMESPACE, NETCONF_MESSAGE_ID_ATTR);
 	xc = NULL; /* Dont print xt itself */
 	while ((xc = xml_child_each(xt, xc, -1)) != NULL)
 	    cli_xml2file(xc, 2, 1, cligen_output);

apps/restconf/restconf_api_fcgi.c

@@ -220,8 +220,11 @@ restconf_reply_send(void  *req0,
     return retval;
 }
 
-/*!
+/*! Get input data from http request, eg such as curl -X PUT http://... <indata>
  * @param[in]  req        Fastcgi request handle
+ * @retval     indata     
+ * @retval     NULL       Error
+ * @note: creates a new cbuf which differs from native api where a pointer is returned
  */
 cbuf *
 restconf_get_indata(void *req0)

apps/restconf/restconf_api_native.c

@@ -172,9 +172,9 @@ restconf_reply_send(void  *req0,
     return retval;
 }
 
-/*! get input data
+/*! Get input data from http request, eg such as curl -X PUT http://... <indata>
- * @param[in]  req        Fastcgi request handle
+ * @param[in]  req        Request handle
- * @note Pulls up an event buffer and then copies it to a cbuf. This is not efficient.
+ * @note: reuses cbuf from stream-data
  */
 cbuf *
 restconf_get_indata(void *req0)

apps/restconf/restconf_evhtp.c

@@ -396,7 +396,7 @@ restconf_evhtp_reply(restconf_conn        *rc,
      * [RFC7231]).
      */
     if (sd->sd_code != 204 && sd->sd_code > 199)
-	if (restconf_reply_header(sd, "Content-Length", "%lu", sd->sd_body_len) < 0)
+	if (restconf_reply_header(sd, "Content-Length", "%zu", sd->sd_body_len) < 0)
 	    goto done;	
     /* Create reply and write headers */
     if (native_send_reply(rc, sd, req) < 0)
@@ -515,6 +515,8 @@ restconf_path_root(evhtp_request_t *req,
     if (clicon_debug_get())
 	evhtp_headers_for_each(req->headers_in, evhtp_print_header, h);
     /* Query vector, ie the ?a=x&b=y stuff */
+    if (sd->sd_qvec)
+	cvec_free(sd->sd_qvec);
     if ((sd->sd_qvec = cvec_new(0)) ==NULL){
 	clicon_err(OE_UNIX, errno, "cvec_new");
 	evhtp_internal_error(req);
@@ -527,6 +529,7 @@ restconf_path_root(evhtp_request_t *req,
 	    clicon_err(OE_CFG, errno, "evbuffer_pullup");
 	    goto done;
 	}
+	cbuf_reset(sd->sd_indata);
 	/* Note the pullup may not be null-terminated */
 	cbuf_append_buf(sd->sd_indata, buf, len);
     }

apps/restconf/restconf_lib.c

@@ -499,6 +499,14 @@ restconf_insert_attributes(cxobj *xdata,
  * @param[in] ys   Yang node of (unknown) statement belonging to extension
  * @retval     0   OK, all callbacks executed OK
  * @retval    -1   Error in one callback
+ * @note This extension adds semantics to YANG according to RFC8040 as follows:
+ *          - The list-stmt is not required to have a key-stmt defined.(NB!!)
+ *          - The if-feature-stmt is ignored if present.
+ *          - The config-stmt is ignored if present.
+ *          - The available identity values for any 'identityref'
+ *              leaf or leaf-list nodes are limited to the module
+ *              containing this extension statement and the modules
+ *              imported into that module.
  */
 int
 restconf_main_extension_cb(clicon_handle h,
@@ -522,6 +530,9 @@ restconf_main_extension_cb(clicon_handle h,
 	goto ok;
     if ((yn = ys_dup(yc)) == NULL)
 	goto done;
+    /* yang-data extension: The list-stmt is not required to have a key-stmt defined.
+     */
+    yang_flag_set(yn, YANG_FLAG_NOKEY);
     if (yn_insert(yang_parent_get(ys), yn) < 0)
 	goto done;
  ok:

apps/restconf/restconf_main_fcgi.c

@@ -375,10 +375,6 @@ main(int    argc,
     if ((yspec = yspec_new()) == NULL)
 	goto done;
     clicon_dbspec_yang_set(h, yspec);
-    /* Treat unknown XML as anydata */
-    if (clicon_option_bool(h, "CLICON_YANG_UNKNOWN_ANYDATA") == 1)
-	xml_bind_yang_unknown_anydata(1);
-
     /* Initialize plugin module by creating a handle holding plugin and callback lists */
     if (clixon_plugin_module_init(h) < 0)
 	goto done;
@@ -423,6 +419,12 @@ main(int    argc,
     if (yang_spec_parse_module(h, "ietf-restconf", NULL, yspec)< 0)
 	goto done;
     
+#ifdef YANG_PATCH
+    /* Load yang restconf patch module */
+    if (yang_spec_parse_module(h, "ietf-yang-patch", NULL, yspec)< 0)
+       goto done;
+#endif // YANG_PATCH
+
     /* Add netconf yang spec, used as internal protocol */
     if (netconf_module_load(h) < 0)
 	goto done;

apps/restconf/restconf_main_native.c

@@ -265,7 +265,7 @@ buf_write(char   *buf,
 	}
 	memcpy(dbgstr, buf, sz);
 	dbgstr[sz] = '\0';
-	clicon_debug(1, "%s buflen:%lu buf:%s", __FUNCTION__, buflen, dbgstr);
+	clicon_debug(1, "%s buflen:%zu buf:%s", __FUNCTION__, buflen, dbgstr);
 	free(dbgstr);
     }
     while (totlen < buflen){
@@ -467,14 +467,16 @@ alpn_select_proto_cb(SSL                  *ssl,
 	inp++;
 	if (clicon_debug_get()) /* debug print the protoocol */
 	    alpn_proto_dump(__FUNCTION__, (const char*)inp, len);
+#ifdef HAVE_LIBEVHTP
 	if (pref < 10 && len == 8 && strncmp((char*)inp, "http/1.1", len) == 0){
 	    *outlen = len;
 	    *out = inp;
 	    pref = 10;
 	}
+#endif
 #ifdef HAVE_LIBNGHTTP2
 	/* Higher pref than http/1.1 */
-	else if (pref < 20 && len == 2 && strncmp((char*)inp, "h2", len) == 0){
+	if (pref < 20 && len == 2 && strncmp((char*)inp, "h2", len) == 0){
 	    *outlen = len;
 	    *out = inp;
 	    pref = 20;
@@ -591,15 +593,7 @@ restconf_close_ssl_socket(restconf_conn *rc,
 {
     int retval = -1;
     int                 ret;
-#ifdef HAVE_LIBEVHTP
-    evhtp_connection_t *evconn;
 
-    if ((evconn = rc->rc_evconn) != NULL){
-	clicon_debug(1, "%s evconn-free (%p)", __FUNCTION__, evconn);
-	if (evconn)
-	    evhtp_connection_free(evconn); /* evhtp */
-    }
-#endif /* HAVE_LIBEVHTP */
     if (rc->rc_ssl != NULL){
 	if (shutdown && (ret = SSL_shutdown(rc->rc_ssl)) < 0){
 #if 0
@@ -612,6 +606,10 @@ Note that in this case SSL_ERROR_ZERO_RETURN does not necessarily indicate that
 	}
 	SSL_free(rc->rc_ssl);
 	rc->rc_ssl = NULL;
+#ifdef HAVE_LIBEVHTP
+	if (rc->rc_evconn)
+	    rc->rc_evconn->ssl = NULL;
+#endif
     }
     if (close(rc->rc_s) < 0){
 	clicon_err(OE_UNIX, errno, "close");
@@ -649,7 +647,7 @@ send_badrequest(clicon_handle       h,
     cprintf(cb, "HTTP/1.1 400 Bad Request\r\nConnection: close\r\n");
     if (body){
 	cprintf(cb, "Content-Type: %s\r\n", media);
-	cprintf(cb, "Content-Length: %lu\r\n", strlen(body));
+	cprintf(cb, "Content-Length: %zu\r\n", strlen(body));
     }
     else
 	cprintf(cb, "Content-Length: 0\r\n");
@@ -687,6 +685,10 @@ restconf_connection(int   s,
     ssize_t               n;
     char                  buf[BUFSIZ]; /* from stdio.h, typically 8K XXX: reduce for test */
     int                   readmore = 1;
+    int                   sslerr;
+#ifdef HAVE_LIBNGHTTP2
+    int                   ret;
+#endif
 #ifdef HAVE_LIBEVHTP
     clicon_handle         h;
     evhtp_connection_t   *evconn = NULL;
@@ -708,27 +710,53 @@ restconf_connection(int   s,
 	       curl -Ssik --key /var/tmp/./test_restconf_ssl_certs.sh/certs/limited.key --cert /var/tmp/./test_restconf_ssl_certs.sh/certs/limited.crt -X GET https://localhost/restconf/data/example:x
 	    */
 	    if ((n = SSL_read(rc->rc_ssl, buf, sizeof(buf))) < 0){
-		clicon_err(OE_XML, errno, "SSL_read");
+		sslerr = SSL_get_error(rc->rc_ssl, n);
-		goto done;
+		clicon_debug(1, "%s SSL_read() n:%zd errno:%d sslerr:%d", __FUNCTION__, n, errno, sslerr);
+		switch (sslerr){
+		case SSL_ERROR_WANT_READ:            /* 2 */
+		    /* SSL_ERROR_WANT_READ is returned when the last operation was a read operation 
+		     * from a nonblocking BIO. 
+		     * That is, it can happen if restconf_socket_init() below is called 
+		     * with SOCK_NONBLOCK
+		     */
+		    clicon_debug(1, "%s SSL_read SSL_ERROR_WANT_READ", __FUNCTION__);
+		    usleep(1000);
+		    readmore = 1;
+		    break;
+		default:
+		    clicon_err(OE_XML, errno, "SSL_read");
+		    goto done;              
+		} /* switch */
+		continue; /* readmore */
 	    }
 	}
 	else{
 	    if ((n = read(rc->rc_s, buf, sizeof(buf))) < 0){ /* XXX atomicio ? */
-		if (errno == ECONNRESET) {/* Connection reset by peer */
+		switch(errno){
+		case ECONNRESET:/* Connection reset by peer */
 		    clicon_debug(1, "%s %d Connection reset by peer", __FUNCTION__, rc->rc_s);
 		    clixon_event_unreg_fd(rc->rc_s, restconf_connection);
 		    close(rc->rc_s);
 		    restconf_conn_free(rc);
 		    goto ok; /* Close socket and ssl */
+		    break;
+		case EAGAIN:
+		    clicon_debug(1, "%s read EAGAIN", __FUNCTION__);
+		    usleep(1000);
+		    readmore = 1;
+		    break;
+		default:;
+		    clicon_err(OE_XML, errno, "read");
+		    goto done;
+		    break;
 		}
-		clicon_err(OE_XML, errno, "read");
+		continue;
-		goto done;
 	    }
 	}
-	clicon_debug(1, "%s read:%ld", __FUNCTION__, n);
+	clicon_debug(1, "%s (ssl)read:%zd", __FUNCTION__, n);
 	if (n == 0){
 	    clicon_debug(1, "%s n=0 closing socket", __FUNCTION__);
-	    if (restconf_close_ssl_socket(rc, 1) < 0)
+	    if (restconf_close_ssl_socket(rc, 0) < 0)
 		goto done;
 	    restconf_conn_free(rc);    
 	    rc = NULL;
@@ -762,7 +790,6 @@ restconf_connection(int   s,
 		clixon_event_unreg_fd(rc->rc_s, restconf_connection);
 		clicon_debug(1, "%s evconn-free (%p) 2", __FUNCTION__, evconn);
 		restconf_conn_free(rc);
-		evhtp_connection_free(evconn);
 		goto ok;
 	    } /* connection_parse_nobev */
 	    clicon_debug(1, "%s connection_parse OK", __FUNCTION__);
@@ -866,9 +893,14 @@ restconf_connection(int   s,
 #endif /* HAVE_LIBEVHTP */
 #ifdef HAVE_LIBNGHTTP2
 	case HTTP_2:
-	    if (http2_recv(rc, (unsigned char *)buf, n) < 0)
+	    if ((ret = http2_recv(rc, (unsigned char *)buf, n)) < 0)
 		goto done;
-	    //notused	    sd = restconf_stream_find(rc, 0); /* default stream */
+	    if (ret == 0){
+		restconf_close_ssl_socket(rc, 1);
+		if (restconf_conn_free(rc) < 0)
+		    goto done;
+		goto ok;
+	    }
 	    /* There may be more data frames */
 	    readmore++;
 	    break;
@@ -988,7 +1020,7 @@ ssl_alpn_check(clicon_handle        h,
 	}
 	if (alpn != NULL){
 	    cprintf(cberr, "<errors xmlns=\"urn:ietf:params:xml:ns:yang:ietf-restconf\"><error><error-type>protocol</error-type><error-tag>malformed-message</error-tag><error-message>ALPN: protocol not recognized: %s</error-message></error></errors>", alpn);
-	    clicon_log(LOG_NOTICE, "Warning: %s", cbuf_get(cberr));
+	    clicon_log(LOG_INFO, "%s Warning: %s", __FUNCTION__, cbuf_get(cberr));
 	    if (send_badrequest(h, rc->rc_s, rc->rc_ssl,
 				"application/yang-data+xml",
 				cbuf_get(cberr)) < 0)
@@ -996,17 +1028,9 @@ ssl_alpn_check(clicon_handle        h,
 	}
 	else{
 	    /* XXX Sending badrequest here gives a segv in SSL_shutdown() later or a SIGPIPE here */
-	    clicon_log(LOG_NOTICE, "Warning: ALPN: No protocol selected");
+	    clicon_log(LOG_INFO, "%s Warning: ALPN: No protocol selected", __FUNCTION__);
 	}
-	restconf_conn_free(rc);
-#ifdef HAVE_LIBEVHTP
-	{
-	    evhtp_connection_t *evconn;
 
-	    if ((evconn = rc->rc_evconn) != NULL)
-		evhtp_connection_free(evconn); /* evhtp */
-	}
-#endif /* HAVE_LIBEVHTP */
 	if (rc->rc_ssl){
             /* nmap ssl-known-key SEGV at s->method->ssl_shutdown(s); 
 	     * OR OpenSSL error: : SSL_shutdown, err: SSL_ERROR_SYSCALL(5)
@@ -1014,7 +1038,7 @@ ssl_alpn_check(clicon_handle        h,
 	    if ((ret = SSL_shutdown(rc->rc_ssl)) < 0){ 
 		int e = SSL_get_error(rc->rc_ssl, ret);
 		if (e == SSL_ERROR_SYSCALL){
-		    clicon_log(LOG_NOTICE, "Warning: SSL_shutdown SSL_ERROR_SYSCALL");
+		    clicon_log(LOG_INFO, "%s Warning: SSL_shutdown SSL_ERROR_SYSCALL", __FUNCTION__);
 		    /* Continue */
 		}
 		else {
@@ -1024,6 +1048,7 @@ ssl_alpn_check(clicon_handle        h,
 	    }
 	    SSL_free(rc->rc_ssl);
 	}
+	restconf_conn_free(rc);
     }
     retval = 0; /* ALPN not OK */
  done:
@@ -1194,7 +1219,9 @@ restconf_accept_client(int   fd,
 	    } /* SSL_accept */
 	} /* while(readmore) */
 	/* Sets data and len to point to the client's requested protocol for this connection. */
+#ifndef OPENSSL_NO_NEXTPROTONEG
 	SSL_get0_next_proto_negotiated(rc->rc_ssl, &alpn, &alpnlen);
+#endif /* !OPENSSL_NO_NEXTPROTONEG */
 	if (alpn == NULL) {
 	    /* Returns a pointer to the selected protocol in data with length len. */
 	    SSL_get0_alpn_selected(rc->rc_ssl, &alpn, &alpnlen);
@@ -1559,7 +1586,7 @@ restconf_openssl_init(clicon_handle h,
 	}
 	int status = setrlimit(RLIMIT_CORE, &rlp);
 	if (status != 0) {
-	    clicon_log(LOG_NOTICE, "%s: setrlimit() failed, %s", __func__, strerror(errno));
+	    clicon_log(LOG_INFO, "%s: setrlimit() failed, %s", __FUNCTION__, strerror(errno));
 	}
     }
 
@@ -1671,10 +1698,6 @@ restconf_clixon_init(clicon_handle h,
     if ((yspec = yspec_new()) == NULL)
 	goto done;
     clicon_dbspec_yang_set(h, yspec);
-    /* Treat unknown XML as anydata */
-    if (clicon_option_bool(h, "CLICON_YANG_UNKNOWN_ANYDATA") == 1)
-	xml_bind_yang_unknown_anydata(1);
-    
     /* Load restconf plugins before yangs are loaded (eg extension callbacks) */
     if ((dir = clicon_restconf_dir(h)) != NULL)
 	if (clixon_plugins_load(h, CLIXON_PLUGIN_INIT, dir, NULL) < 0)
@@ -1714,6 +1737,12 @@ restconf_clixon_init(clicon_handle h,
     if (yang_spec_parse_module(h, "ietf-restconf", NULL, yspec)< 0)
 	goto done;
     
+#ifdef YANG_PATCH
+    /* Load yang restconf patch module */
+    if (yang_spec_parse_module(h, "ietf-yang-patch", NULL, yspec)< 0)
+       goto done;
+#endif // YANG_PATCH
+
     /* Add netconf yang spec, used as internal protocol */
     if (netconf_module_load(h) < 0)
 	goto done;
@@ -1733,7 +1762,7 @@ restconf_clixon_init(clicon_handle h,
     if (clicon_nsctx_global_set(h, nsctx_global) < 0)
 	goto done;
     if (inline_config != NULL && strlen(inline_config)){
-	clicon_debug(1, "%s using restconf inline config", __FUNCTION__);
+	clicon_debug(1, "%s reading from inline config", __FUNCTION__);
 	if ((ret = clixon_xml_parse_string(inline_config, YB_MODULE, yspec, &xrestconf, &xerr)) < 0)
 	    goto done;
 	if (ret == 0){
@@ -1755,6 +1784,7 @@ restconf_clixon_init(clicon_handle h,
 		goto done;
     }
     else if (clicon_option_bool(h, "CLICON_BACKEND_RESTCONF_PROCESS") == 0){
+	clicon_debug(1, "%s reading from clixon config", __FUNCTION__);
 	/* If not read from backend, try to get restconf config from local config-file */
 	if ((xrestconf = clicon_conf_restconf(h)) != NULL){
 	    /*! Basic config init, set auth-type, pretty, etc ret 0 means disabled */
@@ -1772,6 +1802,7 @@ restconf_clixon_init(clicon_handle h,
     /* If no local config, or it is disabled, try to query backend of config. 
      */
     else {
+	clicon_debug(1, "%s reading from backend datastore config", __FUNCTION__);
 	if ((ret = restconf_clixon_backend(h, xrestconfp)) < 0)
 	    goto done;
 	if (ret == 0)

apps/restconf/restconf_methods.c

@@ -74,6 +74,7 @@
 #include "restconf_api.h"
 #include "restconf_err.h"
 #include "restconf_methods.h"
+#include "restconf_methods_post.h"
 
 /*! REST OPTIONS method
  * According to restconf
@@ -513,11 +514,12 @@ api_data_write(clicon_handle h,
     /* Create text buffer for transfer to backend */
     if ((cbx = cbuf_new()) == NULL)
 	goto done;
-    cprintf(cbx, "<rpc xmlns=\"%s\" username=\"%s\" xmlns:%s=\"%s\">",
+    cprintf(cbx, "<rpc xmlns=\"%s\" username=\"%s\" xmlns:%s=\"%s\" %s>",
 	    NETCONF_BASE_NAMESPACE,
 	    username?username:"",
 	    NETCONF_BASE_PREFIX,
-	    NETCONF_BASE_NAMESPACE); /* bind nc to netconf namespace */
+	    NETCONF_BASE_NAMESPACE,  /* bind nc to netconf namespace */
+	    NETCONF_MESSAGE_ID_ATTR);
     cprintf(cbx, "<edit-config");
     /* RFC8040 Sec 1.4:
      * If this is a "data" request and the NETCONF server supports :startup,
@@ -578,6 +580,536 @@ api_data_write(clicon_handle h,
    return retval;
 } /* api_data_write */
 
+#ifdef YANG_PATCH
+/*! YANG PATCH method
+ * @param[in]  h         Clixon handle
+ * @param[in]  req       Generic Www handle
+ * @param[in]  api_path0 According to restconf (Sec 3.5.3.1 in rfc8040)
+ * @param[in]  pcvec     Vector of path ie DOCUMENT_URI element
+ * @param[in]  pi        Offset, where to start pcvec
+ * @param[in]  qvec      Vector of query string (QUERY_STRING)
+ * @param[in]  data      Stream input data
+ * @param[in]  pretty    Set to 1 for pretty-printed xml/json output
+ * @param[in]  media_out Output media
+ * Netconf:  <edit-config> (nc:operation="merge")
+ * See RFC8072
+ * YANG patch can be used to "create", "delete", "insert", "merge", "move", "replace", and/or
+   "remove" a resource within the target resource.
+ * Currently "move" not supported
+ */
+static int
+api_data_yang_patch(clicon_handle h,
+	       void         *req,
+	       char         *api_path0,
+	       cvec         *pcvec,
+	       int           pi,
+	       cvec         *qvec,
+	       char         *data,
+	       int           pretty,
+	       restconf_media media_out,
+	       ietf_ds_t     ds)
+{
+    int            retval = -1;
+    int            i;
+    cxobj         *xdata0 = NULL; /* Original -d data struct (including top symbol) */
+    cbuf          *cbx = NULL;
+    cxobj         *xtop = NULL; /* top of api-path */
+    cxobj         *xbot = NULL; /* bottom of api-path */
+    yang_stmt     *ybot = NULL; /* yang of xbot */
+    cxobj         *xbot_tmp = NULL;
+    yang_stmt     *yspec;
+    char          *api_path;
+    cxobj         *xret = NULL;
+    cxobj         *xretcom = NULL; /* return from commit */
+    cxobj         *xretdis = NULL; /* return from discard-changes */
+    cxobj         *xerr = NULL;    /* malloced must be freed */
+    int            ret;
+    cvec          *nsc = NULL;
+    yang_bind      yb;
+    char          *xpath = NULL;
+    const int temp_str_malloc_size = 5000;
+    char *path_orig_1 = NULL;
+
+    clicon_debug(1, "%s api_path:\"%s\"",  __FUNCTION__, api_path0);
+    if ((yspec = clicon_dbspec_yang(h)) == NULL){
+	clicon_err(OE_FATAL, 0, "No DB_SPEC");
+	goto done;
+    }
+    api_path=api_path0;
+    /* strip /... from start */
+    for (i=0; i<pi; i++)
+	api_path = index(api_path+1, '/');
+	/* Translate yang-patch path to xpath: xpath (cbpath) and namespace context (nsc) */
+    char yang_patch_path[] = "/ietf-yang-patch:yang-patch";
+	if ((ret = api_path2xpath(yang_patch_path, yspec, &xpath, &nsc, &xerr)) < 0)
+	    goto done;
+	if (ret == 0){ /* validation failed */
+	    if (api_return_err0(h, req, xerr, pretty, media_out, 0) < 0)
+		goto done;
+	    goto ok;
+	}
+    /* Create config top-of-tree */
+    if ((xtop = xml_new(NETCONF_INPUT_CONFIG, NULL, CX_ELMNT)) == NULL)
+	goto done;
+
+    /* Translate yang-patch path to xml in the form of xtop/xbot */
+    xbot = xtop;
+	if ((ret = api_path2xml(yang_patch_path, yspec, xtop, YC_DATANODE, 1, &xbot, &ybot, &xerr)) < 0)
+	    goto done;
+	if (ret == 0){ /* validation failed */
+	    if (api_return_err(h, req, xerr, pretty, media_out, 0) < 0)
+		goto done;
+	    goto ok;
+	}
+ 
+	yb = YB_MODULE;
+	if ((ret = clixon_json_parse_string(data, yb, yspec, &xbot, &xerr)) < 0){
+	    if (netconf_malformed_message_xml(&xerr, clicon_err_reason) < 0)
+		goto done;
+	    if (api_return_err0(h, req, xerr, pretty, media_out, 0) < 0)
+		goto done;
+	    goto ok;
+	}
+	if (ret == 0){
+	    if (api_return_err0(h, req, xerr, pretty, media_out, 0) < 0)
+		goto done;
+	    goto ok;
+	}
+    /* 
+     * RFC 8072 2.1: The message-body MUST identify exactly one resource instance
+     */
+    int            nrchildren0 = 0;
+    cxobj         *x = NULL;
+    if (xml_child_nr_type(xbot, CX_ELMNT) - nrchildren0 != 1){
+	if (netconf_malformed_message_xml(&xerr, "The message-body MUST contain exactly one instance of the expected data resource") < 0)
+	    goto done;
+	if (api_return_err0(h, req, xerr, pretty, media_out, 0) < 0)
+	    goto done;
+	goto ok;
+    }
+
+    size_t veclen;
+    cxobj **vec = NULL;
+    while ((x = xml_child_each(xbot, x, CX_ELMNT)) != NULL){
+    ret = xpath_vec(x, nsc, "edit", &vec, &veclen);
+	if (xml_flag(x, XML_FLAG_MARK)){
+	    xml_flag_reset(x, XML_FLAG_MARK);
+	    continue;
+	}
+    }
+    path_orig_1 = malloc(temp_str_malloc_size);
+    if (path_orig_1 == NULL) {
+        goto done;
+    } else {
+        strcpy(path_orig_1, restconf_uripath(h));
+    }
+
+    // Loop through the edits
+    for (int i = 0; i < veclen; i++) {
+        cxobj *xn = vec[i];
+
+        // Get target
+        char *target_val = NULL;
+        cxobj **target_vec = NULL;
+        size_t target_veclen;
+        ret = xpath_vec(xn, nsc, "target", &target_vec, &target_veclen);
+        if (ret < 0) {
+            goto done;
+        }
+        for (int j = 0; j < target_veclen; j++) {
+            cxobj *target_xn = target_vec[j];
+            target_val = xml_body(target_xn);
+        }
+
+        // Get operation
+        char *op_val = NULL;
+        cxobj **operation_vec = NULL;
+        size_t operation_veclen;
+        ret = xpath_vec(xn, nsc, "operation", &operation_vec, &operation_veclen);
+        if (ret < 0) {
+            goto done;
+        }
+        for (int j = 0; j < operation_veclen; j++) {
+            cxobj *operation_xn = operation_vec[j];
+            op_val = xml_body(operation_xn);
+        }
+
+        // Get "point" and "where" for insert operations
+        char *point_val = NULL;
+        cxobj **point_vec = NULL;
+        size_t point_veclen;
+        if (strcmp(op_val, "insert") == 0) {
+            ret = xpath_vec(xn, nsc, "point", &point_vec, &point_veclen);
+            if (ret < 0) {
+                goto done;
+            }
+            for (int j = 0; j < point_veclen; j++) {
+                cxobj *point_xn = point_vec[j];
+                point_val = xml_body(point_xn);
+            }
+        }
+        char *where_val = NULL;
+        cxobj **where_vec = NULL;
+        size_t where_veclen;
+        if (strcmp(op_val, "insert") == 0) {
+            ret = xpath_vec(xn, nsc, "where", &where_vec, &where_veclen);
+            if (ret < 0) {
+                goto done;
+            }
+            for (int j = 0; j < where_veclen; j++) {
+                cxobj *where_xn = where_vec[j];
+                where_val = xml_body(where_xn);
+            }
+        }
+
+        // Construct request URI
+        char* simple_patch_request_uri = NULL;
+        simple_patch_request_uri = malloc(temp_str_malloc_size);
+        strcpy(simple_patch_request_uri, path_orig_1);
+
+        int plain_patch_val = 0;
+        char* api_path_target = NULL;
+        api_path_target = malloc(temp_str_malloc_size);
+        strcpy(api_path_target, api_path);
+        if (strcmp(op_val, "merge") == 0) {
+            plain_patch_val = 1;
+            strcat(api_path_target, target_val);
+            strcat(simple_patch_request_uri, target_val);
+        }
+
+        if (xerr)
+	        xml_free(xerr);
+        if ((xtop = xml_new(NETCONF_INPUT_CONFIG, NULL, CX_ELMNT)) == NULL)
+	        goto done;
+
+        // Get key field
+        /* Translate api_path to xml in the form of xtop/xbot */
+        xbot_tmp = xtop;
+	    if ((ret = api_path2xml(api_path_target, yspec, xtop, YC_DATANODE, 1, &xbot_tmp, &ybot, &xerr)) < 0)
+	        goto done;
+	    if (ret == 0){ /* validation failed */
+	        if (api_return_err0(h, req, xerr, pretty, media_out, 0) < 0)
+	    	goto done;
+	        goto ok;
+	    }
+        char *key_node_id = xml_name(xbot_tmp);
+        char *path = NULL;
+        if ((path = restconf_param_get(h, "REQUEST_URI")) != NULL){
+            for (int i1 = 0; i1 <pi; i1++)
+	        path = index(path+1, '/');
+        }
+        const char colon[2] = ":";
+        char *modname = strtok(&(path[1]), colon);
+
+        cxobj **key_vec = NULL;
+
+        key_vec = xml_childvec_get(xbot_tmp);
+        cxobj *key_xn = NULL;
+        if (key_vec != NULL) {
+            key_xn = key_vec[0];
+        }
+
+        // Get values (for "delete", there are no values)
+        cxobj **values_vec = NULL;
+        size_t values_veclen;
+        xpath_vec(xn, nsc, "value", &values_vec, &values_veclen);
+        key_node_id = NULL;
+
+        // Loop through the values
+        for (int j = 0; j < values_veclen; j++) {
+            cxobj *values_xn = values_vec[j];
+            cxobj** values_child_vec = xml_childvec_get(values_xn);
+            if (key_node_id == NULL)
+                key_node_id = xml_name(*values_child_vec);
+
+            char *patch_header = NULL;
+            patch_header = malloc(temp_str_malloc_size);
+            if (patch_header == NULL) {
+                goto done;
+            }
+            strcpy(patch_header, modname);
+            strcat(patch_header, ":");
+            strcat(patch_header, key_node_id);
+            cxobj *x_simple_patch = xml_new(patch_header, NULL, CX_ELMNT);
+            if (x_simple_patch == NULL)
+                goto done;
+            int value_vec_len = xml_child_nr(*values_child_vec);
+            cxobj** value_vec = xml_childvec_get(*values_child_vec);
+            cxobj * value_vec_tmp = NULL;
+
+            // For "replace", delete the item and then POST it
+            // TODO - in an ordered list, insert it into its original position
+            if (strcmp(op_val,"replace") == 0) {
+                char *delete_req_uri = malloc(temp_str_malloc_size);
+                if (delete_req_uri == NULL)
+                    break;
+
+                strcpy(delete_req_uri, simple_patch_request_uri);
+                strcat(delete_req_uri, target_val);
+
+                // Delete the object with the old values
+                ret = api_data_delete(h, req, delete_req_uri, pi, pretty, YANG_DATA_JSON, ds );
+                free(delete_req_uri);
+
+                // Now insert the object with the new values
+                char *json_simple_patch = malloc(temp_str_malloc_size);
+                if (json_simple_patch == NULL)
+                    goto done;
+                memset(json_simple_patch, 0, temp_str_malloc_size);
+
+                for (int k = 0; k < value_vec_len; k++) {
+                    if (value_vec[k] != NULL) {
+                        value_vec_tmp = xml_dup(value_vec[k]);
+                        xml_addsub(x_simple_patch, value_vec_tmp);
+                    }
+                }
+                cbuf* cb = cbuf_new();
+                xml2json_cbuf(cb, x_simple_patch, 1);
+
+                // Some ugly text processing to get the JSON to match what api_data_post() expects
+                char *json_simple_patch_tmp = cbuf_get(cb);
+                int brace_count = 0;
+                for (int l = 0; l < strlen(json_simple_patch_tmp); l++) {
+                    char c = json_simple_patch_tmp[l];
+                    if (c == '{') {
+                        brace_count++;
+                        if (brace_count == 2) {
+                            json_simple_patch[strlen(json_simple_patch)] = '[';
+                        }
+                    }
+                    json_simple_patch[strlen(json_simple_patch)] = c;
+                }
+                /* strip /... from end */
+                char *post_req_uri = malloc(temp_str_malloc_size);
+                if (post_req_uri == NULL)
+                    break;
+                memset(post_req_uri, 0, temp_str_malloc_size);
+                if (post_req_uri == NULL)
+                    break;
+                int idx = strlen(target_val);
+                for (int l = strlen(target_val); l>= 0; l--) {
+                    if (target_val[l] == '/') {
+                        idx = l;
+                        break;
+                    }
+                }
+                strncpy(post_req_uri, target_val, idx);
+                strcat(simple_patch_request_uri, post_req_uri);
+                free(post_req_uri);
+                for (int l = strlen(json_simple_patch); l>= 0; l--) {
+                    char c = json_simple_patch[l];
+                    if (c == '}') {
+                       json_simple_patch[l] = ']';
+                       json_simple_patch[l + 1] = '}';
+                       break;
+                    }
+                }
+
+                // Send the POST request
+                ret = api_data_post(h, req, simple_patch_request_uri, pi, qvec, json_simple_patch, pretty, YANG_DATA_JSON, media_out, ds );
+                if (value_vec_tmp != NULL)
+                    free(value_vec_tmp);
+                free(x_simple_patch);
+                free(patch_header); // NULL check was already done before
+                if (ret != 0)
+                    goto done;
+                break;
+            }
+
+            // For "create", put all the data values into a single POST request
+            if (strcmp(op_val,"create") == 0) {
+                for (int k = 0; k < value_vec_len; k++) {
+                    if (value_vec[k] != NULL) {
+                        value_vec_tmp = xml_dup(value_vec[k]);
+                        xml_addsub(x_simple_patch, value_vec_tmp);
+                    }
+                }
+
+                // Send the POST request
+                cbuf* cb = cbuf_new();
+                xml2json_cbuf(cb, x_simple_patch, 1);
+                char *json_simple_patch = cbuf_get(cb);
+                ret = api_data_post(h, req, simple_patch_request_uri, pi, qvec, json_simple_patch, pretty, YANG_DATA_JSON, media_out, ds );
+                if (value_vec_tmp != NULL)
+                    free(value_vec_tmp);
+                free(x_simple_patch);
+                free(patch_header); // NULL check was already done before
+                if (ret != 0)
+                    goto done;
+                break;
+            }
+            // For "insert", make a api_data_post request
+            if (strcmp(op_val, "insert") == 0) {
+                char *json_simple_patch = malloc(temp_str_malloc_size);
+                if (json_simple_patch == NULL)
+                    goto done;
+                memset(json_simple_patch, 0, temp_str_malloc_size);
+
+                // Loop through the XML, and get each value
+                for (int k = 0; k < value_vec_len; k++) {
+                    if (value_vec[k] != NULL) {
+                        value_vec_tmp = xml_dup(value_vec[k]);
+                        xml_addsub(x_simple_patch, value_vec_tmp);
+                    }
+                }
+                cbuf* cb = cbuf_new();
+                xml2json_cbuf(cb, x_simple_patch, 1);
+                
+                // Some ugly text processing to get the JSON to match what api_data_post() expects
+                char *json_simple_patch_tmp = cbuf_get(cb);
+                int brace_count = 0;
+                for (int l = 0; l < strlen(json_simple_patch_tmp); l++) {
+                    char c = json_simple_patch_tmp[l];
+                    if (c == '{') {
+                        brace_count++;
+                        if (brace_count == 2) {
+                            json_simple_patch[strlen(json_simple_patch)] = '[';
+                        }
+                    }
+                    json_simple_patch[strlen(json_simple_patch)] = c;
+                }
+                for (int l = strlen(json_simple_patch); l>= 0; l--) {
+                    char c = json_simple_patch[l];
+                    if (c == '}') {
+                       json_simple_patch[l] = ']';
+                       json_simple_patch[l + 1] = '}';
+                       break;
+                    }
+                }
+
+                // Set the insert attributes
+                cvec* qvec_tmp = NULL;
+                qvec_tmp = cvec_new(0);
+                if (qvec_tmp == NULL)
+                    goto done;
+                cg_var *cv;
+                if ((cv = cvec_add(qvec_tmp, CGV_STRING)) == NULL){
+                    goto done;
+                }
+                cv_name_set(cv, "insert");
+                cv_string_set(cv, where_val);
+                char *point_str = malloc(temp_str_malloc_size);
+                if (point_str == NULL)
+                    goto done;
+                memset(point_str, 0, temp_str_malloc_size);
+                strcpy(point_str, api_path);
+                strcat(point_str, point_val);
+                if ((cv = cvec_add(qvec_tmp, CGV_STRING)) == NULL){
+                    goto done;
+                }
+                cv_name_set(cv, "point");
+                cv_string_set(cv, point_str);
+
+                // Send the POST request
+                ret = api_data_post(h, req, simple_patch_request_uri, pi, qvec_tmp, json_simple_patch, pretty, YANG_DATA_JSON, media_out, ds );
+                if (cb != NULL)
+                    cbuf_free(cb);
+                if (value_vec_tmp != NULL)
+                    free(value_vec_tmp);
+                free(point_str); // NULL check was already done above
+                free(json_simple_patch); // NULL check was already done above
+                free(patch_header); // NULL check was already done before
+                if (x_simple_patch != NULL)
+                    free(x_simple_patch);
+                break;
+            }
+
+            // For merge", make single simple patch requests for each value
+            if (strcmp(op_val,"merge") == 0) {
+                if (key_xn != NULL)
+                    xml_addsub(x_simple_patch, key_xn);
+    
+                char *json_simple_patch = malloc(temp_str_malloc_size);
+                if (json_simple_patch == NULL)
+                    goto done;
+
+                // Loop through the XML, create JSON from each one, and submit a simple patch
+                for (int k = 0; k < value_vec_len; k++) {
+                    if (value_vec[k] != NULL) {
+                        value_vec_tmp = xml_dup(value_vec[k]);
+                        xml_addsub(x_simple_patch, value_vec_tmp);
+                    }
+                    cbuf* cb = cbuf_new();
+                    xml2json_cbuf(cb, x_simple_patch, 1);
+
+                    // Some ugly text processing to get the JSON to match what api_data_write() expects for a simple patch
+                    char *json_simple_patch_tmp = cbuf_get(cb);
+                    memset(json_simple_patch, 0, temp_str_malloc_size);
+                    int brace_count = 0;
+                    for (int l = 0; l < strlen(json_simple_patch_tmp); l++) {
+                        char c = json_simple_patch_tmp[l];
+                        if (c == '{') {
+                            brace_count++;
+                            if (brace_count == 2) {
+                                json_simple_patch[strlen(json_simple_patch)] = '[';
+                            }
+                        }
+                        json_simple_patch[strlen(json_simple_patch)] = c;
+                    }
+                    for (int l = strlen(json_simple_patch); l>= 0; l--) {
+                        char c = json_simple_patch[l];
+                        if (c == '}') {
+                           json_simple_patch[l] = ']';
+                           json_simple_patch[l + 1] = '}';
+                           break;
+                        }
+                    }
+                    if (value_vec_tmp != NULL)
+                        free(value_vec_tmp);
+                    // Send the simple patch request
+                    ret = api_data_write(h, req, simple_patch_request_uri, pcvec, pi, qvec, json_simple_patch, pretty, YANG_DATA_JSON, media_out, plain_patch_val, ds );
+                    cbuf_free(cb);
+                }
+                free(json_simple_patch); // NULL check was already done above
+                free(patch_header); // NULL check was already done before
+                if (x_simple_patch != NULL)
+                    free(x_simple_patch);
+            }
+        }
+        if ((strcmp(op_val, "delete") == 0) ||
+            (strcmp(op_val, "remove") == 0)) {
+                strcat(simple_patch_request_uri, target_val);
+                if (strcmp(op_val, "delete") == 0) {
+                    // TODO - send error
+                } else {
+                    // TODO - do not send error
+                }
+                api_data_delete(h, req, simple_patch_request_uri, pi, pretty, YANG_DATA_JSON, ds); 
+        }
+        if (simple_patch_request_uri)
+            free(simple_patch_request_uri);
+        if (api_path_target)
+            free(api_path_target);
+    }
+ ok:
+    retval = 0;
+ done:
+    if (path_orig_1 != NULL)
+    free(path_orig_1);
+    if (vec)
+    free(vec);
+    if (xpath)
+	free(xpath);
+    if (nsc)
+	xml_nsctx_free(nsc);
+    if (xret)
+	xml_free(xret);
+    if (xerr)
+	xml_free(xerr);
+    if (xretcom)
+	xml_free(xretcom);
+    if (xretdis)
+	xml_free(xretdis);
+    if (xtop)
+	xml_free(xtop);
+    if (xdata0)
+	xml_free(xdata0);
+     if (cbx)
+	cbuf_free(cbx); 
+    return retval;
+}
+#endif // YANG_PATCH
+
 /*! Generic REST PUT  method 
  * @param[in]  h        Clixon handle
  * @param[in]  req      Generic Www handle
@@ -671,10 +1203,16 @@ api_data_patch(clicon_handle h,
 	ret = api_data_write(h, req, api_path0, pcvec, pi, qvec, data, pretty,
 			     media_in, media_out, 1, ds);
 	break;
-    case YANG_PATCH_XML:
     case YANG_PATCH_JSON: 	/* RFC 8072 patch */
+    case YANG_PATCH_XML:
+#ifdef YANG_PATCH
+	ret = api_data_yang_patch(h, req, api_path0, pcvec, pi, qvec, data, pretty,
+			     media_out, ds);
+#else
 	ret = restconf_notimplemented(h, req, pretty, media_out);
+#endif
 	break;
+    break;
     default:
 	ret = restconf_unsupported_media(h, req, pretty, media_out);
 	break;
@@ -753,11 +1291,12 @@ api_data_delete(clicon_handle h,
     /* For internal XML protocol: add username attribute for access control
      */
     username = clicon_username_get(h);
-    cprintf(cbx, "<rpc xmlns=\"%s\" username=\"%s\" xmlns:%s=\"%s\">",
+    cprintf(cbx, "<rpc xmlns=\"%s\" username=\"%s\" xmlns:%s=\"%s\" %s>",
 	    NETCONF_BASE_NAMESPACE,
 	    username?username:"",
 	    NETCONF_BASE_PREFIX,
-	    NETCONF_BASE_NAMESPACE); /* bind nc to netconf namespace */
+	    NETCONF_BASE_NAMESPACE,
+	    NETCONF_MESSAGE_ID_ATTR); /* bind nc to netconf namespace */
 
     cprintf(cbx, "<edit-config");
     /* RFC8040 Sec 1.4:

apps/restconf/restconf_methods_post.c

@@ -155,6 +155,7 @@ api_data_post(clicon_handle h,
 	      cvec         *qvec, 
 	      char         *data,
 	      int           pretty,
+	      restconf_media media_in,
 	      restconf_media media_out,
 	      ietf_ds_t     ds)
 {
@@ -178,7 +179,6 @@ api_data_post(clicon_handle h,
     cxobj         *x;            
     char          *username;
     int            ret;
-    restconf_media media_in;
     int            nrchildren0 = 0;
     yang_bind      yb;
     
@@ -231,7 +231,6 @@ api_data_post(clicon_handle h,
      * If xbot is top-level (api_path=null) it does not have a spec therefore look for 
      * top-level (yspec) otherwise assume parent (xbot) is populated.
      */
-    media_in = restconf_content_type(h);
     switch (media_in){
     case YANG_DATA_XML:
 	if ((ret = clixon_xml_parse_string(data, yb, yspec, &xbot, &xerr)) < 0){
@@ -336,11 +335,12 @@ api_data_post(clicon_handle h,
     /* For internal XML protocol: add username attribute for access control
      */
     username = clicon_username_get(h);
-    cprintf(cbx, "<rpc xmlns=\"%s\" username=\"%s\" xmlns:%s=\"%s\">",
+    cprintf(cbx, "<rpc xmlns=\"%s\" username=\"%s\" xmlns:%s=\"%s\" %s>",
 	    NETCONF_BASE_NAMESPACE,
 	    username?username:"",
 	    NETCONF_BASE_PREFIX,
-	    NETCONF_BASE_NAMESPACE); /* bind nc to netconf namespace */
+	    NETCONF_BASE_NAMESPACE,
+	    NETCONF_MESSAGE_ID_ATTR); /* bind nc to netconf namespace */
 
     cprintf(cbx, "<edit-config");
     /* RFC8040 Sec 1.4:
@@ -755,13 +755,13 @@ api_operations_post(clicon_handle h,
      * <rpc username="foo"><myfn xmlns="uri"/>
      */
     if ((username = clicon_username_get(h)) != NULL){
-	if (clixon_xml_parse_va(YB_NONE, NULL, &xtop, NULL, "<rpc xmlns=\"%s\" username=\"%s\"/>",
+	if (clixon_xml_parse_va(YB_NONE, NULL, &xtop, NULL, "<rpc xmlns=\"%s\" username=\"%s\" %s/>",
-				NETCONF_BASE_NAMESPACE, username) < 0)
+				NETCONF_BASE_NAMESPACE, username, NETCONF_MESSAGE_ID_ATTR) < 0)
 	    goto done;
     }
     else
-	if (clixon_xml_parse_va(YB_NONE, NULL, &xtop, NULL, "<rpc xmlns=\"%s\"/>",
+	if (clixon_xml_parse_va(YB_NONE, NULL, &xtop, NULL, "<rpc xmlns=\"%s\" %s/>",
-				NETCONF_BASE_NAMESPACE) < 0)
+				NETCONF_BASE_NAMESPACE, NETCONF_MESSAGE_ID_ATTR) < 0)
 	    goto done;
     if (xml_rootchild(xtop, 0, &xtop) < 0)
 	goto done;

apps/restconf/restconf_methods_post.h

@@ -44,6 +44,7 @@
 int api_data_post(clicon_handle h, void *req, char *api_path,
 		  int pi, cvec *qvec, char *data,
 		  int pretty,
+		  restconf_media media_in, 
 		  restconf_media media_out, ietf_ds_t ds);
 
 int api_operations_post(clicon_handle h, void *req, char *api_path,

apps/restconf/restconf_native.c

@@ -177,6 +177,14 @@ restconf_conn_free(restconf_conn *rc)
 	clicon_err(OE_RESTCONF, EINVAL, "rc is NULL");
 	return -1;
     }
+#ifdef HAVE_LIBNGHTTP2
+    if (rc->rc_ngsession)
+	nghttp2_session_del(rc->rc_ngsession);
+#endif
+#ifdef HAVE_LIBEVHTP
+    if (rc->rc_evconn)
+	evhtp_connection_free(rc->rc_evconn); /* evhtp */
+#endif
     /* Free all streams */
     while ((sd = rc->rc_streams) != NULL) {
 	DELQ(sd, rc->rc_streams,  restconf_stream_data *);

apps/restconf/restconf_nghttp2.c

@@ -182,15 +182,23 @@ session_send_callback(nghttp2_session *session,
     ssize_t        totlen = 0;
     int            s;
     SSL           *ssl; 
+    int            sslerr;
     
-    clicon_debug(1, "%s buflen:%lu", __FUNCTION__, buflen);
+    clicon_debug(1, "%s buflen:%zu", __FUNCTION__, buflen);
     s = rc->rc_s;
     ssl = rc->rc_ssl;
     while (totlen < buflen){
 	if (ssl){
 	    if ((len = SSL_write(ssl, buf+totlen, buflen-totlen)) <= 0){
 		er = errno;
-		switch (SSL_get_error(ssl, len)){
+		sslerr = SSL_get_error(ssl, len);
+		clicon_debug(1, "%s errno:;%d sslerr:%d", __FUNCTION__, errno, sslerr);
+		switch (sslerr){
+		case SSL_ERROR_WANT_WRITE:           /* 3 */
+		    clicon_debug(1, "%s write SSL_ERROR_WANT_WRITE", __FUNCTION__);
+		    usleep(1000);
+		    continue;
+		    break;
 		case SSL_ERROR_SYSCALL:              /* 5 */
 		    if (er == ECONNRESET) {/* Connection reset by peer */
 			if (ssl)
@@ -200,8 +208,12 @@ session_send_callback(nghttp2_session *session,
 			goto ok; /* Close socket and ssl */
 		    }
 		    else if (er == EAGAIN){
+			/* same as want_write above, but different behaviour on different 
+			 * platforms, linux here, freebsd want_write, or possibly differnt
+			 * ssl lib versions?
+			 */
 			clicon_debug(1, "%s write EAGAIN", __FUNCTION__);
-			usleep(10000);
+			usleep(1000);
 			continue;
 		    }
 		    else{
@@ -247,7 +259,7 @@ session_send_callback(nghttp2_session *session,
 	clicon_debug(1, "%s retval:%d", __FUNCTION__, retval);
 	return retval;
     }
-    clicon_debug(1, "%s retval:%lu", __FUNCTION__, totlen);
+    clicon_debug(1, "%s retval:%zd", __FUNCTION__, totlen);
     return totlen;
 }
 
@@ -367,7 +379,7 @@ restconf_sd_read(nghttp2_session     *session,
 #endif
     assert(cbuf_len(cb) > sd->sd_body_offset);
     remain = cbuf_len(cb) - sd->sd_body_offset;
-    clicon_debug(1, "%s length:%lu totlen:%d, offset:%lu remain:%lu",
+    clicon_debug(1, "%s length:%zu totlen:%d, offset:%zu remain:%zu",
 		 __FUNCTION__,
 		 length,
 		 cbuf_len(cb),
@@ -383,7 +395,7 @@ restconf_sd_read(nghttp2_session     *session,
     }
     memcpy(buf, cbuf_get(cb) + sd->sd_body_offset, len);
     sd->sd_body_offset += len;
-    clicon_debug(1, "%s retval:%lu", __FUNCTION__, len);
+    clicon_debug(1, "%s retval:%zu", __FUNCTION__, len);
     return len;
 }
 
@@ -397,7 +409,7 @@ restconf_submit_response(nghttp2_session      *session,
     nghttp2_data_provider data_prd;
     nghttp2_error         ngerr;
     cg_var               *cv;
-    nghttp2_nv           *hdrs;
+    nghttp2_nv           *hdrs = NULL;
     nghttp2_nv           *hdr;
     int                   i = 0;
     char                  valstr[16];
@@ -437,6 +449,8 @@ restconf_submit_response(nghttp2_session      *session,
     retval = 0;
  done:
     clicon_debug(1, "%s retval:%d", __FUNCTION__, retval);
+    if (hdrs)
+	free(hdrs);
     return retval;
 }
 
@@ -469,7 +483,7 @@ http2_exec(restconf_conn        *rc,
      * [RFC7231]).
      */
     if (sd->sd_code != 204 && sd->sd_code > 199)
-	if (restconf_reply_header(sd, "Content-Length", "%lu", sd->sd_body_len) < 0)
+	if (restconf_reply_header(sd, "Content-Length", "%zu", sd->sd_body_len) < 0)
 	    goto done;	
     if (sd->sd_code){
 	if (restconf_submit_response(session, rc, stream_id, sd) < 0)
@@ -821,6 +835,20 @@ on_extension_chunk_recv_callback(nghttp2_session *session,
     return 0;
 }
 
+/*! Library provides the error code, and message for debugging purpose.
+ */
+static int
+error_callback(nghttp2_session *session,
+	       const char *msg,
+	       size_t len,
+	       void *user_data)
+{
+    //    restconf_conn *rc = (restconf_conn *)user_data;
+    clicon_debug(1, "%s", __FUNCTION__);
+    return 0;
+}
+
+#if (NGHTTP2_VERSION_NUM > 0x011201) /* Unsure of version number */
 /*! Library provides the error code, and message for debugging purpose.
  */
 static int
@@ -835,9 +863,16 @@ error_callback2(nghttp2_session *session,
     clicon_err(OE_NGHTTP2, lib_error_code, "%s", msg);
     return 0;
 }
+#endif
 
-/*
+/*! Process an HTTP/2 request received in buffer, process request and send reply
- * XXX see session_recv
+ *
+ * @param[in] rc   Restconf connection
+ * @param[in] buf  Character buffer
+ * @param[in] n    Lenght of buf
+ * @retval    1    OK
+ * @retval    0    Invald request
+ * @retval   -1    Fatal error
  */
 int
 http2_recv(restconf_conn       *rc,
@@ -855,6 +890,18 @@ http2_recv(restconf_conn       *rc,
     }
     /* may make additional pending frames */
     if ((ngerr = nghttp2_session_mem_recv(rc->rc_ngsession, buf, n)) < 0){
+	if (ngerr == NGHTTP2_ERR_BAD_CLIENT_MAGIC){
+	    /* :enum:`NGHTTP2_ERR_BAD_CLIENT_MAGIC`
+	     *     Invalid client magic was detected.  This error only returns
+	     *     when |session| was configured as server and
+	     *     `nghttp2_option_set_no_recv_client_magic()` is not used with
+	     *     nonzero value. */
+	    clicon_log(LOG_INFO, "%s Received bad client magic byte strin", __FUNCTION__);
+	    /* unsure if this does anything, byt does not seem to hurt */
+	    if ((ngerr = nghttp2_session_terminate_session(rc->rc_ngsession, ngerr)) < 0)
+		clicon_err(OE_NGHTTP2, ngerr, "nghttp2_session_terminate_session %d", ngerr);
+	    goto fail;
+	}
 	clicon_err(OE_NGHTTP2, ngerr, "nghttp2_session_mem_recv");
 	goto done;
     }
@@ -866,9 +913,13 @@ http2_recv(restconf_conn       *rc,
 	clicon_err(OE_NGHTTP2, ngerr, "nghttp2_session_send");
 	goto done;
     }
-    retval = 0;
+    retval = 1; /* OK */
  done:
+    clicon_debug(1, "%s retval:%d", __FUNCTION__, retval);
     return retval;
+ fail:
+    retval = 0;
+    goto done;
 }
 
 /* Send HTTP/2 client connection header, which includes 24 bytes
@@ -930,10 +981,13 @@ http2_session_init(restconf_conn *rc)
     nghttp2_session_callbacks_set_unpack_extension_callback(callbacks, unpack_extension_callback);
 #endif
     nghttp2_session_callbacks_set_on_extension_chunk_recv_callback(callbacks, on_extension_chunk_recv_callback);
+    nghttp2_session_callbacks_set_error_callback(callbacks, error_callback);
+#if (NGHTTP2_VERSION_NUM > 0x011201) /* Unsure of version number */
     nghttp2_session_callbacks_set_error_callback2(callbacks, error_callback2);
+#endif
 
     /* Create session for server use, register callbacks */
-    if ((ngerr = nghttp2_session_server_new(&session, callbacks, rc)) < 0){
+    if ((ngerr = nghttp2_session_server_new3(&session, callbacks, rc, NULL, NULL)) < 0){
 	clicon_err(OE_NGHTTP2, ngerr, "nghttp2_session_server_new");
 	goto done;
     }

apps/restconf/restconf_root.c

@@ -331,7 +331,7 @@ api_data(clicon_handle h,
 	retval = api_data_get(h, req, api_path, pcvec, pi, qvec, pretty, media_out, ds);
     }
     else if (strcmp(request_method, "POST")==0) {
-	retval = api_data_post(h, req, api_path, pi, qvec, data, pretty, media_out, ds);
+	retval = api_data_post(h, req, api_path, pi, qvec, data, pretty, restconf_content_type(h), media_out, ds);
     }
     else if (strcmp(request_method, "PUT")==0) {
 	if (read_only) 
@@ -588,6 +588,10 @@ api_root_restconf(clicon_handle        h,
     retval = 0;
  done:
     clicon_debug(1, "%s retval:%d", __FUNCTION__, retval);
+#ifdef WITH_RESTCONF_FCGI
+    if (cb)
+	cbuf_free(cb);
+#endif
     if (xerr)
 	xml_free(xerr);
     if (username)

apps/restconf/restconf_stream_fcgi.c

@@ -269,8 +269,8 @@ restconf_stream(clicon_handle h,
 	clicon_err(OE_XML, errno, "cbuf_new");
 	goto done;
     }
-    cprintf(cb, "<rpc xmlns=\"%s\"><create-subscription xmlns=\"%s\"><stream>%s</stream>",
+    cprintf(cb, "<rpc xmlns=\"%s\" %s><create-subscription xmlns=\"%s\"><stream>%s</stream>",
-	    NETCONF_BASE_NAMESPACE, EVENT_RFC5277_NAMESPACE, name);
+	    NETCONF_BASE_NAMESPACE, NETCONF_MESSAGE_ID_ATTR, EVENT_RFC5277_NAMESPACE, name);
     /* Print all fields */
     for (i=0; i<cvec_len(qvec); i++){
         cv = cvec_i(qvec, i);

configure

@@ -1378,6 +1378,7 @@ Optional Packages:
   --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
   --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
   --with-cligen=dir       Use CLIGEN installation in this dir
+  --with-restconf=native  Integration with embedded web server (DEFAULT)
   --with-restconf=fcgi    FCGI interface for stand-alone web rev-proxy eg
                           nginx (default)
   --with-restconf=native  Integrate restconf with embedded http server
@@ -2264,7 +2265,7 @@ ac_configure="$SHELL $ac_aux_dir/configure"  # Please don't use this var.
 
 
 CLIXON_VERSION_MAJOR="5"
-CLIXON_VERSION_MINOR="2"
+CLIXON_VERSION_MINOR="3"
 CLIXON_VERSION_PATCH="0"
 CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}.PRE\""
 
@@ -4995,15 +4996,17 @@ fi
 # This is for restconf.  There are three options:
 # --without-restconf     No restconf support
 # --with-restconf=fcgi   FCGI interface for separate web reverse proxy like nginx
-# --with-restconf=native Integration with embedded web server
+# --with-restconf=native Integration with embedded web server (DEFAULT)
 
 # Check whether --with-restconf was given.
 if test "${with_restconf+set}" = set; then :
   withval=$with_restconf;
 else
-  with_restconf=fcgi
+  with_restconf=native
 fi
 
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: restconf mode ${with_restconf}" >&5
+$as_echo "restconf mode ${with_restconf}" >&6; }
 # Actions for each specific package
 if test "x${with_restconf}" == xfcgi; then
    # Lives in libfcgi-dev
@@ -5063,6 +5066,7 @@ cat >>confdefs.h <<_ACEOF
 _ACEOF
 
 elif test "x${with_restconf}" == xnative; then
+
    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for OPENSSL_init_ssl  in -lssl" >&5
 $as_echo_n "checking for OPENSSL_init_ssl  in -lssl... " >&6; }
 if ${ac_cv_lib_ssl_OPENSSL_init_ssl_+:} false; then :

configure.ac

@@ -49,7 +49,7 @@ AC_INIT(lib/clixon/clixon.h.in)
 AC_CONFIG_AUX_DIR(config-aux)
 
 CLIXON_VERSION_MAJOR="5"
-CLIXON_VERSION_MINOR="2"
+CLIXON_VERSION_MINOR="3"
 CLIXON_VERSION_PATCH="0"
 CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}.PRE\""
 
@@ -206,11 +206,12 @@ AC_CHECK_LIB(cligen, cligen_init,, AC_MSG_ERROR([CLIgen missing. Try: git clone
 # This is for restconf.  There are three options:
 # --without-restconf     No restconf support
 # --with-restconf=fcgi   FCGI interface for separate web reverse proxy like nginx
-# --with-restconf=native Integration with embedded web server
+# --with-restconf=native Integration with embedded web server (DEFAULT)
 AC_ARG_WITH([restconf],
-	AS_HELP_STRING([--with-restconf=fcgi],[FCGI interface for stand-alone web rev-proxy eg nginx (default)]),
+	AS_HELP_STRING([--with-restconf=native],[Integration with embedded web server (DEFAULT)]),
 	,
-	[with_restconf=fcgi])
+	[with_restconf=native])
+AC_MSG_RESULT(restconf mode ${with_restconf})
 # Actions for each specific package
 if test "x${with_restconf}" == xfcgi; then
    # Lives in libfcgi-dev
@@ -218,6 +219,7 @@ if test "x${with_restconf}" == xfcgi; then
    AC_DEFINE(WITH_RESTCONF_FCGI, 1, [Use fcgi restconf mode]) # For c-code that cant use strings
    AC_DEFINE_UNQUOTED(WWWDIR, "$wwwdir", [WWW dir for fcgi stuff / nginx])
 elif test "x${with_restconf}" == xnative; then
+
    AC_CHECK_LIB(ssl, OPENSSL_init_ssl ,, AC_MSG_ERROR([libssl missing]))
    AC_CHECK_LIB(crypto, CRYPTO_new_ex_data, , AC_MSG_ERROR([libcrypto missing])) 
    # Check if evhtp is enabled for http/1

docker/main/Dockerfile.native

@@ -41,10 +41,12 @@ RUN apk add --update git make build-base gcc flex bison curl-dev
 # Create a directory to hold source-code, dependencies etc
 RUN mkdir /clixon
 
-# evhtp
+# evhtp dependencies
-# dependencies
 RUN apk add --update libevent libevent-dev
 
+# nghttp2 dependencies
+RUN apk add --update nghttp2
+
 # clone libevhtp
 WORKDIR /clixon
 
@@ -68,14 +70,14 @@ RUN ./configure --prefix=/clixon/build
 RUN make
 RUN make install
 
+# Need to add www user manually
+RUN adduser -D -H -G www-data www-data
+
 # Copy Clixon from local dir
 RUN mkdir /clixon/clixon
 WORKDIR /clixon/clixon
 COPY clixon .
 
-# Need to add www user manually
-RUN adduser -D -H -G www-data www-data
-
 # Configure, build and install clixon
 RUN ./configure --prefix=/clixon/build --with-cligen=/clixon/build --enable-optyangs --with-restconf=native --enable-nghttp2 --enable-evhtp
 
@@ -117,6 +119,9 @@ RUN adduser -D -H -G www-data www-data
 # for libevtp
 RUN apk add --update openssl libevent
 
+# nghttp2 dependencies
+RUN apk add --update nghttp2
+
 # Test-specific (for test scripts)
 RUN apk add --update sudo curl procps grep make bash # iproute2 # contains ip - but CAP_SYS_ADMIN isssue
 
@@ -135,5 +140,5 @@ RUN mkdir /www-data
 RUN chown clicon /www-data
 RUN chgrp clicon /www-data
 
-# Log to stderr.
+# Start the backend and restconf deamons
 CMD /usr/local/bin/startsystem.sh

example/README.md

@@ -1,7 +1,7 @@
 # Clixon examples
 
-See the separate
+See also the separate
-[clixon-examples](https://github.com/clicon/clixon-examples) repo.
+[clixon-examples](https://github.com/clicon/clixon-examples) repo
 
 The only Clixon example remaining is for internal testing:
   * [Main example](main/README.md)

example/main/Makefile.in

@@ -101,6 +101,7 @@ BE_SRC   = $(APPNAME)_backend.c
 BE_OBJ   = $(BE_SRC:%.c=%.o)
 $(BE_PLUGIN): $(BE_OBJ)
 ifeq ($(LINKAGE),static)
+# can include -L in LDFLAGS?
 	$(CC) -Wall -shared $(LDFLAGS) -o $@ -lc $< -lclixon -L ../../apps/backend/ -lclixon_backend 
 else
 	$(CC) -Wall -shared $(LDFLAGS) -o $@ -lc $< -lclixon -lclixon_backend

example/main/README.md

@@ -1,5 +1,6 @@
 # Clixon main example
 
+  * [Background](#background)
   * [Content](#content)
   * [Compile and run](#compile)
   * [Using the CLI](#using-the-cli)
@@ -13,40 +14,45 @@
   * [Docker](#docker)
   * [Plugins](#plugins)
   
+
+## Background
+
+The aim of the main clixon example is to illustrate common features
+and for internal testing.  See the simpler [hello world](https://github.com/clicon/clixon-examples/tree/master/hello) if you want to start from the simplest possible example.
+
+See also other examples in: [clixon-examples](https://github.com/clicon/clixon-examples).
+
 ## Content
 
-This directory contains a Clixon example used primarily as a part of the Clixon test suites. It can be used as a basis for making new Clixon applications. But please consider also the minimal [hello](../hello) example as well. It contains the following files:
+This directory contains a Clixon example used primarily as a part of the Clixon test suites. It can be used as a basis for making new Clixon applications.  It contains the following files:
-* `example.xml`       The configuration file. See [yang/clixon-config@<date>.yang](../../yang/clixon-config@2019-03-05.yang) for the documentation of all available fields.
+* `clixon-example@2020-12-20.yang`      The yang spec of the example.
-* `clixon-example@2019-01-13.yang`      The yang spec of the example.
+* `example_backend.c`      Backend callback plugin including example of:
-* `example_cli.cli`   CLIgen specification.
-* `example_cli.c`     CLI callback plugin containing functions called in the cli file above: a generic callback (`mycallback`) and an example RPC call (`example_client_rpc`).
-* `example_backend.c` Backend callback plugin including example of:
-  * transaction callbacks (validate/commit),
-  * notification,
-  * rpc handler
-  * state-data handler, ie non-config data
 * `example_backend_nacm.c` Secondary backend plugin. Plugins are loaded alphabetically.
-* `example_restconf.c` Restconf callback plugin containing an HTTP basic authentication callback
+* `example_cli.c`          CLI callback plugin containing functions called in the .cli file
-* `example_netconf.c` Netconf callback plugin
+* `example_cli.cli`        CLIgen specification of example CLI commands
-* `Makefile.in`       Example makefile where plugins are built and installed
+* `example_netconf.c`      Netconf callback plugin
+* `example_restconf.c`     Restconf callback plugin containing HTTP basic authentication
+* `example.xml`            Main configuration file.
+* `Makefile.in`            Example makefile where plugins are built and installed
+
+See [yang/clixon-config@<date>.yang](https://github.com/clicon/clixon/blob/master/yang/clixon/clixon-config%402021-05-20.yang) for documentation of all available fields in `example.xml`.
 
 ## Compile and run
 
 Before you start,
-* You must configure with: `--enable-optyangs` to run the main example.
+* You must configure with: `--enable-optyangs` to install all yang files required for the example. This is not necessary for the base colixon system
-* Make [group setup](../../doc/FAQ.md#do-i-need-to-setup-anything-important)
+* Setup clicon [groups](https://github.com/clicon/clixon/blob/master/doc/FAQ.md#do-i-need-to-setup-anything)
-* Setup [restconf](../../doc/FAQ.md#how-do-i-use-restconf)
-
 
 ```
-    cd example
+    cd example/main
     make && sudo make install
 ```
+
 Start backend:
 ```
     sudo clixon_backend -f /usr/local/etc/example.xml -s init
 ```
-Edit cli:
+Start cli:
 ```
     clixon_cli -f /usr/local/etc/example.xml
 ```
@@ -54,13 +60,13 @@ Send netconf command:
 ```
     clixon_netconf -f /usr/local/etc/example.xml
 ```
-Start clixon restconf daemon
+Start clixon restconf daemon (default config listens on http IPv4 0.0.0.0 on port 8080):
 ```
-    sudo /usr/local/bin/clixon_restconf -f /usr/local/etc/example.xml
+    sudo clixon_restconf -f /usr/local/etc/example.xml
 ```
 Send restconf command
 ```
-    curl -X GET http://127.0.0.1/restconf/data
+    curl -X GET http://127.0.0.1:8080/restconf/data
 ```
 
 ## Using the CLI
@@ -71,53 +77,139 @@ There are also many other commands available as examples. View the source file (
 The following example shows how to add an interface in candidate, validate and commit it to running, then look at it (as xml) and finally delete it.
 ```
 clixon_cli -f /usr/local/etc/example.xml 
-cli> set interfaces interface eth9 ?
+cli> set interfaces interface eth1 ?
- description               enabled                   ipv4                     
+  <cr>
- ipv6                      link-up-down-trap-enable  type                     
+  description           A textual description of the interface.
-cli> set interfaces interface eth9 type ex:eth
+  enabled               This leaf contains the configured, desired state of the
+                        interface.
+  ipv4                  Parameters for the IPv4 address family.
+  ipv6                  Parameters for the IPv6 address family.
+  type                  The type of the interface.
+cli> set interfaces interface eth1 type ianaift:ip
+cli> set interfaces interface eth1 enabled true
+cli> set interfaces interface eth1 ipv4 address 1.2.3.4 prefix-length 24
 cli> validate 
 cli> commit 
 cli> show configuration xml 
 <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
    <interface>
-      <name>eth9</name>
+      <name>eth1</name>
-      <type>ex:eth</type>
+      <type>ianaift:ip</type>
       <enabled>true</enabled>
+      <ip:ipv4 xmlns:ip="urn:ietf:params:xml:ns:yang:ietf-ip">
+         <ip:enabled>true</ip:enabled>
+         <ip:forwarding>false</ip:forwarding>
+         <ip:address>
+            <ip:ip>1.2.3.4</ip:ip>
+            <ip:prefix-length>24</ip:prefix-length>
+         </ip:address>
+      </ip:ipv4>
    </interface>
 </interfaces>
-cli> delete interfaces interface eth9
+cli> delete interfaces interface eth1
+cli> commit
 ```
 
 ## Using Netconf
 
 The following example shows how to set data using netconf:
 ```
-<rpc><edit-config><target><candidate/></target><config>
+sh> clixon_netconf -qf /usr/local/etc/example.xml
-      <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
+<?xml version="1.0" encoding="UTF-8"?>
-         <interface>
+<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><capabilities><capability>urn:ietf:params:netconf:base:1.1</capability></capabilities></hello>]]>]]>
-            <name>eth1</name>
+<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="0">
-            <enabled>true</enabled>
+   <edit-config>
-            <ipv4>
+      <target><candidate/></target>
-               <address>
+      <config>
-                  <ip>9.2.3.4</ip>
+         <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
-                  <prefix-length>24</prefix-length>
+            <interface>
-               </address>
+               <name>eth1</name>
-            </ipv4>
+	       <type>ianaift:ip</type>
-         </interface>
+               <enabled>true</enabled>
-      </interfaces>
+               <ipv4 xmlns="urn:ietf:params:xml:ns:yang:ietf-ip">
-</config></edit-config></rpc>]]>]]>
+                  <address>
+                     <ip>1.2.3.4</ip>
+                     <prefix-length>24</prefix-length>
+                  </address>
+               </ipv4>
+            </interface>
+         </interfaces>
+      </config>
+   </edit-config>
+</rpc>]]>]]>
+# Reply: <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="0"><ok/></rpc-reply>]]>]]>
+<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
+   <commit/>
+</rpc>]]>]]>
 ```
 
-### Getting data using netconf
+Getting data:
 ```
-<rpc><get-config><source><candidate/></source></get-config></rpc>]]>]]>
+# Reply: <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1"><ok/></rpc-reply>]]>]]>
-<rpc><get-config><source><candidate/></source><filter/></get-config></rpc>]]>]]>
+<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2">
-<rpc><get-config><source><candidate/></source><filter type="xpath"/></get-config></rpc>]]>]]>
+   <get-config>
-<rpc><get-config><source><candidate/></source><filter type="subtree"><data><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces"><interface><name>eth9</name><type>ex:eth</type></interface></interfaces></data></filter></get-config></rpc>]]>]]>
+      <source><candidate/></source>
-<rpc><get-config><source><candidate/></source><filter type="xpath" select="/interfaces/interface"/></get-config></rpc>]]>]]>
+   </get-config>
-<rpc><validate><source><candidate/></source></validate></rpc>]]>]]>
+</rpc>]]>]]>
+# Reply: <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2"><data><interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces"><interface><name>eth1</name><type>ianaift:ip</type><enabled>true</enabled><ip:ipv4 xmlns:ip="urn:ietf:params:xml:ns:yang:ietf-ip"><ip:enabled>true</ip:enabled><ip:forwarding>false</ip:forwarding><ip:address><ip:ip>1.2.3.4</ip:ip><ip:prefix-length>24</ip:prefix-length></ip:address></ip:ipv4></interface></interfaces></data></rpc-reply>]]>]]>
+```
+
+Examples of a filtered GET statement:
+```
+<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1"><get-config><source><candidate/></source><filter type="xpath" select="/if:interfaces/if:interface[if:name='eth1']" xmlns:if="urn:ietf:params:xml:ns:yang:ietf-interfaces"/></get-config></rpc>]]>]]>
+```
+
+## Restconf 
+
+By default clixon from release 5.3 uses "native" restconf, see next
+section for an alternative. General clixon [restconf
+documentation](https://clixon-docs.readthedocs.io/en/latest/restconf.html). By
+default restconf supports http/1.1 and http/2 with the standard way
+(ALPN vs switch protocol) of selecting and upgrading from 1.1 to 2.
+
+In the example, a restconf config is included in the [config file](example.xml):
+```
+  <restconf>
+     <enable>true</enable>
+     <auth-type>none</auth-type>
+     <socket>
+        <namespace>default</namespace>
+	<address>0.0.0.0</address>
+	<port>80</port>
+	<ssl>false</ssl>
+     </socket>
+  </restconf>
+```
+
+In this example, a listening socket is opened using http on port 80. You can extend the restconf config by modifying the entry or add multiple `<socket>` entries, such as IPv6, TLS and another network namespace, for example:
+```
+   <socket>
+      <namespace>dataplane</namespace>
+      <address>::</address>
+      <port>443</port>
+      <ssl>true</ssl>
+   </socket>
+```
+
+For TLS, cert files need to be given, such as follows:
+```
+<restconf>
+   ...
+   <server-cert-path>/path/to/server/cert</server-cert-path>
+   <server-key-path>/path/to/server/key</server-key-path>
+   <server-ca-cert-path>/path/to/ca/cert</server-ca-cert-path>
+```
+
+For more info, such as client-certs, authentication, etc, see: [restconf documentation](https://clixon-docs.readthedocs.io/en/latest/restconf.html)
+
+## Restconf using nginx
+
+Alternatively, restconf can use a reverse-proxy such as nginx.
+
+Configure:
+```
+  ./configure --with-restconf=fcgi
 ```
-## Restconf
 
 Setup a web/reverse-proxy server.
 For example, using nginx, install, and edit config file: /etc/nginx/sites-available/default:
@@ -141,11 +233,13 @@ server {
         }
 }
 ```
+
 Start nginx daemon
 ```
    sudo /etc/init.d/nginx start
    sudo systemctl start nginx.service # alternative using systemd
 ```
+
 Start the clixon restconf daemon
 ```
    sudo /usr/local/sbin/clixon_restconf -f /usr/local/etc/example.xml
@@ -155,8 +249,6 @@ then access using curl or wget:
    curl -X GET http://127.0.0.1/restconf/data/ietf-interfaces:interfaces/interface=eth1/type
 ```
 
-More info: (restconf)[../../apps/restconf/README.md].
-
 ## Streams
 
 The example has an EXAMPLE stream notification triggering every 5s. To start a notification 
@@ -181,9 +273,6 @@ cli> no notify
 cli>
 ```
 
-Restconf support is also supported, see (restconf)[../../apps/restconf/README.md].
-
-
 ## RPC Operations
 
 Clixon implements Yang RPC operations by a mechanism that enables you
@@ -307,8 +396,7 @@ Example systemd files for backend and restconf daemons are found under the [syst
 
 ## Docker
 
-See [docker](../../docker/system) for instructions on how to build this example
+See [clixon docker main example](../../docker/main) for instructions on how to build this example as a docker container.
-as a docker container.
 
 ## Plugins
 

example/main/clixon-example@2020-03-11.yang

@@ -1,185 +0,0 @@
-module clixon-example {
-    yang-version 1.1;
-    namespace "urn:example:clixon";
-    prefix ex;
-    description
-	"Clixon example used as a part of the Clixon test suite.
-         It can be used as a basis for making new Clixon applications.";
-    revision 2020-03-11 {
-	description "Added container around translation list. Released in Clixon 4.4.0";
-    }
-    revision 2019-11-05 {
-	description "Augment interface. Released in Clixon 4.3.0";
-    }
-    revision 2019-07-23 {
-	description "Extension e4. Released in Clixon 4.1.0";
-    }
-    revision 2019-01-13 {
-	description "Released in Clixon 3.9";
-    }
-    import ietf-interfaces {
-	prefix if;
-    }
-    import ietf-ip {
-	prefix ip;
-    }
-    import iana-if-type {
-      prefix ianaift;
-    }
-    /* Example interface type for tests, local callbacks, etc */
-    identity eth {
-	base if:interface-type;
-    }
-    identity loopback {
-	base if:interface-type;
-    }
-    /* Translation function example - See also example_cli */
-    container translate{
-	description "dont have lists directly under top since restconf cant address list directly";
-	list translate{
-	    key k;
-	    leaf k{
-		type string;
-	    }
-	    leaf value{
-		type string;
-	    }
-	}
-    }
-    /* State data (not config) for the example application*/
-    container state {
-         config false;
-         description "state data for the example application (must be here for example get operation)";
-         leaf-list op {
-            type string;
-         }
-    }
-    augment "/if:interfaces/if:interface" {
-	container my-status {
-	    config false;
-	    description "For testing augment+state";
-	    leaf int {
-		type int32;
-	    }
-	    leaf str {
-		type string;
-	    }
-	}
-    }
-    /* yang extension implemented by the example backend code. */
-    extension e4 {
-       description
-	   "The first child of the ex:e4 (unknown) statement is inserted into 
-	    the module as a regular data statement. This means that 'uses bar;'
-	    in the ex:e4 statement below is a valid data node";
-       argument arg;
-    }
-    grouping bar {
-      leaf bar{
-        type string;
-      }
-    }
-    ex:e4 arg1{
-      uses bar;
-    }
-
-    /* Example notification as used in RFC 5277 and RFC 8040 */
-    notification event {
-         description "Example notification event.";
-         leaf event-class {
-           type string;
-           description "Event class identifier.";
-         }
-         container reportingEntity {
-           description "Event specific information.";
-           leaf card {
-             type string;
-             description "Line card identifier.";
-           }
-         }
-         leaf severity {
-           type string;
-           description "Event severity description.";
-         }
-    }
-    rpc client-rpc {
-	description "Example local client-side RPC that is processed by the
-                     the netconf/restconf and not sent to the backend.
-                     This is a clixon implementation detail: some rpc:s
-                     are better processed by the client for API or perf reasons";
-	input {
-	    leaf x {
-		type string;
-	    }
-	}
-	output {
-	    leaf x {
-		type string;
-	    }
-	}
-    }
-    rpc empty {
-	description "Smallest possible RPC with no input or output sections";
-    }
-    rpc optional {
-	description "Small RPC with optional input and output";
-	input {
-	    leaf x {
-		type string;
-	    }
-	}
-	output {
-	    leaf x {
-		type string;
-	    }
-	}
-    }
-    rpc example {
-	description "Some example input/output for testing RFC7950 7.14.
-                     RPC simply echoes the input for debugging.";
-	input {
-	    leaf x {
-		description
-         	    "If a leaf in the input tree has a 'mandatory' statement with
-                   the value 'true', the leaf MUST be present in an RPC invocation.";
-		type string;
-		mandatory true;
-	    }
-	    leaf y {
-		description
-                 "If a leaf in the input tree has a 'mandatory' statement with the
-                  value 'true', the leaf MUST be present in an RPC invocation.";
-		type string;
-		default "42";
-	    }
-	    leaf-list z {
-		description
-		    "If a leaf-list in the input tree has one or more default 
-                     values, the server MUST use these values (XXX not supported)";
-		type string;
-	    }
-	    leaf w {
-		description
-		    "If any node has a 'when' statement that would evaluate to 
-                    'false',then this node MUST NOT be present in the input tree.
-                     (XXX not supported)";
-		type string;
-		when "/translate/k=5/value='w'";
-	    }
-	}
-	output {
-	    leaf x {
-		type string;
-	    }
-	    leaf y {
-		type string;
-	    }
-	    leaf z {
-		type string;
-	    }
-	    leaf w {
-		type string;
-	    }
-	}
-    }
-}

example/main/example.xml

@@ -1,6 +1,8 @@
 <clixon-config xmlns="http://clicon.org/config">
   <CLICON_CONFIGFILE>/usr/local/etc/example.xml</CLICON_CONFIGFILE>
   <CLICON_FEATURE>ietf-netconf:startup</CLICON_FEATURE>
+  <CLICON_FEATURE>clixon-restconf:allow-auth-none</CLICON_FEATURE>
+  <CLICON_FEATURE>clixon-restconf:fcgi</CLICON_FEATURE>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_MODULE_MAIN>clixon-example</CLICON_YANG_MODULE_MAIN>
   <CLICON_CLI_MODE>example</CLICON_CLI_MODE>
@@ -14,11 +16,12 @@
   <CLICON_CLI_GENMODEL_COMPLETION>1</CLICON_CLI_GENMODEL_COMPLETION>
   <CLICON_CLI_GENMODEL_TYPE>VARS</CLICON_CLI_GENMODEL_TYPE>
   <CLICON_CLI_AUTOCLI_EXCLUDE>clixon-restconf</CLICON_CLI_AUTOCLI_EXCLUDE>
-  <CLICON_XMLDB_DIR>/usr/local/var/example</CLICON_XMLDB_DIR>
   <CLICON_CLI_LINESCROLLING>0</CLICON_CLI_LINESCROLLING>
   <CLICON_CLI_TAB_MODE>0</CLICON_CLI_TAB_MODE>
+  <CLICON_XMLDB_DIR>/usr/local/var/example</CLICON_XMLDB_DIR>
   <CLICON_STARTUP_MODE>init</CLICON_STARTUP_MODE>
   <CLICON_NACM_MODE>disabled</CLICON_NACM_MODE>
   <CLICON_STREAM_DISCOVERY_RFC5277>true</CLICON_STREAM_DISCOVERY_RFC5277>
   <CLICON_MODULE_LIBRARY_RFC7895>false</CLICON_MODULE_LIBRARY_RFC7895>
+  <restconf><enable>true</enable><auth-type>none</auth-type><socket><namespace>default</namespace><address>0.0.0.0</address><port>8081</port><ssl>false</ssl></socket></restconf>
 </clixon-config>

example/main/example_backend.c

@@ -1012,6 +1012,7 @@ example_exit(clicon_handle h)
     return 0;
 }
 
+/* Forward declaration */
 clixon_plugin_api *clixon_plugin_init(clicon_handle h);
 
 static clixon_plugin_api api = {

include/clixon_custom.h

@@ -108,7 +108,12 @@
  * added to its parent but then it is more difficult to check trhe when condition.
  * This fix add the parent x0p as a "candidate" so that the xpath-eval function can use it as
  * an alernative if it exists.
- * Note although this solves many usecases involving parents and absolute paths, itstill does not
+ * Note although this solves many usecases involving parents and absolute paths, it still does not
  * solve all usecases, such as absolute usecases where the added node is looked for
  */
 #define XML_PARENT_CANDIDATE
+
+/*! Enable yang patch RFC 8072 
+ * Remove this when regression test
+ */
+#undef YANG_PATCH

lib/clixon/clixon_netconf_lib.h

@@ -52,6 +52,10 @@
 #define NETCONF_BASE_NAMESPACE "urn:ietf:params:xml:ns:netconf:base:1.0"
 #define NETCONF_BASE_PREFIX "nc"
 
+/* In cases where message-id is not given by external client, use this */
+#define NETCONF_MESSAGE_ID_DEFAULT "42"
+#define NETCONF_MESSAGE_ID_ATTR "message-id=\"42\""
+
 /* Netconf base capability as defined in RFC4741, Sec 8.1
  */
 #define NETCONF_BASE_CAPABILITY_1_0 "urn:ietf:params:netconf:base:1.0"
@@ -97,6 +101,7 @@ int netconf_invalid_value(cbuf *cb, char *type, char *message);
 int netconf_invalid_value_xml(cxobj **xret, char *type, char *message);
 int netconf_too_big(cbuf *cb, char *type, char *message);
 int netconf_missing_attribute(cbuf *cb,	char *type, char *info, char *message);
+int netconf_missing_attribute_xml(cxobj **xret, char *type, char *info, char *message);
 int netconf_bad_attribute(cbuf *cb, char *type, char *info, char *message);
 int netconf_bad_attribute_xml(cxobj **xret, char *type, char *info, char *message);
 int netconf_unknown_attribute(cbuf *cb,	char *type, char *info, char *message);
@@ -122,8 +127,7 @@ int netconf_operation_failed(cbuf *cb, char *type, char *message);
 int netconf_operation_failed_xml(cxobj **xret, char *type, char *message);
 int netconf_malformed_message(cbuf *cb, char *message);
 int netconf_malformed_message_xml(cxobj **xret, char *message);
-int netconf_data_not_unique_xml(cxobj **xret, cxobj *x,	cvec *cvk);
+int netconf_data_not_unique_xml(cxobj **xret, cxobj *x, cvec *cvk);
-
 int netconf_minmax_elements_xml(cxobj **xret, cxobj *xp, char *name, int max);
 int netconf_trymerge(cxobj *x, yang_stmt *yspec, cxobj **xret);
 int netconf_module_features(clicon_handle h);

lib/clixon/clixon_xml_bind.h

@@ -44,6 +44,7 @@
  * Prototypes
  */
 int xml_bind_yang_unknown_anydata(int val);
+int xml_bind_netconf_message_id_optional(int val);
 int xml_bind_yang_rpc(cxobj *xrpc, yang_stmt *yspec, cxobj **xerr);
 int xml_bind_yang_rpc_reply(cxobj *xrpc, char *name, yang_stmt *yspec, cxobj **xerr);
 int xml_bind_yang0(cxobj *xt, yang_bind yb, yang_stmt *yspec, cxobj **xerr);

lib/clixon/clixon_yang.h

@@ -53,8 +53,12 @@
  */
 #define YANG_FLAG_MARK  0x01  /* (Dynamic) marker for dynamic algorithms, eg expand and DAG */
 #define YANG_FLAG_TMP   0x02  /* (Dynamic) marker for dynamic algorithms, eg DAG detection */
+#define YANG_FLAG_NOKEY 0x04  /* Key not mandatory in this list, see eg yang-data extension in
+			       * RFC 8040 / ietf-restconf.yang
+			       * see restconf_main_extension_cb
+			       */
 #ifdef XML_EXPLICIT_INDEX
-#define YANG_FLAG_INDEX 0x04  /* This yang node under list is (extra) index. --> you can access
+#define YANG_FLAG_INDEX 0x08  /* This yang node under list is (extra) index. --> you can access
 			       * list elements using this index with binary search */
 #endif
 
@@ -198,6 +202,7 @@ char      *yang_argument_get(yang_stmt *ys);
 int        yang_argument_set(yang_stmt *ys, char *arg);
 
 cg_var    *yang_cv_get(yang_stmt *ys);
+int        yang_cv_set(yang_stmt *ys, cg_var *cv);
 cvec      *yang_cvec_get(yang_stmt *ys);
 int        yang_cvec_set(yang_stmt *ys, cvec *cvv);
 uint16_t   yang_flag_get(yang_stmt *ys, uint16_t flag);

lib/src/clixon_client.c

@@ -170,9 +170,10 @@ clixon_client_lock(int         sock,
 	clicon_err(OE_PLUGIN, errno, "cbuf_new");
 	goto done;
     }
-    cprintf(msg, "<rpc xmlns=\"%s\">"
+    cprintf(msg, "<rpc xmlns=\"%s\" %s>"
 	    "<%slock><target><%s/></target></%slock></rpc>",
 	    NETCONF_BASE_NAMESPACE,
+	    NETCONF_MESSAGE_ID_ATTR,
 	    lock?"":"un", db, lock?"":"un");
     if (clicon_rpc1(sock, msg, msgret) < 0)
 	goto done;
@@ -422,6 +423,7 @@ clixon_client_get_xdata(int         sock,
     cprintf(msg, "<rpc xmlns=\"%s\"", NETCONF_BASE_NAMESPACE);
     cprintf(msg, " xmlns:%s=\"%s\"",
 	    NETCONF_BASE_PREFIX, NETCONF_BASE_NAMESPACE);
+    cprintf(msg, " %s", NETCONF_MESSAGE_ID_ATTR);
     cprintf(msg, "><get-config><source><%s/></source>", db);
     if (xpath && strlen(xpath)){
 	cprintf(msg, "<%s:filter %s:type=\"xpath\" xmlns=\"%s\" %s:select=\"%s\"",

lib/src/clixon_datastore.c

@@ -539,7 +539,7 @@ xmldb_empty_get(clicon_handle h,
     return de->de_empty;
 }
 
-/*! Get modified flag from datastore
+/*! Set modified flag from datastore
  * @param[in]  h     Clicon handle
  * @param[in]  db    Database name
  * @param[in]  value 0 or 1

lib/src/clixon_datastore_read.c

@@ -557,7 +557,7 @@ xmldb_readfile(clicon_handle    h,
 			    }
 			    cprintf(cberr, "Internal error: %s", clicon_err_reason);
 			    clicon_err_reset();
-			    if (netconf_operation_failed_xml(xerr, "application", cbuf_get(cberr))< 0)
+			    if (xerr && netconf_operation_failed_xml(xerr, "application", cbuf_get(cberr))< 0)
 				goto done;
 			    cbuf_free(cberr);
 			    goto fail;

lib/src/clixon_datastore_write.c

@@ -459,7 +459,7 @@ text_modify(clicon_handle       h,
 		changed++;
 		if (op==OP_NONE)
 		    xml_flag_set(x0, XML_FLAG_NONE); /* Mark for potential deletion */
-		if (x1bstr){ /* empty type does not have body */
+		if (x1bstr){ /* empty type does not have body */ /* XXX Here x0 = <b></b> */
 		    if ((x0b = xml_new("body", x0, CX_BODY)) == NULL)
 			goto done; 
 		}
@@ -499,23 +499,26 @@ text_modify(clicon_handle       h,
 		    if (assign_namespace_body(x1, x0) < 0)
 			goto done;
 		}
-		if ((x0b = xml_body_get(x0)) != NULL){
+		/* XXX here x1bstr is checked for null, while adding an empty string above */
-		    x0bstr = xml_value(x0b);
+		if ((x0b = xml_body_get(x0)) == NULL && x1bstr && strlen(x1bstr)){
-		    if (x0bstr==NULL || strcmp(x0bstr, x1bstr)){
+		    if ((x0b = xml_new("body", x0, CX_BODY)) == NULL)
-			if ((op != OP_NONE) && !permit && xnacm){
+			goto done;
-			    if ((ret = nacm_datanode_write(h, x1, x1t,
+		}
-							   x0bstr==NULL?NACM_CREATE:NACM_UPDATE,
+		x0bstr = xml_value(x0b);
-							   username, xnacm, cbret)) < 0)
+		if (x0bstr==NULL || strcmp(x0bstr, x1bstr)){
-				goto done;
+		    if ((op != OP_NONE) && !permit && xnacm){
-			    if (ret == 0)
+			if ((ret = nacm_datanode_write(h, x1, x1t,
-				goto fail;
+						       x0bstr==NULL?NACM_CREATE:NACM_UPDATE,
-			}
+						       username, xnacm, cbret)) < 0)
-			if (xml_value_set(x0b, x1bstr) < 0)
 			    goto done;
-			/* If a default value ies replaced, then reset default flag */
+			if (ret == 0)
-			if (xml_flag(x0, XML_FLAG_DEFAULT))
+			    goto fail;
-			    xml_flag_reset(x0, XML_FLAG_DEFAULT);
 		    }
+		    if (xml_value_set(x0b, x1bstr) < 0)
+			goto done;
+		    /* If a default value ies replaced, then reset default flag */
+		    if (xml_flag(x0, XML_FLAG_DEFAULT))
+			xml_flag_reset(x0, XML_FLAG_DEFAULT);
 		}
 	    } /* x1bstr */
 	    if (changed){ 

lib/src/clixon_json.c

@@ -1238,7 +1238,7 @@ _json_parse(char      *str,
 		goto done;
 	    }
 	    cprintf(cberr, "Top-level JSON object %s is not qualified with namespace which is a MUST according to RFC 7951", xml_name(x));
-	    if (netconf_malformed_message_xml(xerr, cbuf_get(cberr)) < 0)
+	    if (xerr && netconf_malformed_message_xml(xerr, cbuf_get(cberr)) < 0)
 		goto done;
 	    goto fail;
 	}

lib/src/clixon_netconf_lib.c

@@ -65,6 +65,7 @@
 #include "clixon_xml.h"
 #include "clixon_options.h"
 #include "clixon_data.h"
+#include "clixon_xml_bind.h"
 #include "clixon_xml_map.h"
 #include "clixon_xml_io.h"
 #include "clixon_xpath_ctx.h"
@@ -130,6 +131,10 @@ netconf_invalid_value_xml(cxobj **xret,
     char  *encstr = NULL;
     cxobj *xa;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -228,44 +233,86 @@ netconf_too_big(cbuf *cb,
     goto done;
 }
 
+/*! Create Netconf missing-attribute error XML tree according to RFC 6241 App A
+ *
+ * An expected attribute is missing.
+ * @param[out] xret    Error XML tree. Free with xml_free after use
+ * @param[in]  type    Error type: "rpc", "application" or "protocol"
+ * @param[in]  attr    bad-attribute and/or bad-element xml
+ * @param[in]  message Error message (will be XML encoded)
+ */
+int
+netconf_missing_attribute_xml(cxobj **xret,
+			      char   *type,
+			      char   *attr,
+			      char   *message)
+{
+    int   retval = -1;
+    cxobj *xerr = NULL;
+    char  *encstr = NULL;
+    cxobj *xa;
+
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
+    if (*xret == NULL){
+	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
+	    goto done;
+    }
+    else if (xml_name_set(*xret, "rpc-reply") < 0)
+	goto done;
+    if ((xa = xml_new("xmlns", *xret, CX_ATTR)) == NULL)
+	goto done;
+    if (xml_value_set(xa, NETCONF_BASE_NAMESPACE) < 0)
+	goto done;
+    if ((xerr = xml_new("rpc-error", *xret, CX_ELMNT)) == NULL)
+	goto done;
+    if (clixon_xml_parse_va(YB_NONE, NULL, &xerr, NULL, "<error-type>%s</error-type>"
+			    "<error-tag>missing-attribute</error-tag>"
+			    "<error-info><bad-attribute>%s</bad-attribute></error-info>"
+			    "<error-severity>error</error-severity>", type, attr) < 0)
+	goto done;
+    if (message){
+	if (xml_chardata_encode(&encstr, "%s", message) < 0)
+	    goto done;
+	if (clixon_xml_parse_va(YB_NONE, NULL, &xerr, NULL, "<error-message>%s</error-message>",
+				encstr) < 0)
+	    goto done;
+    }
+    retval = 0;
+ done:
+    if (encstr)
+	free(encstr);
+    return retval;
+}
+
 /*! Create Netconf missing-attribute error XML tree according to RFC 6241 App A
  *
  * An expected attribute is missing.
  * @param[out] cb      CLIgen buf. Error XML is written in this buffer
  * @param[in]  type    Error type: "rpc", "application" or "protocol"
- * @param[in]  info    bad-attribute or bad-element xml
+ * @param[in]  attr    bad-attribute 
- * @param[in]  message Error message (will be XML encoded)
+ * @param[in]  message Error message (will be XML encoded) or NULL
  */
 int
 netconf_missing_attribute(cbuf *cb,
 			  char *type,
-			  char *info,
+			  char *attr,
 			  char *message)
 {
-    int   retval = -1;
+    int    retval = -1;
-    char *encstr = NULL;
+    cxobj *xret = NULL;
 
-    if (cprintf(cb, "<rpc-reply xmlns=\"%s\"><rpc-error>"
+    if (netconf_missing_attribute_xml(&xret, type, attr, message) < 0)
-		"<error-type>%s</error-type>"
+	goto done;
-		"<error-tag>missing-attribute</error-tag>"
+    if (clicon_xml2cbuf(cb, xret, 0, 0, -1) < 0)
-		"<error-info>%s</error-info>"
+	goto done;
-		"<error-severity>error</error-severity>",
-		NETCONF_BASE_NAMESPACE,	type, info) <0)
-	goto err;
-    if (message){
-	if (xml_chardata_encode(&encstr, "%s", message) < 0)
-	    goto done;
-	if (cprintf(cb, "<error-message>%s</error-message>", encstr) < 0)
-	    goto err;
-    }
-    if (cprintf(cb, "</rpc-error></rpc-reply>") <0)
-	goto err;
     retval = 0;
  done:
+    if (xret)
+	xml_free(xret);
     return retval;
- err:
-    clicon_err(OE_XML, errno, "cprintf");
-    goto done;
 }
 
 /*! Create Netconf bad-attribute error XML tree according to RFC 6241 App A
@@ -317,6 +364,10 @@ netconf_bad_attribute_xml(cxobj **xret,
     char  *encstr = NULL;
     cxobj *xa;
 
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -410,6 +461,10 @@ netconf_common_xml(cxobj **xret,
     char  *encstr = NULL;
     cxobj *xa;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -666,6 +721,10 @@ netconf_access_denied_xml(cxobj **xret,
     char  *encstr = NULL;
     cxobj *xa;
 
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -902,6 +961,10 @@ netconf_data_missing_xml(cxobj **xret,
     cxobj *xerr;
     cxobj *xa;
 
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -966,6 +1029,10 @@ netconf_operation_not_supported_xml(cxobj **xret,
     char  *encstr = NULL;
     cxobj *xa;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -1077,6 +1144,10 @@ netconf_operation_failed_xml(cxobj **xret,
     char  *encstr = NULL;
     cxobj *xa;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -1161,6 +1232,10 @@ netconf_malformed_message_xml(cxobj **xret,
     char  *encstr = NULL;
     cxobj *xa;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -1211,8 +1286,12 @@ netconf_data_not_unique_xml(cxobj **xret,
     cxobj  *xerr;
     cxobj  *xinfo;
     cbuf   *cb = NULL;
-    cxobj *xa;
+    cxobj  *xa;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -1277,6 +1356,10 @@ netconf_minmax_elements_xml(cxobj **xret,
     cbuf  *cb = NULL;
     cxobj *xa;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_new("rpc-reply", NULL, CX_ELMNT)) == NULL)
 	    goto done;
@@ -1334,6 +1417,10 @@ netconf_trymerge(cxobj       *x,
     char  *reason = NULL;
     cxobj *xc;
     
+    if (xret == NULL){
+	clicon_err(OE_NETCONF, EINVAL, "xret is NULL");
+	goto done;	
+    }
     if (*xret == NULL){
 	if ((*xret = xml_dup(x)) == NULL)
 	    goto done;
@@ -1429,6 +1516,16 @@ netconf_module_load(clicon_handle h)
     /* Load restconf yang. Note this is also a part of clixon-config */
     if (yang_spec_parse_module(h, "clixon-restconf", NULL, yspec)< 0)
 	goto done;
+#if 1
+    /* XXX: Both the following settings are because clicon-handle is not part of all API
+     * functions
+     * Treat unknown XML as anydata */
+    if (clicon_option_bool(h, "CLICON_YANG_UNKNOWN_ANYDATA") == 1)
+	xml_bind_yang_unknown_anydata(1);
+    /* Make message-id attribute optional */
+    if (clicon_option_bool(h, "CLICON_NETCONF_MESSAGE_ID_OPTIONAL") == 1)
+	xml_bind_netconf_message_id_optional(1);
+#endif
     retval = 0;
  done:
     return retval;

lib/src/clixon_path.c

@@ -731,7 +731,8 @@ api_path2xpath_cvv(cvec       *api_path,
 		cvk = yang_cvec_get(y); /* Use Y_LIST cache, see ys_populate_list() */
 		cvi = NULL;
 		/* Iterate over individual yang keys  */
-		cprintf(xpath, "/");
+		if (i != offset)
+		    cprintf(xpath, "/");
 		if (xprefix)
 		    cprintf(xpath, "%s:", xprefix);
 		cprintf(xpath, "%s", name);
@@ -752,7 +753,8 @@ api_path2xpath_cvv(cvec       *api_path,
 		}
 		break;
 	    case Y_LEAF_LIST: /* XXX: LOOP? */
-		cprintf(xpath, "/");
+		if (i != offset)
+		    cprintf(xpath, "/");
 		if (xprefix)
 		    cprintf(xpath, "%s:", xprefix);
 		cprintf(xpath, "%s", name);
@@ -1129,6 +1131,9 @@ api_path2xml_vec(char      **vec,
 }
 
 /*! Create xml tree from api-path
+ *
+ * Create an XML tree from "scratch" using api-path, ie no other input than the api-path itself,
+ * ie not from an existing datastore for example.
  * @param[in]     api_path   (Absolute) API-path as defined in RFC 8040
  * @param[in]     yspec      Yang spec
  * @param[in,out] xtop       Incoming XML tree

lib/src/clixon_proc.c

@@ -648,7 +648,7 @@ clixon_process_operation(clicon_handle  h,
 		sched++;/* start: immediate stop/restart: not immediate: wait timeout */
 	    }
 	    else{
-		clicon_debug(1, "%s name:%s op %s cancelled by wrwap", __FUNCTION__, name, clicon_int2str(proc_operation_map, op0));		
+		clicon_debug(1, "%s name:%s op %s cancelled by wrap", __FUNCTION__, name, clicon_int2str(proc_operation_map, op0));		
 	    }
 	    break; 	    /* hit break here */
 	}
@@ -682,48 +682,54 @@ clixon_process_status(clicon_handle  h,
     int              run;
     int              i;
     char             timestr[28];
+    int              match = 0;
 
-    if (_proc_entry_list == NULL)
+    if (_proc_entry_list != NULL){
-	goto ok;
+	pe = _proc_entry_list;
-    pe = _proc_entry_list;
+	do {
-    do {
+	    if (strcmp(pe->pe_name, name) == 0){
-	if (strcmp(pe->pe_name, name) == 0){
+		clicon_debug(1, "%s found %s pid:%d", __FUNCTION__, name, pe->pe_pid);
-	    /* Check if running */
+		/* Check if running */
-	    run = 0;
+		run = 0;
-	    if (pe->pe_pid && proc_op_run(pe->pe_pid, &run) < 0)
+		if (pe->pe_pid && proc_op_run(pe->pe_pid, &run) < 0)
-		goto done;
-	    cprintf(cbret, "<rpc-reply xmlns=\"%s\"><active xmlns=\"%s\">%s</active>",
-		    NETCONF_BASE_NAMESPACE, CLIXON_LIB_NS, run?"true":"false");
-	    if (pe->pe_description)
-		cprintf(cbret, "<description xmlns=\"%s\">%s</description>", CLIXON_LIB_NS, pe->pe_description);
-	    cprintf(cbret, "<command xmlns=\"%s\">", CLIXON_LIB_NS);
-	    /* the command may include any data, including XML (such as restconf -R command) and 
-	       therefore needs CDATA encoding */
-	    cprintf(cbret, "<![CDATA[");
-	    for (i=0; i<pe->pe_argc-1; i++){
-		if (i)
-		    cprintf(cbret, " ");
-		cprintf(cbret, "%s", pe->pe_argv[i]);
-	    }
-	    cprintf(cbret, "]]>");
-	    cprintf(cbret, "</command>");
-	    cprintf(cbret, "<status xmlns=\"%s\">%s</status>", CLIXON_LIB_NS,
-		    clicon_int2str(proc_state_map, pe->pe_state));
-	    if (timerisset(&pe->pe_starttime)){
-		if (time2str(pe->pe_starttime, timestr, sizeof(timestr)) < 0){
-		    clicon_err(OE_UNIX, errno, "time2str");
 		    goto done;
+		cprintf(cbret, "<rpc-reply xmlns=\"%s\"><active xmlns=\"%s\">%s</active>",
+			NETCONF_BASE_NAMESPACE, CLIXON_LIB_NS, run?"true":"false");
+		if (pe->pe_description)
+		    cprintf(cbret, "<description xmlns=\"%s\">%s</description>", CLIXON_LIB_NS, pe->pe_description);
+		cprintf(cbret, "<command xmlns=\"%s\">", CLIXON_LIB_NS);
+		/* the command may include any data, including XML (such as restconf -R command) and 
+		   therefore needs CDATA encoding */
+		cprintf(cbret, "<![CDATA[");
+		for (i=0; i<pe->pe_argc-1; i++){
+		    if (i)
+			cprintf(cbret, " ");
+		    cprintf(cbret, "%s", pe->pe_argv[i]);
 		}
-		cprintf(cbret, "<starttime xmlns=\"%s\">%s</starttime>", CLIXON_LIB_NS, timestr);
+		cprintf(cbret, "]]>");
+		cprintf(cbret, "</command>");
+		cprintf(cbret, "<status xmlns=\"%s\">%s</status>", CLIXON_LIB_NS,
+			clicon_int2str(proc_state_map, pe->pe_state));
+		if (timerisset(&pe->pe_starttime)){
+		    if (time2str(pe->pe_starttime, timestr, sizeof(timestr)) < 0){
+			clicon_err(OE_UNIX, errno, "time2str");
+			goto done;
+		    }
+		    cprintf(cbret, "<starttime xmlns=\"%s\">%s</starttime>", CLIXON_LIB_NS, timestr);
+		}
+		if (pe->pe_pid)
+		    cprintf(cbret, "<pid xmlns=\"%s\">%u</pid>", CLIXON_LIB_NS, pe->pe_pid);
+		cprintf(cbret, "</rpc-reply>");
+		match++;
+		break; 	    /* hit break here */
 	    }
-	    if (pe->pe_pid)
+	    pe = NEXTQ(process_entry_t *, pe);
-		cprintf(cbret, "<pid xmlns=\"%s\">%u</pid>", CLIXON_LIB_NS, pe->pe_pid);
+	} while (pe != _proc_entry_list);
-	    cprintf(cbret, "</rpc-reply>");
+    }
-	    break; 	    /* hit break here */
+    if (!match){ /* No match, return error */
-	}
+	if (netconf_unknown_element(cbret, "application", (char*)name, "Process service is not known") < 0)
-	pe = NEXTQ(process_entry_t *, pe);
+	    goto done;	    
-    } while (pe != _proc_entry_list);
+    }
- ok:
     retval = 0;
  done:
     clicon_debug(1, "%s retval:%d", __FUNCTION__, retval);

lib/src/clixon_proto_client.c

@@ -451,6 +451,7 @@ clicon_rpc_get_config(clicon_handle h,
 	cprintf(cb, " username=\"%s\"", username);
     cprintf(cb, " xmlns:%s=\"%s\"",
 	    NETCONF_BASE_PREFIX, NETCONF_BASE_NAMESPACE);
+    cprintf(cb, " %s", NETCONF_MESSAGE_ID_ATTR);
     cprintf(cb, "><get-config><source><%s/></source>", db);
     if (xpath && strlen(xpath)){
 	cprintf(cb, "<%s:filter %s:type=\"xpath\" %s:select=\"%s\"",
@@ -541,6 +542,7 @@ clicon_rpc_edit_config(clicon_handle       h,
     cprintf(cb, " xmlns:%s=\"%s\"", NETCONF_BASE_PREFIX, NETCONF_BASE_NAMESPACE);
     if ((username = clicon_username_get(h)) != NULL)
 	cprintf(cb, " username=\"%s\"", username);
+    cprintf(cb, " %s", NETCONF_MESSAGE_ID_ATTR);
     cprintf(cb, "><edit-config><target><%s/></target>", db);
     cprintf(cb, "<default-operation>%s</default-operation>", 
 	    xml_operation2str(op));
@@ -595,10 +597,11 @@ clicon_rpc_copy_config(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\">"
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s>"
 				 "<copy-config><source><%s/></source><target><%s/></target></copy-config></rpc>",
 				 NETCONF_BASE_NAMESPACE,
 				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
 				 db1, db2)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
@@ -641,10 +644,12 @@ clicon_rpc_delete_config(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\">"
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s>"
 				 "<edit-config><target><%s/></target><default-operation>none</default-operation><config operation=\"delete\"/></edit-config></rpc>",
 				 NETCONF_BASE_NAMESPACE,
-				 username?username:"", db)) == NULL)
+				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
+				 db)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -682,10 +687,12 @@ clicon_rpc_lock(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\">"
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s>"
 				 "<lock><target><%s/></target></lock></rpc>",
 				 NETCONF_BASE_NAMESPACE,
-				 username?username:"", db)) == NULL)
+				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
+				 db)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -723,10 +730,12 @@ clicon_rpc_unlock(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\">"
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s>"
 				 "<unlock><target><%s/></target></unlock></rpc>",
 				 NETCONF_BASE_NAMESPACE,
-				 username?username:"", db)) == NULL)
+				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
+				 db)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -805,6 +814,7 @@ clicon_rpc_get(clicon_handle   h,
 	cprintf(cb, " username=\"%s\"", username);
     cprintf(cb, " xmlns:%s=\"%s\"",
 	    NETCONF_BASE_PREFIX, NETCONF_BASE_NAMESPACE);
+    cprintf(cb, " %s", NETCONF_MESSAGE_ID_ATTR);
     cprintf(cb, "><get");
     /* Clixon extension, content=all,config, or nonconfig */
     if ((int)content != -1)
@@ -889,8 +899,9 @@ clicon_rpc_close_session(clicon_handle h)
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\" message-id=\"%u\"><close-session/></rpc>",
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s><close-session/></rpc>",
-				 NETCONF_BASE_NAMESPACE, username?username:"", 42)) == NULL)
+				 NETCONF_BASE_NAMESPACE, username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -932,9 +943,11 @@ clicon_rpc_kill_session(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(my_session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\"><kill-session><session-id>%u</session-id></kill-session></rpc>",
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s><kill-session><session-id>%u</session-id></kill-session></rpc>",
 				 NETCONF_BASE_NAMESPACE,
-				 username?username:"", session_id)) == NULL)
+				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
+				 session_id)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -972,9 +985,11 @@ clicon_rpc_validate(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\"><validate><source><%s/></source></validate></rpc>",
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s><validate><source><%s/></source></validate></rpc>",
 				 NETCONF_BASE_NAMESPACE,
-				 username?username:"", db)) == NULL)
+				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
+				 db)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -1010,9 +1025,10 @@ clicon_rpc_commit(clicon_handle h)
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\"><commit/></rpc>",
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s><commit/></rpc>",
 				 NETCONF_BASE_NAMESPACE,
-				 username?username:"")) == NULL)
+				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -1048,9 +1064,10 @@ clicon_rpc_discard_changes(clicon_handle h)
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\"  username=\"%s\"><discard-changes/></rpc>",
+				 "<rpc xmlns=\"%s\"  username=\"%s\" %s><discard-changes/></rpc>",
 				 NETCONF_BASE_NAMESPACE,
-				 username?username:"")) == NULL)
+				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR)) == NULL)
 	goto done;
     if (clicon_rpc_msg(h, msg, &xret) < 0)
 	goto done;
@@ -1094,12 +1111,13 @@ clicon_rpc_create_subscription(clicon_handle    h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\"><create-subscription xmlns=\"%s\">"
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s><create-subscription xmlns=\"%s\">"
 				 "<stream>%s</stream>"
 				 "<filter type=\"xpath\" select=\"%s\" />"
 				 "</create-subscription></rpc>", 
 				 NETCONF_BASE_NAMESPACE,
 				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
 				 EVENT_RFC5277_NAMESPACE,
 				 stream?stream:"", filter?filter:"")) == NULL)
 	goto done;
@@ -1139,9 +1157,10 @@ clicon_rpc_debug(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\"><debug xmlns=\"%s\"><level>%d</level></debug></rpc>",
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s><debug xmlns=\"%s\"><level>%d</level></debug></rpc>",
 				 NETCONF_BASE_NAMESPACE,
 				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
 				 CLIXON_LIB_NS,
 				 level)) == NULL)
 	goto done;
@@ -1189,9 +1208,10 @@ clicon_rpc_restconf_debug(clicon_handle h,
 	goto done;
     username = clicon_username_get(h);
     if ((msg = clicon_msg_encode(session_id,
-				 "<rpc xmlns=\"%s\" username=\"%s\"><edit-config><target><candidate/></target><config><restconf xmlns=\"%s\"><debug>%d</debug></restconf></config></edit-config></rpc>",
+				 "<rpc xmlns=\"%s\" username=\"%s\" %s><edit-config><target><candidate/></target><config><restconf xmlns=\"%s\"><debug>%d</debug></restconf></config></edit-config></rpc>",
 				 NETCONF_BASE_NAMESPACE,
 				 username?username:"",
+				 NETCONF_MESSAGE_ID_ATTR,
 				 CLIXON_RESTCONF_NS,
 				 level)) == NULL)
 	goto done;

lib/src/clixon_validate.c

@@ -105,7 +105,6 @@ validate_leafref(cxobj     *xt,
 {
     int          retval = -1;
     yang_stmt   *ypath;
-    yang_stmt   *yp;
     cxobj      **xvec = NULL;
     cxobj       *x;
     int          i;
@@ -119,19 +118,13 @@ validate_leafref(cxobj     *xt,
     if ((leafrefbody = xml_body(xt)) == NULL)
 	goto ok;
     if ((ypath = yang_find(ytype, Y_PATH, NULL)) == NULL){
-	if (netconf_missing_element_xml(xret, "application", yang_argument_get(ytype), "Leafref requires path statement") < 0)
+	if (xret && netconf_missing_element_xml(xret, "application", yang_argument_get(ytype), "Leafref requires path statement") < 0)
 	    goto done;
 	goto fail;
     }
     /* See comment^: If path is defined in typedef or not */
-    if ((yp = yang_parent_get(ytype)) != NULL &&
+    if (xml_nsctx_node(xt, &nsc) < 0)
-	yang_keyword_get(yp) == Y_TYPEDEF){
+	goto done;
-	if (xml_nsctx_yang(ys, &nsc) < 0)
-	    goto done;
-    }
-    else
-	if (xml_nsctx_yang(ytype, &nsc) < 0)
-	    goto done;
     path = yang_argument_get(ypath);
     if (xpath_vec(xt, nsc, "%s", &xvec, &xlen, path) < 0) 
 	goto done;
@@ -148,7 +141,7 @@ validate_leafref(cxobj     *xt,
 	    goto done;
 	}
 	cprintf(cberr, "Leafref validation failed: No leaf %s matching path %s", leafrefbody, path);
-	if (netconf_bad_element_xml(xret, "application", leafrefbody, cbuf_get(cberr)) < 0)
+	if (xret && netconf_bad_element_xml(xret, "application", leafrefbody, cbuf_get(cberr)) < 0)
 	    goto done;
 	goto fail;
     }
@@ -218,7 +211,7 @@ validate_identityref(cxobj     *xt,
     /* Get idref value. Then see if this value is derived from ytype.
      */
     if ((node = xml_body(xt)) == NULL){ /* It may not be empty */
-	if (netconf_bad_element_xml(xret, "application", xml_name(xt), "Identityref should not be empty") < 0)
+	if (xret && netconf_bad_element_xml(xret, "application", xml_name(xt), "Identityref should not be empty") < 0)
 	    goto done;
 	goto fail;
     }
@@ -226,13 +219,13 @@ validate_identityref(cxobj     *xt,
 	goto done;
     /* This is the type's base reference */
     if ((ybaseref = yang_find(ytype, Y_BASE, NULL)) == NULL){
-	if (netconf_missing_element_xml(xret, "application", yang_argument_get(ytype), "Identityref validation failed, no base") < 0)
+	if (xret && netconf_missing_element_xml(xret, "application", yang_argument_get(ytype), "Identityref validation failed, no base") < 0)
 	    goto done;
 	goto fail;
     }
     /* This is the actual base identity */
     if ((ybaseid = yang_find_identity(ybaseref, yang_argument_get(ybaseref))) == NULL){
-	if (netconf_missing_element_xml(xret, "application", yang_argument_get(ybaseref), "Identityref validation failed, no base identity") < 0)
+	if (xret && netconf_missing_element_xml(xret, "application", yang_argument_get(ybaseref), "Identityref validation failed, no base identity") < 0)
 	    goto done;
 	goto fail;
     }
@@ -261,7 +254,7 @@ validate_identityref(cxobj     *xt,
     if (cvec_find(idrefvec, idref) == NULL){
 	cprintf(cberr, "Identityref validation failed, %s not derived from %s", 
 		node, yang_argument_get(ybaseid));
-	if (netconf_operation_failed_xml(xret, "application", cbuf_get(cberr)) < 0)
+	if (xret && netconf_operation_failed_xml(xret, "application", cbuf_get(cberr)) < 0)
 	    goto done;
 	goto fail;
     }
@@ -344,7 +337,7 @@ xml_yang_validate_rpc(clicon_handle h,
 	goto done;
     /* Only accept resolved NETCONF base namespace */
     if (namespace == NULL || strcmp(namespace, NETCONF_BASE_NAMESPACE) != 0){
-	if (netconf_unknown_namespace_xml(xret, "protocol", rpcprefix, "No appropriate namespace associated with prefix")< 0)
+	if (xret && netconf_unknown_namespace_xml(xret, "protocol", rpcprefix, "No appropriate namespace associated with prefix")< 0)
 	    goto done;
 	goto fail;
     }
@@ -352,7 +345,7 @@ xml_yang_validate_rpc(clicon_handle h,
     /* xn is name of rpc, ie <rcp><xn/></rpc> */
     while ((xn = xml_child_each(xrpc, xn, CX_ELMNT)) != NULL) {
 	if ((yn = xml_spec(xn)) == NULL){
-	    if (netconf_unknown_element_xml(xret, "application", xml_name(xn), NULL) < 0)
+	    if (xret && netconf_unknown_element_xml(xret, "application", xml_name(xn), NULL) < 0)
 		goto done;
 	    goto fail;
 	}
@@ -441,7 +434,7 @@ check_choice(cxobj     *xt,
 	    continue; /* not choice */
 	    break;
 	}
-	if (netconf_bad_element_xml(xret, "application", xml_name(x), "Element in choice statement already exists") < 0)
+	if (xret && netconf_bad_element_xml(xret, "application", xml_name(x), "Element in choice statement already exists") < 0)
 	    goto done;
 	goto fail;
     } /* while */
@@ -494,7 +487,7 @@ check_list_key(cxobj     *xt,
 	while ((cvi = cvec_each(cvk, cvi)) != NULL) {
 	    keyname = cv_string_get(cvi);	    
 	    if (xml_find_type(xt, NULL, keyname, CX_ELMNT) == NULL){
-		if (netconf_missing_element_xml(xret, "application", keyname, "Mandatory key") < 0)
+		if (xret && netconf_missing_element_xml(xret, "application", keyname, "Mandatory key") < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -564,7 +557,7 @@ check_mandatory(cxobj     *xt,
 		    goto done;
 		}
 		cprintf(cb, "Mandatory variable of %s in module %s", xml_name(xt), yang_argument_get(ys_module(yc)));
-		if (netconf_missing_element_xml(xret, "application", yang_argument_get(yc), cbuf_get(cb)) < 0)
+		if (xret && netconf_missing_element_xml(xret, "application", yang_argument_get(yc), cbuf_get(cb)) < 0)
 			    goto done;
 		goto fail;
 	    }
@@ -581,7 +574,7 @@ check_mandatory(cxobj     *xt,
 	    if (x == NULL){
 		/* @see RFC7950: 15.6 Error Message for Data That Violates 
 		 * a Mandatory "choice" Statement */
-		if (netconf_data_missing_xml(xret, yang_argument_get(yc), NULL) < 0)
+		if (xret && netconf_data_missing_xml(xret, yang_argument_get(yc), NULL) < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -714,7 +707,7 @@ check_unique_list(cxobj     *x,
 	if (cvi==NULL){
 	    /* Last element (i) is newly inserted, see if it is already there */
 	    if (check_insert_duplicate(vec, i, vlen, sorted) < 0){
-		if (netconf_data_not_unique_xml(xret, x, cvk) < 0)
+		if (xret && netconf_data_not_unique_xml(xret, x, cvk) < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -758,7 +751,7 @@ check_min_max(cxobj     *xp,
     if ((ymin = yang_find(y, Y_MIN_ELEMENTS, NULL)) != NULL){
 	cv = yang_cv_get(ymin);
 	if (nr < cv_uint32_get(cv)){
-	    if (netconf_minmax_elements_xml(xret, xp, yang_argument_get(y), 0) < 0)
+	    if (xret && netconf_minmax_elements_xml(xret, xp, yang_argument_get(y), 0) < 0)
 		goto done;
 	    goto fail;
 	}
@@ -767,7 +760,7 @@ check_min_max(cxobj     *xp,
 	cv = yang_cv_get(ymax);
 	if (cv_uint32_get(cv) > 0 && /* 0 means unbounded */
 	    nr > cv_uint32_get(cv)){
-	    if (netconf_minmax_elements_xml(xret, xp, yang_argument_get(y), 1) < 0)
+	    if (xret && netconf_minmax_elements_xml(xret, xp, yang_argument_get(y), 1) < 0)
 		goto done;
 	    goto fail;
 	}
@@ -858,7 +851,7 @@ check_list_unique_minmax(cxobj  *xt,
 		/* Only lists and leaf-lists are allowed to be many 
 		 * This checks duplicate container and leafs
 		 */
-		if (netconf_minmax_elements_xml(xret, xt, xml_name(x), 1) < 0)
+		if (xret && netconf_minmax_elements_xml(xret, xt, xml_name(x), 1) < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -1025,14 +1018,14 @@ xml_yang_validate_add(clicon_handle h,
 		 * are considered as "" */
 		cvtype = cv_type_get(cv);
 		if (cv_isint(cvtype) || cvtype == CGV_BOOL || cvtype == CGV_DEC64){
-		    if (netconf_bad_element_xml(xret, "application",  yang_argument_get(yt), "Invalid NULL value") < 0)
+		    if (xret && netconf_bad_element_xml(xret, "application",  yang_argument_get(yt), "Invalid NULL value") < 0)
 			goto done;
 		    goto fail;
 		}
 	    }
 	    else{
 		if (cv_parse1(body, cv, &reason) != 1){
-		    if (netconf_bad_element_xml(xret, "application",  yang_argument_get(yt), reason) < 0)
+		    if (xret && netconf_bad_element_xml(xret, "application",  yang_argument_get(yt), reason) < 0)
 			goto done;
 		    goto fail;
 		}
@@ -1040,7 +1033,7 @@ xml_yang_validate_add(clicon_handle h,
 	    if ((ret = ys_cv_validate(h, cv, yt, NULL, &reason)) < 0)
 		goto done;
 	    if (ret == 0){
-		if (netconf_bad_element_xml(xret, "application",  yang_argument_get(yt), reason) < 0)
+		if (xret && netconf_bad_element_xml(xret, "application",  yang_argument_get(yt), reason) < 0)
 		    goto done;
 		goto fail;
 	    }
@@ -1165,7 +1158,7 @@ xml_yang_validate_all(clicon_handle h,
 	    goto done;
 	if (ns)
 	    cprintf(cb, " in namespace: %s", ns);
-	if (netconf_unknown_element_xml(xret, "application", xml_name(xt), cbuf_get(cb)) < 0)
+	if (xret && netconf_unknown_element_xml(xret, "application", xml_name(xt), cbuf_get(cb)) < 0)
 	    goto done;
 	goto fail;
     }
@@ -1224,7 +1217,7 @@ xml_yang_validate_all(clicon_handle h,
 		}
 		cprintf(cb, "Failed MUST xpath '%s' of '%s' in module %s",
 			xpath, xml_name(xt),  yang_argument_get(ys_module(ys)));
-		if (netconf_operation_failed_xml(xret, "application", 
+		if (xret && netconf_operation_failed_xml(xret, "application", 
 						 ye?yang_argument_get(ye):cbuf_get(cb)) < 0)
 		    goto done;
 		goto fail;
@@ -1254,7 +1247,7 @@ xml_yang_validate_all(clicon_handle h,
 		cprintf(cb, "Failed WHEN condition of %s in module %s",
 			xml_name(xt),
 			yang_argument_get(ys_module(ys)));
-		if (netconf_operation_failed_xml(xret, "application", 
+		if (xret && netconf_operation_failed_xml(xret, "application", 
 						 cbuf_get(cb)) < 0)
 		    goto done;
 		goto fail;
@@ -1276,7 +1269,7 @@ xml_yang_validate_all(clicon_handle h,
 			xpath,
 			xml_name(xt),
 			yang_argument_get(ys_module(ys)));
-		if (netconf_operation_failed_xml(xret, "application", 
+		if (xret && netconf_operation_failed_xml(xret, "application", 
 						 cbuf_get(cb)) < 0)
 		    goto done;
 		goto fail;

lib/src/clixon_xml_bind.c

@@ -83,6 +83,7 @@
  * Local variables
  */
 static int _yang_unknown_anydata = 0;
+static int _netconf_message_id_optional = 0;
 
 /*! Kludge to equate unknown XML with anydata
  * The problem with this is that its global and should be bound to a handle
@@ -94,6 +95,16 @@ xml_bind_yang_unknown_anydata(int val)
     return 0;
 }
 
+/*! Kludge to set message_id_optional
+ * The problem with this is that its global and should be bound to a handle
+ */
+int
+xml_bind_netconf_message_id_optional(int val)
+{
+    _netconf_message_id_optional = val;
+    return 0;
+}
+
 /*! After yang binding, bodies of containers and lists are stripped from XML bodies
  * May apply to other nodes?
  */
@@ -510,16 +521,16 @@ xml_bind_yang0(cxobj     *xt,
  * @retval     -1      Error
  * The 
  * @code
- *   if (xml_bind_yang_rpc(h, x, NULL) < 0)
+ *   if (xml_bind_yang_rpc(x, NULL) < 0)
  *      err;
  * @endcode
  * @see xml_bind_yang  For other generic cases
  * @see xml_bind_yang_rpc_reply
  */
 int
-xml_bind_yang_rpc(cxobj     *xrpc,
+xml_bind_yang_rpc(cxobj        *xrpc,
-		  yang_stmt *yspec,
+		  yang_stmt    *yspec,
-		  cxobj    **xerr)
+		  cxobj       **xerr)
 {
     int        retval = -1;
     yang_stmt *yrpc = NULL;    /* yang node */
@@ -576,6 +587,17 @@ xml_bind_yang_rpc(cxobj     *xrpc,
 	    goto done;
 	goto fail;
     }
+    if (_netconf_message_id_optional == 0){
+	/* RFC 6241 4.1:
+	 *    The <rpc> element has a mandatory attribute "message-id"
+	 */
+	if (xml_find_type(xrpc, NULL, "message-id", CX_ATTR) == NULL){
+	    if (xerr &&
+		netconf_missing_attribute_xml(xerr, "rpc", "message-id", "Incoming rpc") < 0)
+		goto done;
+	    goto fail;
+	}
+    }
     x = NULL;
     while ((x = xml_child_each(xrpc, x, CX_ELMNT)) != NULL) {
 	rpcname = xml_name(x);

lib/src/clixon_xpath.c

@@ -239,7 +239,6 @@ xpath_tree_print(FILE       *f,
  * @param[in]  xs    XPATH tree
  * @param[out] xpath XPath string as CLIgen buf
  * @see xpath_tree_print
- * @note XXX Not complete 
  */
 int
 xpath_tree2cbuf(xpath_tree *xs,
@@ -247,6 +246,7 @@ xpath_tree2cbuf(xpath_tree *xs,
 {
     int   retval = -1;
 
+    /* 1. Before first child */
     switch (xs->xs_type){
     case XP_ABSPATH:
 	if (xs->xs_int == A_DESCENDANT_OR_SELF)
@@ -283,16 +283,28 @@ xpath_tree2cbuf(xpath_tree *xs,
     default:
 	break;
     }
+    /* 2. First child */
     if (xs->xs_c0 && xpath_tree2cbuf(xs->xs_c0, xcb) < 0)
 	goto done;
+    /* 3. Between first and second child */
     switch (xs->xs_type){
     case XP_AND: /* and or */
     case XP_ADD: /* div mod + * - */
-    case XP_RELEX: /* !=, >= <= < > = */
-    case XP_UNION:
 	if (xs->xs_c1)
 	    cprintf(xcb, " %s ", clicon_int2str(xpopmap, xs->xs_int));
 	break;
+    case XP_RELEX: /* !=, >= <= < > = */
+    case XP_UNION: /* | */
+	if (xs->xs_c1)
+	    cprintf(xcb, "%s", clicon_int2str(xpopmap, xs->xs_int));
+	break;
+    case XP_PATHEXPR:
+	/* [19]    PathExpr ::=  | FilterExpr '/' RelativeLocationPath	
+		                 | FilterExpr '//' RelativeLocationPath	
+	*/
+	if (xs->xs_s0)
+	    cprintf(xcb, "%s", xs->xs_s0);
+	break;
     case XP_RELLOCPATH:
 	if (xs->xs_c1){
 	    if (xs->xs_int == A_DESCENDANT_OR_SELF)
@@ -311,8 +323,10 @@ xpath_tree2cbuf(xpath_tree *xs,
     default:
 	break;
     }
+    /* 4. Second child */
     if (xs->xs_c1 && xpath_tree2cbuf(xs->xs_c1, xcb) < 0)
 	goto done;
+    /* 5. After second child */
     switch (xs->xs_type){
     case XP_PRED:
 	if (xs->xs_c1)
@@ -915,6 +929,8 @@ xpath_vec_bool(cxobj      *xcur,
     return retval;
 }
 
+/*!
+ */
 static int
 traverse_canonical(xpath_tree *xs,
 		   yang_stmt  *yspec,

lib/src/clixon_xpath_parse.y

@@ -163,7 +163,7 @@ xpath_parse_exit(clixon_xpath_yacc *xpy)
  * @param[in]  type   XPATH tree node type
  * @param[in]  i0     step-> axis_type 
  * @param[in]  numstr original string xs_double: numeric value 
- * @param[in]  s0     String 0 set if XP_PRIME_STR, XP_PRIME_FN, XP_NODE[_FN] prefix
+ * @param[in]  s0     String 0 set if XP_PRIME_STR, XP_PRIME_FN, XP_NODE[_FN] PATHEXPRE prefix
  * @param[in]  s1     String 1 set if XP_NODE NAME
  * @param[in]  c0     Child 0
  * @param[in]  c1     Child 1
@@ -393,8 +393,8 @@ unionexpr   : unionexpr '|' pathexpr { $$=xp_new(XP_UNION,XO_UNION,NULL,NULL,NUL
 
 pathexpr    : locationpath { $$=xp_new(XP_PATHEXPR,A_NAN,NULL,NULL,NULL,$1, NULL);clicon_debug(3,"pathexpr-> locationpath"); } 
             | filterexpr { $$=xp_new(XP_PATHEXPR,A_NAN,NULL,NULL,NULL,$1, NULL);clicon_debug(3,"pathexpr-> filterexpr"); }
-            | filterexpr '/' rellocpath { $$=xp_new(XP_PATHEXPR,A_NAN,NULL,NULL,NULL,$1, $3);clicon_debug(3,"pathexpr-> filterexpr / rellocpath"); } 
+            | filterexpr '/' rellocpath { $$=xp_new(XP_PATHEXPR,A_NAN,NULL,strdup("/"),NULL,$1, $3);clicon_debug(3,"pathexpr-> filterexpr / rellocpath"); }
-            /* Filterexpr // relativelocationpath */
+            | filterexpr DOUBLESLASH rellocpath { $$=xp_new(XP_PATHEXPR,A_NAN,NULL,strdup("//"),NULL,$1, $3);clicon_debug(3,"pathexpr-> filterexpr // rellocpath"); } 
             ;
 
 filterexpr  : primaryexpr { $$=xp_new(XP_FILTEREXPR,A_NAN,NULL,NULL,NULL,$1, NULL);clicon_debug(3,"filterexpr-> primaryexpr"); } 

lib/src/clixon_yang.c

@@ -256,6 +256,22 @@ yang_cv_get(yang_stmt *ys)
     return ys->ys_cv;
 }
 
+/*! Set yang statement CLIgen variable
+ * @param[in] ys  Yang statement node
+ * @param[in] cv  cligen variable
+ * @note: Frees on replace, not if cv is NULL. This is for some ys_cp/ys_dup cases, which means 
+ *        you need to free it explicitly to set it to NULL proper.
+ */
+int
+yang_cv_set(yang_stmt *ys,
+	    cg_var    *cv)
+{
+    if (cv != NULL && ys->ys_cv != NULL)
+	cv_free(ys->ys_cv);
+    ys->ys_cv = cv;
+    return 0;
+}
+
 /*! Get yang statement CLIgen variable vector
  * @param[in] ys  Yang statement node
  */
@@ -456,13 +472,15 @@ int
 ys_free1(yang_stmt *ys,
 	 int        self)
 {
+    cg_var *cv;
+
     if (ys->ys_argument){
 	free(ys->ys_argument);
 	ys->ys_argument = NULL;
     }
-    if (ys->ys_cv){
+    if ((cv = yang_cv_get(ys)) != NULL){
-	cv_free(ys->ys_cv);
+	yang_cv_set(ys, NULL); /* only frees on replace */
-	ys->ys_cv = NULL;
+	cv_free(cv);
     }
     if (ys->ys_cvec){
 	cvec_free(ys->ys_cvec);
@@ -601,6 +619,8 @@ ys_cp(yang_stmt *ynew,
     int        i;
     yang_stmt *ycn; /* new child */
     yang_stmt *yco; /* old child */
+    cg_var    *cvn;
+    cg_var    *cvo;
 
     memcpy(ynew, yold, sizeof(*yold)); 
     ynew->ys_parent = NULL;
@@ -614,11 +634,14 @@ ys_cp(yang_stmt *ynew,
 	    clicon_err(OE_YANG, errno, "strdup");
 	    goto done;
 	}
-    if (yold->ys_cv)
+    yang_cv_set(ynew, NULL);
-	if ((ynew->ys_cv = cv_dup(yold->ys_cv)) == NULL){
+    if ((cvo = yang_cv_get(yold)) != NULL){
+	if ((cvn = cv_dup(cvo)) == NULL){
 	    clicon_err(OE_YANG, errno, "cv_dup");
 	    goto done;
 	}
+	yang_cv_set(ynew, cvn);
+    }
     if (yold->ys_cvec)
 	if ((ynew->ys_cvec = cvec_dup(yold->ys_cvec)) == NULL){
 	    clicon_err(OE_YANG, errno, "cvec_dup");
@@ -1414,6 +1437,7 @@ ys_real_module(yang_stmt  *ys,
 {
     int        retval = -1;
     yang_stmt *ym = NULL;
+    yang_stmt *ysubm;
     yang_stmt *yb;
     char      *name;
     yang_stmt *yspec;
@@ -1433,8 +1457,12 @@ ys_real_module(yang_stmt  *ys,
 		clicon_err(OE_YANG, ENOENT, "Belongs-to statement of submodule %s has no argument", yang_argument_get(ym)); /* shouldnt happen */
 		goto done;
 	    }
-	    if ((ym = yang_find_module_by_name(yspec, name)) == NULL)
+	    if ((ysubm = yang_find_module_by_name(yspec, name)) == NULL){
+		clicon_err(OE_YANG, ENOENT, "submodule %s references non-existent module %s in its belongs-to statement",
+			   yang_argument_get(ym), name);
 		goto done;
+	    }
+	    ym = ysubm;
 	}
     }
     *ymod = ym;
@@ -1447,7 +1475,8 @@ ys_real_module(yang_stmt  *ys,
  * @param[in] ys    Any yang statement in a yang tree
  * @retval    yspec The top yang specification
  * @see  ys_module
- * @see  yang_augment_node where shortcut is set
+ * @see  yang_augment_node where shortcut is set for augment
+ * @see  yang_myroot for first node under (sub)module
  */
 yang_stmt *
 ys_spec(yang_stmt *ys)
@@ -1765,8 +1794,7 @@ ys_populate_leaf(clicon_handle h,
 
     yparent = ys->ys_parent;     /* Find parent: list/container */
     /* 1. Find type specification and set cv type accordingly */
-    if (yang_type_get(ys, &origtype, &yrestype, &options, NULL, NULL, NULL, &fraction_digits)
+    if (yang_type_get(ys, &origtype, &yrestype, &options, NULL, NULL, NULL, &fraction_digits) < 0)
- < 0)
 	goto done;
     restype = yrestype?yrestype->ys_argument:NULL;
     if (clicon_type2cv(origtype, restype, ys, &cvtype) < 0) /* This handles non-resolved also */
@@ -1823,7 +1851,7 @@ ys_populate_leaf(clicon_handle h,
 	if ((ret = yang_key_match(yparent, ys->ys_argument)) < 0)
 	    goto done;
     }
-    ys->ys_cv = cv;
+    yang_cv_set(ys, cv);
     retval = 0;
   done:
     if (origtype)
@@ -2236,7 +2264,7 @@ ys_populate_feature(clicon_handle h,
     cv_bool_set(cv, found);
     if (found)
 	clicon_debug(1, "%s %s:%s", __FUNCTION__, module, feature);
-    ys->ys_cv = cv;
+    yang_cv_set(ys, cv);
  ok:
     retval = 0;
  done:
@@ -2490,7 +2518,7 @@ yang_if_feature(clicon_handle h,
     yang_stmt  *yfeat; /* feature yang node */
     int         opand = -1; /* -1:not set, 0:or, 1:and */
     int         enabled = 0;
-    
+    cg_var     *cv;
     if ((vec = clicon_strsep(ys->ys_argument, " \t\r\n", &nvec)) == NULL)
 	goto done;
     /* Two steps: first detect operators
@@ -2579,7 +2607,8 @@ yang_if_feature(clicon_handle h,
 	/* Check if this feature is enabled or not 
 	 * Continue loop to catch unbound features and make verdict at end
 	 */
-	if (yfeat->ys_cv == NULL || !cv_bool_get(yfeat->ys_cv)){    /* disabled */
+	cv = yang_cv_get(yfeat);
+	if (cv == NULL || !cv_bool_get(cv)){    /* disabled */
 	    /* if AND then this is permanently disabled */
 	    if (opand && enabled)
 		enabled = 0;
@@ -3019,14 +3048,15 @@ int
 yang_mandatory(yang_stmt *ys)
 {
     yang_stmt *ym;
+    cg_var    *cv;
 
     /* 1) A leaf, choice, anydata, or anyxml node with a "mandatory"
      *    statement with the value "true". */
     if (ys->ys_keyword == Y_LEAF || ys->ys_keyword == Y_CHOICE ||
 	ys->ys_keyword == Y_ANYDATA || ys->ys_keyword == Y_ANYXML){
 	if ((ym = yang_find(ys, Y_MANDATORY, NULL)) != NULL){
-	    if (ym->ys_cv != NULL) /* shouldnt happen */
+	    if ((cv = yang_cv_get(ym)) != NULL) /* shouldnt happen */
-		return cv_bool_get(ym->ys_cv);
+		return cv_bool_get(cv);
 	}
     }
 #if 0 /* See note above */
@@ -3066,9 +3096,9 @@ yang_config(yang_stmt *ys)
     yang_stmt *ym;
 
     if ((ym = yang_find(ys, Y_CONFIG, NULL)) != NULL){
-	if (ym->ys_cv == NULL) /* shouldnt happen */
+	if (yang_cv_get(ym) == NULL) /* shouldnt happen */
 	    return 1; 
-	return cv_bool_get(ym->ys_cv);
+	return cv_bool_get(yang_cv_get(ym));
     }
     return 1;
 }

lib/src/clixon_yang_cardinality.c

@@ -33,6 +33,7 @@
   ***** END LICENSE BLOCK *****
 
  * Yang cardinality functions according to RFC 7950 
+ * That is, how many children any yang node has
  */
 
 #ifdef HAVE_CONFIG_H
@@ -69,28 +70,36 @@
 /*
  * Types
  */
-/* Encode cardinality according to RFC 7950 
+/* Encode cardinality and ordering according to RFC 7950 
  * Example:
- * 7.20.3.1.  The deviation's Substatements
+ * 7.1.1.  The module's Substatements (subset)
- *
+ *               +--------------+----------+-------------+----------+
- *               +--------------+----------+-------------+
+ *               | substatement | section  | cardinality | ordering |
- *               | substatement | section  | cardinality |
+ *               +--------------+----------+-------------+----------+
- *               +--------------+----------+-------------+
+ *               | contact      | 7.1.8    | 0..1        |   2      |
- *               | description  | 7.21.3   | 0..1        |
+ *               | import       | 7.1.5    | 0..n        |   1      |
- *               | deviate      | 7.20.3.2 | 1..n        |
+ *               | include      | 7.1.6    | 0..n        |   1      |
- *               | reference    | 7.21.4   | 0..1        |
+ *               | namespace    | 7.1.3    | 1           |   0      |
- *               +--------------+----------+-------------+
+ *               | organization | 7.1.7    | 0..1        |   2      |
+ *               | yang-version | 7.1.2    | 1           |   0      |
+ *               +--------------+----------+-------------+----------+
  * The cardinalities are (and how many time they occur)
  *   0..1  149 See ycardmap_01
  *   1..n, 1
  *   0..n  176 (no restrictions)
  *   1     10
+ *
+ * Ordering means in which order the statements may occur. If same ordering the may occur 
+ * in any order. In the example above, namespace and yang-version occurs before
+ * import and include which in turn preceedes contact and organization
+ * Note: order is only relevant for modules and sub-modules
  */
 struct ycard{
     enum rfc_6020 yc_parent;
     enum rfc_6020 yc_child;
     int           yc_min;
     int           yc_max;
+    int           yc_order;
 };
 
 /*
@@ -107,341 +116,341 @@ struct ycard{
  */
 #define NMAX 1000000 /* Just a large number */
 static const struct ycard yclist[] = {
-    {Y_ACTION,  Y_DESCRIPTION, 0, 1},
+    {Y_ACTION,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_ACTION,  Y_GROUPING, 0, NMAX},
+    {Y_ACTION,  Y_GROUPING, 0, NMAX, 0},
-    {Y_ACTION,  Y_IF_FEATURE, 0, NMAX},
+    {Y_ACTION,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_ACTION,  Y_INPUT, 0, 1},
+    {Y_ACTION,  Y_INPUT, 0, 1, 0},
-    {Y_ACTION,  Y_OUTPUT, 0, 1},
+    {Y_ACTION,  Y_OUTPUT, 0, 1, 0},
-    {Y_ACTION,  Y_REFERENCE, 0, 1},
+    {Y_ACTION,  Y_REFERENCE, 0, 1, 0},
-    {Y_ACTION,  Y_STATUS, 0, 1},
+    {Y_ACTION,  Y_STATUS, 0, 1, 0},
-    {Y_ACTION,  Y_TYPEDEF, 0, NMAX},
+    {Y_ACTION,  Y_TYPEDEF, 0, NMAX, 0},
-    {Y_ANYDATA,  Y_CONFIG, 0, 1},
+    {Y_ANYDATA,  Y_CONFIG, 0, 1, 0},
-    {Y_ANYDATA,  Y_DESCRIPTION, 0, 1},
+    {Y_ANYDATA,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_ANYDATA,  Y_IF_FEATURE, 0, NMAX},
+    {Y_ANYDATA,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_ANYDATA,  Y_MANDATORY, 0, 1},
+    {Y_ANYDATA,  Y_MANDATORY, 0, 1, 0},
-    {Y_ANYDATA,  Y_MUST, 0, NMAX},
+    {Y_ANYDATA,  Y_MUST, 0, NMAX, 0},
-    {Y_ANYDATA,  Y_REFERENCE, 0, 1},
+    {Y_ANYDATA,  Y_REFERENCE, 0, 1, 0},
-    {Y_ANYDATA,  Y_STATUS, 0, 1},
+    {Y_ANYDATA,  Y_STATUS, 0, 1, 0},
-    {Y_ANYDATA,  Y_WHEN, 0, 1},
+    {Y_ANYDATA,  Y_WHEN, 0, 1, 0},
-    {Y_ANYXML,  Y_CONFIG, 0, 1},
+    {Y_ANYXML,  Y_CONFIG, 0, 1, 0},
-    {Y_ANYXML,  Y_DESCRIPTION, 0, 1},
+    {Y_ANYXML,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_ANYXML,  Y_IF_FEATURE, 0, NMAX},
+    {Y_ANYXML,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_ANYXML,  Y_MANDATORY, 0, 1},
+    {Y_ANYXML,  Y_MANDATORY, 0, 1, 0},
-    {Y_ANYXML,  Y_MUST, 0, NMAX},
+    {Y_ANYXML,  Y_MUST, 0, NMAX, 0},
-    {Y_ANYXML,  Y_REFERENCE, 0, 1},
+    {Y_ANYXML,  Y_REFERENCE, 0, 1, 0},
-    {Y_ANYXML,  Y_STATUS, 0, 1},
+    {Y_ANYXML,  Y_STATUS, 0, 1, 0},
-    {Y_ANYXML,  Y_WHEN, 0, 1},
+    {Y_ANYXML,  Y_WHEN, 0, 1, 0},
-    {Y_ARGUMENT,  Y_YIN_ELEMENT, 0, 1},
+    {Y_ARGUMENT,  Y_YIN_ELEMENT, 0, 1, 0},
-    {Y_AUGMENT,  Y_ACTION, 0, NMAX},
+    {Y_AUGMENT,  Y_ACTION, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_ANYDATA, 0, NMAX},
+    {Y_AUGMENT,  Y_ANYDATA, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_ANYXML, 0, NMAX},
+    {Y_AUGMENT,  Y_ANYXML, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_CASE, 0, NMAX},
+    {Y_AUGMENT,  Y_CASE, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_CHOICE, 0, NMAX},
+    {Y_AUGMENT,  Y_CHOICE, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_CONTAINER, 0, NMAX},
+    {Y_AUGMENT,  Y_CONTAINER, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_DESCRIPTION, 0, 1},
+    {Y_AUGMENT,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_AUGMENT,  Y_IF_FEATURE, 0, NMAX},
+    {Y_AUGMENT,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_LEAF, 0, NMAX},
+    {Y_AUGMENT,  Y_LEAF, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_LEAF_LIST, 0, NMAX},
+    {Y_AUGMENT,  Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_LIST, 0, NMAX},
+    {Y_AUGMENT,  Y_LIST, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_NOTIFICATION, 0, NMAX},
+    {Y_AUGMENT,  Y_NOTIFICATION, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_REFERENCE, 0, 1},
+    {Y_AUGMENT,  Y_REFERENCE, 0, 1, 0},
-    {Y_AUGMENT,  Y_STATUS, 0, 1},
+    {Y_AUGMENT,  Y_STATUS, 0, 1, 0},
-    {Y_AUGMENT,  Y_USES, 0, NMAX},
+    {Y_AUGMENT,  Y_USES, 0, NMAX, 0},
-    {Y_AUGMENT,  Y_WHEN, 0, 1},
+    {Y_AUGMENT,  Y_WHEN, 0, 1, 0},
-    {Y_BELONGS_TO, Y_PREFIX,    1, 1},
+    {Y_BELONGS_TO, Y_PREFIX,    1, 1, 0},
-    {Y_BIT,  Y_DESCRIPTION, 0, 1},
+    {Y_BIT,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_BIT,  Y_IF_FEATURE, 0, NMAX},
+    {Y_BIT,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_BIT,  Y_POSITION, 0, 1},
+    {Y_BIT,  Y_POSITION, 0, 1, 0},
-    {Y_BIT,  Y_REFERENCE, 0, 1},
+    {Y_BIT,  Y_REFERENCE, 0, 1, 0},
-    {Y_BIT,  Y_STATUS, 0, 1},
+    {Y_BIT,  Y_STATUS, 0, 1, 0},
-    {Y_CASE,  Y_ANYDATA, 0, NMAX},
+    {Y_CASE,  Y_ANYDATA, 0, NMAX, 0},
-    {Y_CASE,  Y_ANYXML, 0, NMAX},
+    {Y_CASE,  Y_ANYXML, 0, NMAX, 0},
-    {Y_CASE,  Y_CHOICE, 0, NMAX},
+    {Y_CASE,  Y_CHOICE, 0, NMAX, 0},
-    {Y_CASE,  Y_CONTAINER, 0, NMAX},
+    {Y_CASE,  Y_CONTAINER, 0, NMAX, 0},
-    {Y_CASE,  Y_DESCRIPTION, 0, 1},
+    {Y_CASE,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_CASE,  Y_IF_FEATURE, 0, NMAX},
+    {Y_CASE,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_CASE,  Y_LEAF, 0, NMAX},
+    {Y_CASE,  Y_LEAF, 0, NMAX, 0},
-    {Y_CASE,  Y_LEAF_LIST, 0, NMAX},
+    {Y_CASE,  Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_CASE,  Y_LIST, 0, NMAX},
+    {Y_CASE,  Y_LIST, 0, NMAX, 0},
-    {Y_CASE,  Y_REFERENCE, 0, 1},
+    {Y_CASE,  Y_REFERENCE, 0, 1, 0},
-    {Y_CASE,  Y_STATUS, 0, 1},
+    {Y_CASE,  Y_STATUS, 0, 1, 0},
-    {Y_CASE,  Y_USES, 0, NMAX},
+    {Y_CASE,  Y_USES, 0, NMAX, 0},
-    {Y_CASE,  Y_WHEN, 0, 1},
+    {Y_CASE,  Y_WHEN, 0, 1, 0},
-    {Y_CHOICE,  Y_ANYXML, 0, NMAX},
+    {Y_CHOICE,  Y_ANYXML, 0, NMAX, 0},
-    {Y_CHOICE,  Y_CASE, 0, NMAX},
+    {Y_CHOICE,  Y_CASE, 0, NMAX, 0},
-    {Y_CHOICE,  Y_CHOICE, 0, NMAX},
+    {Y_CHOICE,  Y_CHOICE, 0, NMAX, 0},
-    {Y_CHOICE,  Y_CONFIG, 0, 1},
+    {Y_CHOICE,  Y_CONFIG, 0, 1, 0},
-    {Y_CHOICE,  Y_CONTAINER, 0, NMAX},
+    {Y_CHOICE,  Y_CONTAINER, 0, NMAX, 0},
-    {Y_CHOICE,  Y_DEFAULT, 0, 1},
+    {Y_CHOICE,  Y_DEFAULT, 0, 1, 0},
-    {Y_CHOICE,  Y_DESCRIPTION, 0, 1},
+    {Y_CHOICE,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_CHOICE,  Y_IF_FEATURE, 0, NMAX},
+    {Y_CHOICE,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_CHOICE,  Y_LEAF, 0, NMAX},
+    {Y_CHOICE,  Y_LEAF, 0, NMAX, 0},
-    {Y_CHOICE,  Y_LEAF_LIST, 0, NMAX},
+    {Y_CHOICE,  Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_CHOICE,  Y_LIST, 0, NMAX},
+    {Y_CHOICE,  Y_LIST, 0, NMAX, 0},
-    {Y_CHOICE,  Y_MANDATORY, 0, 1},
+    {Y_CHOICE,  Y_MANDATORY, 0, 1, 0},
-    {Y_CHOICE,  Y_REFERENCE, 0, 1},
+    {Y_CHOICE,  Y_REFERENCE, 0, 1, 0},
-    {Y_CHOICE,  Y_STATUS, 0, 1},
+    {Y_CHOICE,  Y_STATUS, 0, 1, 0},
-    {Y_CHOICE,  Y_WHEN, 0, 1},
+    {Y_CHOICE,  Y_WHEN, 0, 1, 0},
-    {Y_CHOICE, Y_ANYDATA, 0, NMAX},
+    {Y_CHOICE, Y_ANYDATA, 0, NMAX, 0},
-    {Y_CONTAINER, Y_ACTION,   0, NMAX},
+    {Y_CONTAINER, Y_ACTION,   0, NMAX, 0},
-    {Y_CONTAINER, Y_ANYDATA, 0, NMAX},
+    {Y_CONTAINER, Y_ANYDATA, 0, NMAX, 0},
-    {Y_CONTAINER, Y_ANYXML, 0, NMAX},
+    {Y_CONTAINER, Y_ANYXML, 0, NMAX, 0},
-    {Y_CONTAINER, Y_CHOICE, 0, NMAX},
+    {Y_CONTAINER, Y_CHOICE, 0, NMAX, 0},
-    {Y_CONTAINER, Y_CONFIG, 0, 1},
+    {Y_CONTAINER, Y_CONFIG, 0, 1, 0},
-    {Y_CONTAINER, Y_CONTAINER, 0, NMAX},
+    {Y_CONTAINER, Y_CONTAINER, 0, NMAX, 0},
-    {Y_CONTAINER, Y_DESCRIPTION, 0, 1},
+    {Y_CONTAINER, Y_DESCRIPTION, 0, 1, 0},
-    {Y_CONTAINER, Y_GROUPING, 0, NMAX},
+    {Y_CONTAINER, Y_GROUPING, 0, NMAX, 0},
-    {Y_CONTAINER, Y_IF_FEATURE, 0, NMAX},
+    {Y_CONTAINER, Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_CONTAINER, Y_LEAF, 0, NMAX},
+    {Y_CONTAINER, Y_LEAF, 0, NMAX, 0},
-    {Y_CONTAINER, Y_LEAF_LIST, 0, NMAX},
+    {Y_CONTAINER, Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_CONTAINER, Y_LIST, 0, NMAX},
+    {Y_CONTAINER, Y_LIST, 0, NMAX, 0},
-    {Y_CONTAINER, Y_MUST, 0, NMAX},
+    {Y_CONTAINER, Y_MUST, 0, NMAX, 0},
-    {Y_CONTAINER, Y_NOTIFICATION, 0, NMAX},
+    {Y_CONTAINER, Y_NOTIFICATION, 0, NMAX, 0},
-    {Y_CONTAINER, Y_PRESENCE, 0, 1},
+    {Y_CONTAINER, Y_PRESENCE, 0, 1, 0},
-    {Y_CONTAINER, Y_REFERENCE, 0, 1},
+    {Y_CONTAINER, Y_REFERENCE, 0, 1, 0},
-    {Y_CONTAINER, Y_STATUS, 0, 1},
+    {Y_CONTAINER, Y_STATUS, 0, 1, 0},
-    {Y_CONTAINER, Y_TYPEDEF, 0, NMAX},
+    {Y_CONTAINER, Y_TYPEDEF, 0, NMAX, 0},
-    {Y_CONTAINER, Y_USES, 0, NMAX},
+    {Y_CONTAINER, Y_USES, 0, NMAX, 0},
-    {Y_CONTAINER, Y_WHEN, 0, 1},
+    {Y_CONTAINER, Y_WHEN, 0, 1, 0},
-    {Y_DEVIATE,  Y_CONFIG, 0, 1},
+    {Y_DEVIATE,  Y_CONFIG, 0, 1, 0},
-    {Y_DEVIATE,  Y_DEFAULT, 0, NMAX},
+    {Y_DEVIATE,  Y_DEFAULT, 0, NMAX, 0},
-    {Y_DEVIATE,  Y_MANDATORY, 0, 1},
+    {Y_DEVIATE,  Y_MANDATORY, 0, 1, 0},
-    {Y_DEVIATE,  Y_MAX_ELEMENTS, 0, 1},
+    {Y_DEVIATE,  Y_MAX_ELEMENTS, 0, 1, 0},
-    {Y_DEVIATE,  Y_MIN_ELEMENTS, 0, 1},
+    {Y_DEVIATE,  Y_MIN_ELEMENTS, 0, 1, 0},
-    {Y_DEVIATE,  Y_MUST, 0, NMAX},
+    {Y_DEVIATE,  Y_MUST, 0, NMAX, 0},
-    {Y_DEVIATE,  Y_TYPE, 0, 1},
+    {Y_DEVIATE,  Y_TYPE, 0, 1, 0},
-    {Y_DEVIATE,  Y_UNIQUE, 0, NMAX},
+    {Y_DEVIATE,  Y_UNIQUE, 0, NMAX, 0},
-    {Y_DEVIATE,  Y_UNITS, 0, 1},
+    {Y_DEVIATE,  Y_UNITS, 0, 1, 0},
-    {Y_DEVIATION,  Y_DESCRIPTION, 0, 1},
+    {Y_DEVIATION,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_DEVIATION,  Y_DEVIATE, 1, NMAX},
+    {Y_DEVIATION,  Y_DEVIATE, 1, NMAX, 0},
-    {Y_DEVIATION,  Y_REFERENCE, 0, 1},
+    {Y_DEVIATION,  Y_REFERENCE, 0, 1, 0},
-    {Y_ENUM,  Y_DESCRIPTION, 0, 1},
+    {Y_ENUM,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_ENUM,  Y_IF_FEATURE, 0, NMAX},
+    {Y_ENUM,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_ENUM,  Y_REFERENCE, 0, 1},
+    {Y_ENUM,  Y_REFERENCE, 0, 1, 0},
-    {Y_ENUM,  Y_STATUS, 0, 1},
+    {Y_ENUM,  Y_STATUS, 0, 1, 0},
-    {Y_ENUM,  Y_VALUE, 0, 1},
+    {Y_ENUM,  Y_VALUE, 0, 1, 0},
-    {Y_EXTENSION,  Y_ARGUMENT, 0, 1},
+    {Y_EXTENSION,  Y_ARGUMENT, 0, 1, 0},
-    {Y_EXTENSION,  Y_DESCRIPTION, 0, 1},
+    {Y_EXTENSION,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_EXTENSION,  Y_REFERENCE, 0, 1},
+    {Y_EXTENSION,  Y_REFERENCE, 0, 1, 0},
-    {Y_EXTENSION,  Y_STATUS, 0, 1},
+    {Y_EXTENSION,  Y_STATUS, 0, 1, 0},
-    {Y_FEATURE,  Y_DESCRIPTION, 0, 1},
+    {Y_FEATURE,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_FEATURE,  Y_IF_FEATURE, 0, NMAX},
+    {Y_FEATURE,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_FEATURE,  Y_REFERENCE, 0, 1},
+    {Y_FEATURE,  Y_REFERENCE, 0, 1, 0},
-    {Y_FEATURE,  Y_STATUS, 0, 1},
+    {Y_FEATURE,  Y_STATUS, 0, 1, 0},
-    {Y_GROUPING,  Y_ACTION, 0, NMAX},
+    {Y_GROUPING,  Y_ACTION, 0, NMAX, 0},
-    {Y_GROUPING,  Y_ANYDATA, 0, NMAX},
+    {Y_GROUPING,  Y_ANYDATA, 0, NMAX, 0},
-    {Y_GROUPING,  Y_ANYXML, 0, NMAX},
+    {Y_GROUPING,  Y_ANYXML, 0, NMAX, 0},
-    {Y_GROUPING,  Y_CHOICE, 0, NMAX},
+    {Y_GROUPING,  Y_CHOICE, 0, NMAX, 0},
-    {Y_GROUPING,  Y_CONTAINER, 0, NMAX},
+    {Y_GROUPING,  Y_CONTAINER, 0, NMAX, 0},
-    {Y_GROUPING,  Y_DESCRIPTION, 0, 1},
+    {Y_GROUPING,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_GROUPING,  Y_GROUPING, 0, NMAX},
+    {Y_GROUPING,  Y_GROUPING, 0, NMAX, 0},
-    {Y_GROUPING,  Y_LEAF, 0, NMAX},
+    {Y_GROUPING,  Y_LEAF, 0, NMAX, 0},
-    {Y_GROUPING,  Y_LEAF_LIST, 0, NMAX},
+    {Y_GROUPING,  Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_GROUPING,  Y_LIST, 0, NMAX},
+    {Y_GROUPING,  Y_LIST, 0, NMAX, 0},
-    {Y_GROUPING,  Y_NOTIFICATION, 0, NMAX},
+    {Y_GROUPING,  Y_NOTIFICATION, 0, NMAX, 0},
-    {Y_GROUPING,  Y_REFERENCE, 0, 1},
+    {Y_GROUPING,  Y_REFERENCE, 0, 1, 0},
-    {Y_GROUPING,  Y_STATUS, 0, 1},
+    {Y_GROUPING,  Y_STATUS, 0, 1, 0},
-    {Y_GROUPING,  Y_TYPEDEF, 0, NMAX},
+    {Y_GROUPING,  Y_TYPEDEF, 0, NMAX, 0},
-    {Y_GROUPING,  Y_USES, 0, NMAX},
+    {Y_GROUPING,  Y_USES, 0, NMAX, 0},
-    {Y_IDENTITY,  Y_BASE, 0, NMAX},
+    {Y_IDENTITY,  Y_BASE, 0, NMAX, 0},
-    {Y_IDENTITY,  Y_DESCRIPTION, 0, 1},
+    {Y_IDENTITY,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_IDENTITY,  Y_IF_FEATURE, 0, NMAX},
+    {Y_IDENTITY,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_IDENTITY,  Y_REFERENCE, 0, 1},
+    {Y_IDENTITY,  Y_REFERENCE, 0, 1, 0},
-    {Y_IDENTITY,  Y_STATUS, 0, 1},
+    {Y_IDENTITY,  Y_STATUS, 0, 1, 0},
-    {Y_IMPORT, Y_DESCRIPTION,  0, 1},
+    {Y_IMPORT, Y_DESCRIPTION,  0, 1, 0},
-    {Y_IMPORT, Y_PREFIX,       1, 1},
+    {Y_IMPORT, Y_PREFIX,       1, 1, 0},
-    {Y_IMPORT, Y_REFERENCE,    0, 1},
+    {Y_IMPORT, Y_REFERENCE,    0, 1, 0},
-    {Y_IMPORT, Y_REVISION_DATE,0, 1},
+    {Y_IMPORT, Y_REVISION_DATE,0, 1, 0},
-    {Y_INCLUDE, Y_DESCRIPTION,  0, 1},
+    {Y_INCLUDE, Y_DESCRIPTION,  0, 1, 0},
-    {Y_INCLUDE, Y_REFERENCE,    0, 1},
+    {Y_INCLUDE, Y_REFERENCE,    0, 1, 0},
-    {Y_INCLUDE, Y_REVISION_DATE,0, 1},
+    {Y_INCLUDE, Y_REVISION_DATE,0, 1, 0},
-    {Y_INPUT,  Y_ANYDATA, 0, NMAX},
+    {Y_INPUT,  Y_ANYDATA, 0, NMAX, 0},
-    {Y_INPUT,  Y_ANYXML, 0, NMAX},
+    {Y_INPUT,  Y_ANYXML, 0, NMAX, 0},
-    {Y_INPUT,  Y_CHOICE, 0, NMAX},
+    {Y_INPUT,  Y_CHOICE, 0, NMAX, 0},
-    {Y_INPUT,  Y_CONTAINER, 0, NMAX},
+    {Y_INPUT,  Y_CONTAINER, 0, NMAX, 0},
-    {Y_INPUT,  Y_GROUPING, 0, NMAX},
+    {Y_INPUT,  Y_GROUPING, 0, NMAX, 0},
-    {Y_INPUT,  Y_LEAF, 0, NMAX},
+    {Y_INPUT,  Y_LEAF, 0, NMAX, 0},
-    {Y_INPUT,  Y_LEAF_LIST, 0, NMAX},
+    {Y_INPUT,  Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_INPUT,  Y_LIST, 0, NMAX},
+    {Y_INPUT,  Y_LIST, 0, NMAX, 0},
-    {Y_INPUT,  Y_MUST, 0, NMAX},
+    {Y_INPUT,  Y_MUST, 0, NMAX, 0},
-    {Y_INPUT,  Y_TYPEDEF, 0, NMAX},
+    {Y_INPUT,  Y_TYPEDEF, 0, NMAX, 0},
-    {Y_INPUT,  Y_USES, 0, NMAX},
+    {Y_INPUT,  Y_USES, 0, NMAX, 0},
-    {Y_LEAF, Y_CONFIG, 0, 1},
+    {Y_LEAF, Y_CONFIG, 0, 1, 0},
-    {Y_LEAF, Y_DEFAULT, 0, 1},
+    {Y_LEAF, Y_DEFAULT, 0, 1, 0},
-    {Y_LEAF, Y_DESCRIPTION, 0, 1},
+    {Y_LEAF, Y_DESCRIPTION, 0, 1, 0},
-    {Y_LEAF, Y_IF_FEATURE, 0, NMAX},
+    {Y_LEAF, Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_LEAF, Y_MANDATORY, 0, 1},
+    {Y_LEAF, Y_MANDATORY, 0, 1, 0},
-    {Y_LEAF, Y_MUST, 0, NMAX},
+    {Y_LEAF, Y_MUST, 0, NMAX, 0},
-    {Y_LEAF, Y_REFERENCE, 0, 1},
+    {Y_LEAF, Y_REFERENCE, 0, 1, 0},
-    {Y_LEAF, Y_STATUS, 0, 1},
+    {Y_LEAF, Y_STATUS, 0, 1, 0},
-    {Y_LEAF, Y_TYPE, 1, 1},
+    {Y_LEAF, Y_TYPE, 1, 1, 0},
-    {Y_LEAF, Y_UNITS, 0, 1},
+    {Y_LEAF, Y_UNITS, 0, 1, 0},
-    {Y_LEAF, Y_WHEN, 0, 1},
+    {Y_LEAF, Y_WHEN, 0, 1, 0},
-    {Y_LEAF_LIST, Y_CONFIG, 0, 1},
+    {Y_LEAF_LIST, Y_CONFIG, 0, 1, 0},
-    {Y_LEAF_LIST, Y_DEFAULT, 0, NMAX},
+    {Y_LEAF_LIST, Y_DEFAULT, 0, NMAX, 0},
-    {Y_LEAF_LIST, Y_DESCRIPTION, 0, 1},
+    {Y_LEAF_LIST, Y_DESCRIPTION, 0, 1, 0},
-    {Y_LEAF_LIST, Y_IF_FEATURE, 0, NMAX},
+    {Y_LEAF_LIST, Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_LEAF_LIST, Y_MAX_ELEMENTS, 0, 1},
+    {Y_LEAF_LIST, Y_MAX_ELEMENTS, 0, 1, 0},
-    {Y_LEAF_LIST, Y_MIN_ELEMENTS, 0, 1},
+    {Y_LEAF_LIST, Y_MIN_ELEMENTS, 0, 1, 0},
-    {Y_LEAF_LIST, Y_MUST, 0, NMAX},
+    {Y_LEAF_LIST, Y_MUST, 0, NMAX, 0},
-    {Y_LEAF_LIST, Y_ORDERED_BY, 0, 1},
+    {Y_LEAF_LIST, Y_ORDERED_BY, 0, 1, 0},
-    {Y_LEAF_LIST, Y_REFERENCE, 0, 1},
+    {Y_LEAF_LIST, Y_REFERENCE, 0, 1, 0},
-    {Y_LEAF_LIST, Y_STATUS, 0, 1},
+    {Y_LEAF_LIST, Y_STATUS, 0, 1, 0},
-    {Y_LEAF_LIST, Y_TYPE, 1, 1},
+    {Y_LEAF_LIST, Y_TYPE, 1, 1, 0},
-    {Y_LEAF_LIST, Y_UNITS, 0, 1},
+    {Y_LEAF_LIST, Y_UNITS, 0, 1, 0},
-    {Y_LEAF_LIST, Y_WHEN, 0, 1},
+    {Y_LEAF_LIST, Y_WHEN, 0, 1, 0},
-    {Y_LENGTH,  Y_DESCRIPTION, 0, 1},
+    {Y_LENGTH,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_LENGTH,  Y_ERROR_APP_TAG, 0, 1},
+    {Y_LENGTH,  Y_ERROR_APP_TAG, 0, 1, 0},
-    {Y_LENGTH,  Y_ERROR_MESSAGE, 0, 1},
+    {Y_LENGTH,  Y_ERROR_MESSAGE, 0, 1, 0},
-    {Y_LENGTH,  Y_REFERENCE, 0, 1},
+    {Y_LENGTH,  Y_REFERENCE, 0, 1, 0},
-    {Y_LIST, Y_ACTION, 0, NMAX},
+    {Y_LIST, Y_ACTION, 0, NMAX, 0},
-    {Y_LIST, Y_ANYDATA, 0, NMAX},
+    {Y_LIST, Y_ANYDATA, 0, NMAX, 0},
-    {Y_LIST, Y_ANYXML, 0, NMAX},
+    {Y_LIST, Y_ANYXML, 0, NMAX, 0},
-    {Y_LIST, Y_CHOICE, 0, NMAX},
+    {Y_LIST, Y_CHOICE, 0, NMAX, 0},
-    {Y_LIST, Y_CONFIG, 0, 1},
+    {Y_LIST, Y_CONFIG, 0, 1, 0},
-    {Y_LIST, Y_CONTAINER, 0, NMAX},
+    {Y_LIST, Y_CONTAINER, 0, NMAX, 0},
-    {Y_LIST, Y_DESCRIPTION, 0, 1},
+    {Y_LIST, Y_DESCRIPTION, 0, 1, 0},
-    {Y_LIST, Y_GROUPING, 0, NMAX},
+    {Y_LIST, Y_GROUPING, 0, NMAX, 0},
-    {Y_LIST, Y_IF_FEATURE, 0, NMAX},
+    {Y_LIST, Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_LIST, Y_KEY, 0, 1},
+    {Y_LIST, Y_KEY, 0, 1, 0},
-    {Y_LIST, Y_LEAF, 0, NMAX},
+    {Y_LIST, Y_LEAF, 0, NMAX, 0},
-    {Y_LIST, Y_LEAF_LIST, 0, NMAX},
+    {Y_LIST, Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_LIST, Y_LIST, 0, NMAX},
+    {Y_LIST, Y_LIST, 0, NMAX, 0},
-    {Y_LIST, Y_MAX_ELEMENTS, 0, 1},
+    {Y_LIST, Y_MAX_ELEMENTS, 0, 1, 0},
-    {Y_LIST, Y_MIN_ELEMENTS, 0, 1},
+    {Y_LIST, Y_MIN_ELEMENTS, 0, 1, 0},
-    {Y_LIST, Y_MUST, 0, NMAX},
+    {Y_LIST, Y_MUST, 0, NMAX, 0},
-    {Y_LIST, Y_NOTIFICATION, 0, NMAX},
+    {Y_LIST, Y_NOTIFICATION, 0, NMAX, 0},
-    {Y_LIST, Y_ORDERED_BY, 0, 1},
+    {Y_LIST, Y_ORDERED_BY, 0, 1, 0},
-    {Y_LIST, Y_REFERENCE, 0, 1},
+    {Y_LIST, Y_REFERENCE, 0, 1, 0},
-    {Y_LIST, Y_STATUS, 0, 1},
+    {Y_LIST, Y_STATUS, 0, 1, 0},
-    {Y_LIST, Y_TYPEDEF, 0, NMAX},
+    {Y_LIST, Y_TYPEDEF, 0, NMAX, 0},
-    {Y_LIST, Y_UNIQUE, 0, NMAX},
+    {Y_LIST, Y_UNIQUE, 0, NMAX, 0},
-    {Y_LIST, Y_USES, 0, NMAX},
+    {Y_LIST, Y_USES, 0, NMAX, 0},
-    {Y_LIST, Y_WHEN, 0,1},
+    {Y_LIST, Y_WHEN, 0,1, 0},
-    {Y_MODULE, Y_ANYDATA,      0, NMAX},
+    {Y_MODULE, Y_ANYDATA,      0, NMAX, 4},
-    {Y_MODULE, Y_ANYXML,       0, NMAX},
+    {Y_MODULE, Y_ANYXML,       0, NMAX, 4},
-    {Y_MODULE, Y_AUGMENT,      0, NMAX},
+    {Y_MODULE, Y_AUGMENT,      0, NMAX, 4},
-    {Y_MODULE, Y_CHOICE,       0, NMAX},
+    {Y_MODULE, Y_CHOICE,       0, NMAX, 4},
-    {Y_MODULE, Y_CONTACT,      0, 1},
+    {Y_MODULE, Y_CONTACT,      0, 1,    2},
-    {Y_MODULE, Y_CONTAINER,    0, NMAX},
+    {Y_MODULE, Y_CONTAINER,    0, NMAX, 4},
-    {Y_MODULE, Y_DESCRIPTION,  0, 1},
+    {Y_MODULE, Y_DESCRIPTION,  0, 1,    2},
-    {Y_MODULE, Y_DEVIATION,    0, NMAX},
+    {Y_MODULE, Y_DEVIATION,    0, NMAX, 4},
-    {Y_MODULE, Y_EXTENSION,    0, NMAX},
+    {Y_MODULE, Y_EXTENSION,    0, NMAX, 4},
-    {Y_MODULE, Y_FEATURE,      0, NMAX},
+    {Y_MODULE, Y_FEATURE,      0, NMAX, 4},
-    {Y_MODULE, Y_GROUPING,     0, NMAX},
+    {Y_MODULE, Y_GROUPING,     0, NMAX, 4},
-    {Y_MODULE, Y_IDENTITY,     0, NMAX},
+    {Y_MODULE, Y_IDENTITY,     0, NMAX, 4},
-    {Y_MODULE, Y_IMPORT,       0, NMAX},
+    {Y_MODULE, Y_IMPORT,       0, NMAX, 1},
-    {Y_MODULE, Y_INCLUDE,      0, NMAX},
+    {Y_MODULE, Y_INCLUDE,      0, NMAX, 1},
-    {Y_MODULE, Y_LEAF,         0, NMAX},
+    {Y_MODULE, Y_LEAF,         0, NMAX, 4},
-    {Y_MODULE, Y_LEAF_LIST,    0, NMAX},
+    {Y_MODULE, Y_LEAF_LIST,    0, NMAX, 4},
-    {Y_MODULE, Y_LIST,         0, NMAX},
+    {Y_MODULE, Y_LIST,         0, NMAX, 4},
-    {Y_MODULE, Y_NAMESPACE,    1, 1},
+    {Y_MODULE, Y_NAMESPACE,    1, 1, 0},
-    {Y_MODULE, Y_NOTIFICATION, 0, NMAX},
+    {Y_MODULE, Y_NOTIFICATION, 0, NMAX, 4},
-    {Y_MODULE, Y_ORGANIZATION, 0, 1},
+    {Y_MODULE, Y_ORGANIZATION, 0, 1,    2},
-    {Y_MODULE, Y_PREFIX,       1, 1},
+    {Y_MODULE, Y_PREFIX,       1, 1,    0},
-    {Y_MODULE, Y_REFERENCE,    0, 1},
+    {Y_MODULE, Y_REFERENCE,    0, 1,    2},
-    {Y_MODULE, Y_REVISION,     0, NMAX},
+    {Y_MODULE, Y_REVISION,     0, NMAX, 3},
-    {Y_MODULE, Y_RPC,          0, NMAX},
+    {Y_MODULE, Y_RPC,          0, NMAX, 4},
-    {Y_MODULE, Y_TYPEDEF,      0, NMAX},
+    {Y_MODULE, Y_TYPEDEF,      0, NMAX, 4},
-    {Y_MODULE, Y_USES,         0, NMAX},
+    {Y_MODULE, Y_USES,         0, NMAX, 4},
-    {Y_MODULE, Y_YANG_VERSION, 0, 1},
+    {Y_MODULE, Y_YANG_VERSION, 0, 1,    0},
-    {Y_MUST, Y_DESCRIPTION, 0, 1},
+    {Y_MUST, Y_DESCRIPTION, 0, 1, 0},
-    {Y_MUST, Y_ERROR_APP_TAG, 0, 1},
+    {Y_MUST, Y_ERROR_APP_TAG, 0, 1, 0},
-    {Y_MUST, Y_ERROR_MESSAGE, 0, 1},
+    {Y_MUST, Y_ERROR_MESSAGE, 0, 1, 0},
-    {Y_MUST, Y_REFERENCE, 0, 1},
+    {Y_MUST, Y_REFERENCE, 0, 1, 0},
-    {Y_NOTIFICATION,  Y_ANYDATA, 0, NMAX},
+    {Y_NOTIFICATION,  Y_ANYDATA, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_ANYXML, 0, NMAX},
+    {Y_NOTIFICATION,  Y_ANYXML, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_CHOICE, 0, NMAX},
+    {Y_NOTIFICATION,  Y_CHOICE, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_CONTAINER, 0, NMAX},
+    {Y_NOTIFICATION,  Y_CONTAINER, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_DESCRIPTION, 0, 1},
+    {Y_NOTIFICATION,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_NOTIFICATION,  Y_GROUPING, 0, NMAX},
+    {Y_NOTIFICATION,  Y_GROUPING, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_IF_FEATURE, 0, NMAX},
+    {Y_NOTIFICATION,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_LEAF, 0, NMAX},
+    {Y_NOTIFICATION,  Y_LEAF, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_LEAF_LIST, 0, NMAX},
+    {Y_NOTIFICATION,  Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_LIST, 0, NMAX},
+    {Y_NOTIFICATION,  Y_LIST, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_MUST, 0, NMAX},
+    {Y_NOTIFICATION,  Y_MUST, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_REFERENCE, 0, 1},
+    {Y_NOTIFICATION,  Y_REFERENCE, 0, 1, 0},
-    {Y_NOTIFICATION,  Y_STATUS, 0, 1},
+    {Y_NOTIFICATION,  Y_STATUS, 0, 1, 0},
-    {Y_NOTIFICATION,  Y_TYPEDEF, 0, NMAX},
+    {Y_NOTIFICATION,  Y_TYPEDEF, 0, NMAX, 0},
-    {Y_NOTIFICATION,  Y_USES, 0, NMAX},
+    {Y_NOTIFICATION,  Y_USES, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_ANYDATA, 0, NMAX},
+    {Y_OUTPUT,  Y_ANYDATA, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_ANYXML, 0, NMAX},
+    {Y_OUTPUT,  Y_ANYXML, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_CHOICE, 0, NMAX},
+    {Y_OUTPUT,  Y_CHOICE, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_CONTAINER, 0, NMAX},
+    {Y_OUTPUT,  Y_CONTAINER, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_GROUPING, 0, NMAX},
+    {Y_OUTPUT,  Y_GROUPING, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_LEAF, 0, NMAX},
+    {Y_OUTPUT,  Y_LEAF, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_LEAF_LIST, 0, NMAX},
+    {Y_OUTPUT,  Y_LEAF_LIST, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_LIST, 0, NMAX},
+    {Y_OUTPUT,  Y_LIST, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_MUST, 0, NMAX},
+    {Y_OUTPUT,  Y_MUST, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_TYPEDEF, 0, NMAX},
+    {Y_OUTPUT,  Y_TYPEDEF, 0, NMAX, 0},
-    {Y_OUTPUT,  Y_USES, 0, NMAX},
+    {Y_OUTPUT,  Y_USES, 0, NMAX, 0},
-    {Y_PATTERN,  Y_DESCRIPTION, 0, 1},
+    {Y_PATTERN,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_PATTERN,  Y_ERROR_APP_TAG, 0, 1},
+    {Y_PATTERN,  Y_ERROR_APP_TAG, 0, 1, 0},
-    {Y_PATTERN,  Y_ERROR_MESSAGE, 0, 1},
+    {Y_PATTERN,  Y_ERROR_MESSAGE, 0, 1, 0},
-    {Y_PATTERN,  Y_MODIFIER, 0, 1},
+    {Y_PATTERN,  Y_MODIFIER, 0, 1, 0},
-    {Y_PATTERN,  Y_REFERENCE, 0, 1},
+    {Y_PATTERN,  Y_REFERENCE, 0, 1, 0},
-    {Y_RANGE,  Y_DESCRIPTION, 0, 1},
+    {Y_RANGE,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_RANGE,  Y_ERROR_APP_TAG, 0, 1},
+    {Y_RANGE,  Y_ERROR_APP_TAG, 0, 1, 0},
-    {Y_RANGE,  Y_ERROR_MESSAGE, 0, 1},
+    {Y_RANGE,  Y_ERROR_MESSAGE, 0, 1, 0},
-    {Y_RANGE,  Y_REFERENCE, 0, 1},
+    {Y_RANGE,  Y_REFERENCE, 0, 1, 0},
-    {Y_REVISION, Y_DESCRIPTION, 0, 1},
+    {Y_REVISION, Y_DESCRIPTION, 0, 1, 0},
-    {Y_REVISION, Y_REFERENCE,   0, 1},
+    {Y_REVISION, Y_REFERENCE,   0, 1, 0},
-    {Y_RPC,  Y_DESCRIPTION, 0, 1},
+    {Y_RPC,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_RPC,  Y_GROUPING, 0, NMAX},
+    {Y_RPC,  Y_GROUPING, 0, NMAX, 0},
-    {Y_RPC,  Y_IF_FEATURE, 0, NMAX},
+    {Y_RPC,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_RPC,  Y_INPUT, 0, 1},
+    {Y_RPC,  Y_INPUT, 0, 1, 0},
-    {Y_RPC,  Y_OUTPUT, 0, 1},
+    {Y_RPC,  Y_OUTPUT, 0, 1, 0},
-    {Y_RPC,  Y_REFERENCE, 0, 1},
+    {Y_RPC,  Y_REFERENCE, 0, 1, 0},
-    {Y_RPC,  Y_STATUS, 0, 1},
+    {Y_RPC,  Y_STATUS, 0, 1, 0},
-    {Y_RPC,  Y_TYPEDEF, 0, NMAX},
+    {Y_RPC,  Y_TYPEDEF, 0, NMAX, 0},
-    {Y_SUBMODULE, Y_ANYDATA,    0, NMAX},
+    {Y_SUBMODULE, Y_ANYDATA,    0, NMAX, 4},
-    {Y_SUBMODULE, Y_AUGMENT,    0, NMAX},
+    {Y_SUBMODULE, Y_AUGMENT,    0, NMAX, 4},
-    {Y_SUBMODULE, Y_BELONGS_TO, 1, 1},
+    {Y_SUBMODULE, Y_BELONGS_TO, 1, 1,    0},
-    {Y_SUBMODULE, Y_CHOICE,     0, NMAX},
+    {Y_SUBMODULE, Y_CHOICE,     0, NMAX, 4},
-    {Y_SUBMODULE, Y_CONTACT,    0, 1},
+    {Y_SUBMODULE, Y_CONTACT,    0, 1,    2},
-    {Y_SUBMODULE, Y_CONTAINER,  0, NMAX},
+    {Y_SUBMODULE, Y_CONTAINER,  0, NMAX, 4},
-    {Y_SUBMODULE, Y_DESCRIPTION,0, 1},
+    {Y_SUBMODULE, Y_DESCRIPTION,0, 1,    2},
-    {Y_SUBMODULE, Y_DEVIATION,  0, NMAX},
+    {Y_SUBMODULE, Y_DEVIATION,  0, NMAX, 4},
-    {Y_SUBMODULE, Y_EXTENSION,  0, NMAX},
+    {Y_SUBMODULE, Y_EXTENSION,  0, NMAX, 4},
-    {Y_SUBMODULE, Y_FEATURE,    0, NMAX},
+    {Y_SUBMODULE, Y_FEATURE,    0, NMAX, 4},
-    {Y_SUBMODULE, Y_GROUPING,   0, NMAX},
+    {Y_SUBMODULE, Y_GROUPING,   0, NMAX, 4},
-    {Y_SUBMODULE, Y_IDENTITY,   0, NMAX},
+    {Y_SUBMODULE, Y_IDENTITY,   0, NMAX, 4},
-    {Y_SUBMODULE, Y_IMPORT,     0, NMAX},
+    {Y_SUBMODULE, Y_IMPORT,     0, NMAX, 1},
-    {Y_SUBMODULE, Y_INCLUDE,    0, NMAX},
+    {Y_SUBMODULE, Y_INCLUDE,    0, NMAX, 1},
-    {Y_SUBMODULE, Y_LEAF,       0, NMAX},
+    {Y_SUBMODULE, Y_LEAF,       0, NMAX, 4},
-    {Y_SUBMODULE, Y_LEAF_LIST,  0, NMAX},
+    {Y_SUBMODULE, Y_LEAF_LIST,  0, NMAX, 4},
-    {Y_SUBMODULE, Y_LIST,       0, NMAX},
+    {Y_SUBMODULE, Y_LIST,       0, NMAX, 4},
-    {Y_SUBMODULE, Y_NOTIFICATION,0, NMAX},
+    {Y_SUBMODULE, Y_NOTIFICATION,0, NMAX, 4},
-    {Y_SUBMODULE, Y_ORGANIZATION,0, 1},
+    {Y_SUBMODULE, Y_ORGANIZATION,0, 1,   2},
-    {Y_SUBMODULE, Y_REFERENCE,  0, 1},
+    {Y_SUBMODULE, Y_REFERENCE,  0, 1,    2},
-    {Y_SUBMODULE, Y_REVISION,   0, NMAX},
+    {Y_SUBMODULE, Y_REVISION,   0, NMAX, 3},
-    {Y_SUBMODULE, Y_RPC,        0, NMAX},
+    {Y_SUBMODULE, Y_RPC,        0, NMAX, 4},
-    {Y_SUBMODULE, Y_TYPEDEF,    0, NMAX},
+    {Y_SUBMODULE, Y_TYPEDEF,    0, NMAX, 4},
-    {Y_SUBMODULE, Y_USES,       0, NMAX},
+    {Y_SUBMODULE, Y_USES,       0, NMAX, 4},
-    {Y_SUBMODULE, Y_YANG_VERSION,0, 1}, /* "yang-version" statement is mandatory in YANG version "1.1". */
+    {Y_SUBMODULE, Y_YANG_VERSION,0, 1,   0}, /* "yang-version" statement is mandatory in YANG version "1.1". */
-    {Y_TYPE, Y_BASE,          0, NMAX},
+    {Y_TYPE, Y_BASE,          0, NMAX, 0},
-    {Y_TYPE, Y_BIT,           0, NMAX},
+    {Y_TYPE, Y_BIT,           0, NMAX, 0},
-    {Y_TYPE, Y_ENUM,          0, NMAX},
+    {Y_TYPE, Y_ENUM,          0, NMAX, 0},
-    {Y_TYPE, Y_FRACTION_DIGITS, 0, 1},
+    {Y_TYPE, Y_FRACTION_DIGITS, 0, 1, 0},
-    {Y_TYPE, Y_LENGTH,        0, 1},
+    {Y_TYPE, Y_LENGTH,        0, 1, 0},
-    {Y_TYPE, Y_PATH,          0, 1},
+    {Y_TYPE, Y_PATH,          0, 1, 0},
-    {Y_TYPE, Y_PATTERN,       0, NMAX},
+    {Y_TYPE, Y_PATTERN,       0, NMAX, 0},
-    {Y_TYPE, Y_RANGE,         0, 1},
+    {Y_TYPE, Y_RANGE,         0, 1, 0},
-    {Y_TYPE, Y_REQUIRE_INSTANCE, 0, 1},
+    {Y_TYPE, Y_REQUIRE_INSTANCE, 0, 1, 0},
-    {Y_TYPE, Y_TYPE,          0, NMAX},
+    {Y_TYPE, Y_TYPE,          0, NMAX, 0},
-    {Y_TYPEDEF, Y_DEFAULT,    0, 1},
+    {Y_TYPEDEF, Y_DEFAULT,    0, 1, 0},
-    {Y_TYPEDEF, Y_DESCRIPTION,0, 1},
+    {Y_TYPEDEF, Y_DESCRIPTION,0, 1, 0},
-    {Y_TYPEDEF, Y_REFERENCE,  0, 1},
+    {Y_TYPEDEF, Y_REFERENCE,  0, 1, 0},
-    {Y_TYPEDEF, Y_STATUS,     0, 1},
+    {Y_TYPEDEF, Y_STATUS,     0, 1, 0},
-    {Y_TYPEDEF, Y_TYPE,       1, 1},
+    {Y_TYPEDEF, Y_TYPE,       1, 1, 0},
-    {Y_TYPEDEF, Y_UNITS,      0, 1},
+    {Y_TYPEDEF, Y_UNITS,      0, 1, 0},
-    {Y_USES,  Y_AUGMENT, 0, NMAX},
+    {Y_USES,  Y_AUGMENT, 0, NMAX, 0},
-    {Y_USES,  Y_DESCRIPTION, 0, 1},
+    {Y_USES,  Y_DESCRIPTION, 0, 1, 0},
-    {Y_USES,  Y_IF_FEATURE, 0, NMAX},
+    {Y_USES,  Y_IF_FEATURE, 0, NMAX, 0},
-    {Y_USES,  Y_REFERENCE, 0, 1},
+    {Y_USES,  Y_REFERENCE, 0, 1, 0},
-    {Y_USES,  Y_REFINE, 0, NMAX},
+    {Y_USES,  Y_REFINE, 0, NMAX, 0},
-    {Y_USES,  Y_STATUS, 0, 1},
+    {Y_USES,  Y_STATUS, 0, 1, 0},
-    {Y_USES,  Y_WHEN, 0, 1},
+    {Y_USES,  Y_WHEN, 0, 1, 0},
     {0,}
 };
 
@@ -497,12 +506,17 @@ yang_cardinality(clicon_handle h,
     int                 nr;
     const struct ycard *ycplist; /* ycard parent table*/
     const struct ycard *yc;
+    int                 order;
+    yang_stmt          *yprev = NULL;
     
     pk = yang_keyword_get(yt);
     /* 0) Find parent sub-parts of cardinality vector */
     if ((ycplist = ycard_find(pk, 0, yclist, 0)) == NULL)
 	goto ok; /* skip */
-    /* 1) For all children, if neither in 0..n, 0..1, 1 or 1..n   ->ERROR  */
+    /* 1) For all children, if neither in 0..n, 0..1, 1 or 1..n   ->ERROR  
+     * Also: check monotonically increasing order
+     */
+    order = 0;
     ys = NULL;
     while ((ys = yn_each(yt, ys)) != NULL) {
 	ck = yang_keyword_get(ys);
@@ -518,7 +532,21 @@ yang_cardinality(clicon_handle h,
 		       yang_argument_get(yt));
 	    goto done;
 	}
-    }
+	if (order > yc->yc_order){
+	    clicon_err(OE_YANG, 0, "%s: yang node \"%s\"(%s) which is child of \"%s\"(%s) is not in correct order (should not be after \"%s\"(%s))",
+		       modname,
+		       yang_key2str(ck),
+		       yang_argument_get(ys),
+		       yang_key2str(pk),
+		       yang_argument_get(yt),
+		       yang_key2str(yang_keyword_get(yprev)),
+		       yang_argument_get(yprev));
+	    goto done;	    
+	}
+	if (order < yc->yc_order)
+	    order = yc->yc_order;
+	yprev = ys;
+   }
     /* 2) For all in 1 and 1..n list, if 0 such children          ->ERROR */
     for (yc = &ycplist[0]; (int)yc->yc_parent == pk; yc++){
 	if (yc->yc_min  &&
@@ -541,7 +569,7 @@ yang_cardinality(clicon_handle h,
 	    goto done;
 	}
     }
-    
+
     /* 4) Recurse */
     i = 0;
     while (i< yang_len_get(yt)){ /* Note, children may be removed cant use yn_each */

lib/src/clixon_yang_module.c

@@ -112,6 +112,7 @@ modstate_diff_free(modstate_diff_t *md)
  *
  * Load RFC7895 yang spec, module-set-id, etc.
  * @param[in]     h       Clicon handle
+ * @see netconf_module_load
  */
 int
 yang_modules_init(clicon_handle h)
@@ -328,7 +329,7 @@ yang_modules_state_get(clicon_handle    h,
 	 * Note, list is not sorted since it is state (should not be)
 	 */
 	if (clixon_xml_parse_string(cbuf_get(cb), YB_MODULE, yspec, &x, NULL) < 0){
-	    if (netconf_operation_failed_xml(xret, "protocol", clicon_err_reason)< 0)
+	    if (xret && netconf_operation_failed_xml(xret, "protocol", clicon_err_reason)< 0)
 		goto done;
 	    goto fail;
 	}

lib/src/clixon_yang_parse_lib.c

@@ -633,13 +633,16 @@ yang_expand_uses_node(yang_stmt *yn,
 		       yang_argument_get(yg),
 		       yang_argument_get(ygrouping)
 		       );
-	    goto done;
+		goto done;
 	    }
 	    if (yang_when_xpath_set(yg, wxpath) < 0) 
 		goto done;
 	    if (yang_when_nsc_set(yg, wnsc) < 0) 
 		goto done;
 	}
+	/* This is for extensions that allow list keys to be optional, see restconf_main_extension_cb */
+	if (yang_flag_get(ys, YANG_FLAG_NOKEY))
+	    yang_flag_set(yg, YANG_FLAG_NOKEY);
 	yn->ys_stmt[i+k] = yg;
 	yg->ys_parent = yn;
 	k++;
@@ -1032,6 +1035,10 @@ yang_parse_module(clicon_handle h,
 	ymod = NULL;
 	goto done;
     }
+    /* Sanity check that requested module name matches loaded module
+     * If this does not match, the filename and containing module do not match
+     * RFC 7950 Sec 5.2 
+     */
     if ((yrev = yang_find(ymod, Y_REVISION, NULL)) != NULL)
 	revm = cv_uint32_get(yang_cv_get(yrev));
     if (filename2revision(filename, NULL, &revf) < 0)
@@ -1064,14 +1071,18 @@ yang_parse_recurse(clicon_handle h,
 		   yang_stmt    *ymod,
 		   yang_stmt    *ysp)
 {
-    int         retval = -1;
+    int           retval = -1;
-    yang_stmt  *yi = NULL; /* import */
+    yang_stmt    *yi = NULL; /* import */
-    yang_stmt  *yrev;
+    yang_stmt    *yrev;
-    char       *submodule;
+    yang_stmt    *ybelongto;
-    char       *subrevision;
+    yang_stmt    *yrealmod;
-    yang_stmt  *subymod;
+    char         *submodule;
+    char         *subrevision;
+    yang_stmt    *subymod;
     enum rfc_6020 keyw;
 
+    if (ys_real_module(ymod, &yrealmod) < 0)
+	goto done;
     /* go through all import (modules) and include(submodules) of ysp */
     while ((yi = yn_each(ymod, yi)) != NULL){
 	keyw = yang_keyword_get(yi);
@@ -1091,6 +1102,21 @@ yang_parse_recurse(clicon_handle h,
 	    /* recursive call */
 	    if ((subymod = yang_parse_module(h, submodule, subrevision, ysp)) == NULL)
 		goto done;
+	    /* Sanity check: if submodule, its belongs-to statement shall point to the module */
+	    if (keyw == Y_INCLUDE){
+		ybelongto = yang_find(subymod, Y_BELONGS_TO, NULL);
+		if (ybelongto == NULL){
+		    clicon_err(OE_YANG, ENOENT, "Sub-module \"%s\" does not have a belongs-to statement", submodule); /* shouldnt happen */
+		    goto done;
+		}
+		if (strcmp(yang_argument_get(ybelongto), yang_argument_get(yrealmod)) != 0){
+		    clicon_err(OE_YANG, ENOENT, "Sub-module \"%s\" references module \"%s\" in its belongs-to statement but should reference \"%s\"",
+			       submodule,
+			       yang_argument_get(ybelongto),
+			       yang_argument_get(yrealmod));
+		    goto done;
+		}
+	    }
 	    /* Go through its sub-modules recursively */
 	    if (yang_parse_recurse(h, subymod, ysp) < 0){
 		ymod = NULL;
@@ -1173,7 +1199,7 @@ ys_schemanode_check(yang_stmt *ys,
  * Verify the following rule:
  * RFC 7950 7.8.2: The "key" statement, which MUST be present if the list represents
  *                 configuration and MAY be present otherwise
- * Unless CLICON_YANG_LIST_CHECK is false
+ * Unless CLICON_YANG_LIST_CHECK is false (obsolete)
  * OR it is the "errors" rule of the ietf-restconf spec which seems to be a special case.
  */
 static int 
@@ -1199,28 +1225,22 @@ ys_list_check(clicon_handle h,
     if (keyw == Y_LIST &&
 	yang_find(ys, Y_KEY, NULL) == 0){
 	ymod = ys_module(ys);
-#if 1
+	/* Except nokey exceptions such as rrc 8040 yang-data */
-	/* Except restconf error extension from sanity check, dont know why it has no keys */
+	if (!yang_flag_get(yroot, YANG_FLAG_NOKEY)){
-	if (strcmp(yang_find_mynamespace(ys),"urn:ietf:params:xml:ns:yang:ietf-restconf")==0 &&
+	    /* Note obsolete */
-	    strcmp(yang_argument_get(ys),"error") == 0)
+	    if (clicon_option_bool(h, "CLICON_YANG_LIST_CHECK")){
-	    ;
+		clicon_log(LOG_ERR, "Error: LIST \"%s\" in module \"%s\" lacks key statement which MUST be present (See RFC 7950 Sec 7.8.2)", 
-	else
-#endif
-	    {
-		if (clicon_option_bool(h, "CLICON_YANG_LIST_CHECK")){
-		    clicon_log(LOG_ERR, "Error: LIST \"%s\" in module \"%s\" lacks key statement which MUST be present (See RFC 7950 Sec 7.8.2)", 
-			   yang_argument_get(ys),
-			   yang_argument_get(ymod)
-			   );
-
-		    goto done;
-		}
-		else
-		    clicon_log(LOG_WARNING, "Warning: LIST \"%s\" in module \"%s\" lacks key statement which MUST be present (See RFC 7950 Sec 7.8.2)", 
 			   yang_argument_get(ys),
 			   yang_argument_get(ymod)
 			   );
+		goto done;
 	    }
+	    else
+		clicon_log(LOG_WARNING, "Warning: LIST \"%s\" in module \"%s\" lacks key statement which MUST be present (See RFC 7950 Sec 7.8.2)", 
+			   yang_argument_get(ys),
+			   yang_argument_get(ymod)
+			   );
+	}
     }
     /* Traverse subs */
     if (yang_schemanode(ys) || keyw == Y_MODULE || keyw == Y_SUBMODULE){
@@ -1744,29 +1764,33 @@ cg_var *
 ys_parse(yang_stmt   *ys, 
 	 enum cv_type cvtype)
 {
-    int             cvret;
+    int     cvret;
-    char           *reason = NULL;
+    char   *reason = NULL;
+    cg_var *cv = NULL;
 
-    assert(yang_cv_get(ys) == NULL); /* Cv:s are parsed in different places, difficult to separate */
+    if ((cv = yang_cv_get(ys)) != NULL){
-    if ((ys->ys_cv = cv_new(cvtype)) == NULL){
+	/* eg mandatory in uses is already set and then copied */
+	cv_free(cv);
+	yang_cv_set(ys, NULL);
+    }
+    if ((cv = cv_new(cvtype)) == NULL){
 	clicon_err(OE_YANG, errno, "cv_new"); 
 	goto done;
     }
-    if ((cvret = cv_parse1(yang_argument_get(ys), ys->ys_cv, &reason)) < 0){ /* error */
+    if ((cvret = cv_parse1(yang_argument_get(ys), cv, &reason)) < 0){ /* error */
 	clicon_err(OE_YANG, errno, "parsing cv");
-	ys->ys_cv = NULL;
 	goto done;
     }
     if (cvret == 0){ /* parsing failed */
 	clicon_err(OE_YANG, errno, "Parsing CV: %s", reason);
-	ys->ys_cv = NULL;
 	goto done;
     }
+    yang_cv_set(ys, cv);
     /* cvret == 1 means parsing is OK */
   done:
     if (reason)
 	free(reason);
-    return ys->ys_cv;
+    return yang_cv_get(ys);
 }
 
 /*! First round yang syntactic statement specific checks. No context checks.
@@ -1796,6 +1820,7 @@ ys_parse_sub(yang_stmt *ys,
     char      *reason = NULL;
     int        ret;
     uint32_t   minmax;
+    cg_var    *cv = NULL;
     
     arg = yang_argument_get(ys);
     keyword = yang_keyword_get(ys);
@@ -1803,7 +1828,11 @@ ys_parse_sub(yang_stmt *ys,
     case Y_FRACTION_DIGITS:
 	if (ys_parse(ys, CGV_UINT8) == NULL) 
 	    goto done;
-	fd = cv_uint8_get(ys->ys_cv);
+	if ((cv = yang_cv_get(ys)) == NULL){
+	    clicon_err(OE_YANG, ENOENT, "Unexpected NULL cv");
+	    goto done;
+	}
+	fd = cv_uint8_get(cv);
 	if (fd < 1 || fd > 18){
 	    clicon_err(OE_YANG, errno, "%u: Out of range, should be [1:18]", fd);
 	    goto done;
@@ -1818,11 +1847,12 @@ ys_parse_sub(yang_stmt *ys,
     case Y_REVISION_DATE:  /* YYYY-MM-DD encoded as uint32 YYYYMMDD */
 	if (ys_parse_date_arg(arg, &date) < 0)
 	    goto done;
-	if ((ys->ys_cv = cv_new(CGV_UINT32)) == NULL){
+	if ((cv = cv_new(CGV_UINT32)) == NULL){
 	    clicon_err(OE_YANG, errno, "cv_new"); 
 	    goto done;
 	}
-	cv_uint32_set(ys->ys_cv, date);
+	yang_cv_set(ys, cv);
+	cv_uint32_set(cv, date);
 	break;
     case Y_STATUS: /* RFC7950 7.21.2: "current", "deprecated", or "obsolete". */
 	if (strcmp(arg, "current") &&
@@ -1835,13 +1865,14 @@ ys_parse_sub(yang_stmt *ys,
 	break;
     case Y_MAX_ELEMENTS:
     case Y_MIN_ELEMENTS:
-	if ((ys->ys_cv = cv_new(CGV_UINT32)) == NULL){
+	if ((cv = cv_new(CGV_UINT32)) == NULL){
 	    clicon_err(OE_YANG, errno, "cv_new"); 
 	    goto done;
 	}
+	yang_cv_set(ys, cv);
 	if (keyword == Y_MAX_ELEMENTS &&
 	    strcmp(arg, "unbounded") == 0)
-	    cv_uint32_set(ys->ys_cv, 0); /* 0 means unbounded for max */
+	    cv_uint32_set(cv, 0); /* 0 means unbounded for max */
 	else{
 	    if ((ret = parse_uint32(arg, &minmax, &reason)) < 0){
 		clicon_err(OE_YANG, errno, "parse_uint32"); 
@@ -1853,7 +1884,7 @@ ys_parse_sub(yang_stmt *ys,
 		    free(reason);
 		goto done;
 	    }
-	    cv_uint32_set(ys->ys_cv, minmax);
+	    cv_uint32_set(cv, minmax);
 	}
 	break;
     case Y_MODIFIER:
@@ -1865,11 +1896,12 @@ ys_parse_sub(yang_stmt *ys,
     case Y_UNKNOWN:{ /* save (optional) argument in ys_cv */
 	if (extra == NULL)
 	    break;
-	if ((ys->ys_cv = cv_new(CGV_STRING)) == NULL){
+	if ((cv = cv_new(CGV_STRING)) == NULL){
 	    clicon_err(OE_YANG, errno, "cv_new"); 
 	    goto done;
 	}
-	if ((ret = cv_parse1(extra, ys->ys_cv, &reason)) < 0){ /* error */
+	yang_cv_set(ys, cv);
+	if ((ret = cv_parse1(extra, cv, &reason)) < 0){ /* error */
 	    clicon_err(OE_YANG, errno, "parsing cv");
 	    goto done;
 	}

test/all.sh

@@ -32,6 +32,7 @@ for test in $pattern; do
     testfile=$test
     . ./$test 
     errcode=$?
+    endsuite
     if [ $errcode -ne 0 ]; then
 	allerr=1
 	echo -e "\e[31mError in $test errcode=$errcode"

test/lib.sh

@@ -127,7 +127,12 @@ DEMSLEEP=.2
 let DEMLOOP=5*DEMWAIT
 
 # RESTCONF protocol, eg http or https
-: ${RCPROTO:=https}
+
+if [ "${WITH_RESTCONF}" = "fcgi" ]; then
+    : ${RCPROTO:=http}
+else
+    : ${RCPROTO:=https}
+fi
 
 # www user (on linux typically www-data, freebsd www)
 # Start restconf user, can be root which is dropped to wwwuser
@@ -221,6 +226,7 @@ fi
 # 1: auth-type (one of none, client-cert, user)
 # 2: pretty (if true pretty-print restconf return values)
 # Note, if AUTH=none then FEATURE clixon-restconf:allow-auth-none must be enabled
+# Note if https, check if server cert/key exists, if not generate them
 function restconf_config()
 {
     AUTH=$1
@@ -229,11 +235,21 @@ function restconf_config()
     if [ $RCPROTO = http ]; then
 	echo "<restconf><enable>true</enable><auth-type>$AUTH</auth-type><pretty>$PRETTY</pretty><debug>$DBG</debug><socket><namespace>default</namespace><address>0.0.0.0</address><port>80</port><ssl>false</ssl></socket></restconf>"
     else
-	echo "<restconf><enable>true</enable><auth-type>$AUTH</auth-type><pretty>$PRETTY</pretty><server-cert-path>/etc/ssl/certs/clixon-server-crt.pem</server-cert-path><server-key-path>/etc/ssl/private/clixon-server-key.pem</server-key-path><server-ca-cert-path>/etc/ssl/certs/clixon-ca-crt.pem</server-ca-cert-path><debug>$DBG</debug><socket><namespace>default</namespace><address>0.0.0.0</address><port>443</port><ssl>true</ssl></socket></restconf>"
+	certdir=$dir/certs
+	if [ ! -f ${dir}/clixon-server-crt.pem ]; then
+	    certdir=$dir/certs
+	    test -d $certdir || mkdir $certdir
+	    srvcert=${certdir}/clixon-server-crt.pem
+	    srvkey=${certdir}/clixon-server-key.pem
+	    cacert=${certdir}/clixon-ca-crt.pem
+	    cakey=${certdir}/clixon-ca-key.pem
+	    cacerts $cakey $cacert
+	    servercerts $cakey $cacert $srvkey $srvcert
+	fi
+	echo "<restconf><enable>true</enable><auth-type>$AUTH</auth-type><pretty>$PRETTY</pretty><server-cert-path>${certdir}/clixon-server-crt.pem</server-cert-path><server-key-path>${certdir}/clixon-server-key.pem</server-key-path><server-ca-cert-path>${certdir}/clixon-ca-crt.pem</server-ca-cert-path><debug>$DBG</debug><socket><namespace>default</namespace><address>0.0.0.0</address><port>443</port><ssl>true</ssl></socket></restconf>"
     fi
 }
 
-
 # Some tests may set owner of testdir to something strange and quit, need
 # to reset to me
 if [ ! -G $dir ]; then 
@@ -366,9 +382,10 @@ function stop_restconf_pre(){
 }
 
 # Stop restconf daemon after test
-# Two caveats in pkill:
+# Some problems with pkill:
 # 1) Dont use $clixon_restconf (dont work in valgrind)
 # 2) Dont use -u $WWWUSER since clixon_restconf may drop privileges.
+# 3) After fork, it seems to take some time before name is right
 function stop_restconf(){
     sudo pkill -f clixon_restconf
     if [ $valgrindtest -eq 3 ]; then 
@@ -432,6 +449,7 @@ function wait_restconf_stopped(){
 }
 
 # End of test, final tests before normal exit of test
+# Note this is a single test started by new, not a total test suite
 function endtest()
 {
     if [ $valgrindtest -eq 1 ]; then 
@@ -448,6 +466,12 @@ function new(){
     >&2 echo "Test $testi($testnr) [$1]"
 }
 
+# End of complete test-suite, eg a test file
+function endsuite()
+{
+    unset CURLOPTS
+}
+
 # Evaluate and return
 # Example: expectpart $(fn arg) 0 "my return" -- "foo"
 # - evaluated expression

test/mem.sh

@@ -40,7 +40,7 @@ function memonce(){
 	    sudo chmod 660 $valgrindfile
 	    sudo chown www-data $valgrindfile
 	    : ${DEMWAIT:=15} # valgrind backend needs some time to get up 
-	    clixon_restconf="/usr/bin/valgrind --leak-check=full --show-leak-kinds=all --suppressions=./valgrind-clixon.supp --track-fds=yes --trace-children=no  --child-silent-after-fork=yes --log-file=$valgrindfile clixon_restconf"
+	    clixon_restconf="/usr/bin/valgrind --num-callers=50 --leak-check=full --show-leak-kinds=all --suppressions=./valgrind-clixon.supp --track-fds=yes --trace-children=no  --child-silent-after-fork=yes --log-file=$valgrindfile clixon_restconf"
 
 	    ;;
 	*)
@@ -58,6 +58,7 @@ function memonce(){
 	testfile=$test
 	. ./$test 
 	errcode=$?
+	endsuite
 	if [ $errcode -ne 0 ]; then
 	    memerr=1
 	    echo -e "\e[31mError in $test errcode=$errcode"

test/test_augment.sh

@@ -52,8 +52,8 @@ cat <<EOF > $fyang2
 module ietf-interfaces {
   yang-version 1.1;
   namespace "urn:ietf:params:xml:ns:yang:ietf-interfaces";
-  revision "2019-03-04";
   prefix if;
+  revision "2019-03-04";
   identity interface-type {
     description
       "Base identity from which specific interface types are
@@ -110,10 +110,10 @@ module example-augment {
        yang-version 1.1;
        namespace "urn:example:augment";
        prefix mymod;
-       revision "2019-03-04";
        import ietf-interfaces {
          prefix if;
        }
+       revision "2019-03-04";
        identity some-new-iftype {
           base if:interface-type;
        }

test/test_augment_state.sh

@@ -35,8 +35,8 @@ cat <<EOF > $fyang2
 module example-lib {
   yang-version 1.1;
   namespace "urn:example:lib";
-  revision "2019-03-04";
   prefix lib;
+  revision "2019-03-04";
   container global-state {
     config false;
     leaf gbds{
@@ -81,10 +81,10 @@ module example-augment {
   yang-version 1.1;
   namespace "urn:example:augment";
   prefix aug;
-  revision "2020-09-25";
   import example-lib {
      prefix lib;
   }
+  revision "2020-09-25";
   /* Augments global state */
   augment "/lib:global-state" {
     leaf gads{

test/test_augment_trans.sh

@@ -38,8 +38,8 @@ cat <<EOF > $fyang
 module example-lib {
   yang-version 1.1;
   namespace "urn:example:lib";
-  revision "2019-03-04";
   prefix lib;
+  revision "2019-03-04";
   container base-config {
   }
   /* No prefix */
@@ -61,10 +61,10 @@ module example-augment1 {
   yang-version 1.1;
   namespace "urn:example:augment1";
   prefix aug1;
-  revision "2020-09-25";
   import example-lib {
      prefix lib;
   }
+  revision "2020-09-25";
   /* Augments config */
   augment "/lib:base-config/lib:parameter" {
     container aug1{
@@ -80,13 +80,13 @@ module example-augment2 {
   yang-version 1.1;
   namespace "urn:example:augment2";
   prefix aug2;
-  revision "2020-09-25";
   import example-lib {
      prefix lib;
   }
   import example-augment1 {
      prefix aug1;
   }
+  revision "2020-09-25";
   /* Augments config */
   augment "/lib:base-config/lib:parameter/aug1:aug1" {
 /*    when 'lib:name="foobar" and aug:aug1="foobar"'; */

test/test_c++.sh

@@ -154,7 +154,7 @@ new "wait backend"
 wait_backend
 
 new "Netconf runtime test"
-expecteof "$clixon_netconf -qf $cfg" 0 "$DEFAULTHELLO<rpc xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\"><example xmlns=\"urn:example:clixon\"><x>0</x></example></rpc>]]>]]>" '^<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><x xmlns="urn:example:clixon">0</x><y xmlns="urn:example:clixon">42</y></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -qf $cfg" 0 "$DEFAULTHELLO<rpc $DEFAULTNS><example xmlns=\"urn:example:clixon\"><x>0</x></example></rpc>]]>]]>" "^<rpc-reply $DEFAULTNS><x xmlns=\"urn:example:clixon\">0</x><y xmlns=\"urn:example:clixon\">42</y></rpc-reply>]]>]]>$"
 
 if [ $BE -ne 0 ]; then
    new "Kill backend"

test/test_client.sh

@@ -91,7 +91,7 @@ main(int    argc,
     /* Provide a clixon config-file, get a clixon handle */
     if ((h = clixon_client_init("$cfg")) == NULL)
        return -1;
-    /* Make a conenction over netconf or ssh/netconf */
+    /* Make a connection over netconf or ssh/netconf */
     if ((ch = clixon_client_connect(h, CLIXON_CLIENT_NETCONF)) == NULL)
        return -1;
 

test/test_datastore_repair.sh

@@ -37,8 +37,8 @@ EOF
 cat <<EOF > $fyangA
 module A{
   prefix a;
-  revision 2020-02-11;
   namespace "urn:example:a";
+  revision 2020-02-11;
   container x {
     container y {
     }
@@ -56,11 +56,11 @@ EOF
 cat <<EOF > $fyangB
 module B{
   prefix b;
-  revision 2020-02-11;
   namespace "urn:example:b";
   import A {
      prefix "a";
   }
+  revision 2020-02-11;
   augment "/a:x/a:y" {
     container z {
       leaf w {

test/test_debug.sh

@@ -105,6 +105,9 @@ expectpart "$($clixon_cli -1 -f $cfg -l o debug restconf 1)" 0 "^$"
 new "get and put config using restconf"
 expectpart "$(curl $CURLOPTS -H "Accept: application/yang-data+xml" -X GET $RCPROTO://localhost/restconf/data?content=config --next $CURLOPTS -H "Content-Type: application/yang-data+json" -X POST $RCPROTO://localhost/restconf/data -d '{"example:table":{"parameter":{"name":"local0","value":"foo"}}}')" 0 "HTTP/$HVER 200" '<data>' "HTTP/$HVER 201"
 
+# In freebsd, backend dies in stop_restconf below unless sleep
+sleep $DEMSLEEP 
+
 if [ $RC -ne 0 ]; then
     new "Kill restconf daemon"
     stop_restconf 
@@ -115,7 +118,7 @@ if [ $BE -ne 0 ]; then
     # Check if premature kill
     pid=$(pgrep -u root -f clixon_backend)
     if [ -z "$pid" ]; then
-	err "backend already dead"
+	err1 "backend pid !=0" 0
     fi
     # kill backend
     stop_backend -f $cfg

test/test_nacm_ext.sh

@@ -46,10 +46,10 @@ cat <<EOF > $fyang
 module nacm-example{
   yang-version 1.1;
   namespace "urn:example:nacm";
+  prefix nacm;
   import clixon-example {
 	prefix ex;
   }
-  prefix nacm;
   container authentication {
     presence "To keep this from auto-expanding"; 
     description "Example code for enabling www basic auth and some example 

test/test_nacm_module_read.sh

@@ -232,13 +232,13 @@ new "limit rpc ok"
 expectpart "$(curl -u wilma:bar $CURLOPTS -X POST $RCPROTO://localhost/restconf/operations/clixon-example:example -H "Content-Type: application/yang-data+json" -d '{"clixon-example:input":{"x":42}}' )" 0 "HTTP/$HVER 200" '{"clixon-example:output":{"x":"42","y":"42"}}'
 
 new "limit rpc netconf ok"
-expecteof "$clixon_netconf -U wilma -qf $cfg" 0 "$DEFAULTHELLO<rpc xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\"><example xmlns=\"urn:example:clixon\"><x>0</x></example></rpc>]]>]]>" '^<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><x xmlns="urn:example:clixon">0</x><y xmlns="urn:example:clixon">42</y></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -U wilma -qf $cfg" 0 "$DEFAULTHELLO<rpc $DEFAULTNS><example xmlns=\"urn:example:clixon\"><x>0</x></example></rpc>]]>]]>" "^<rpc-reply $DEFAULTNS><x xmlns=\"urn:example:clixon\">0</x><y xmlns=\"urn:example:clixon\">42</y></rpc-reply>]]>]]>$"
 
 new "guest rpc fail"
 expectpart "$(curl -u guest:bar $CURLOPTS -X POST $RCPROTO://localhost/restconf/operations/clixon-example:example -H "Content-Type: application/yang-data+json" -d '{"clixon-example:input":{"x":42}}' )" 0 "HTTP/$HVER 403" '{"ietf-restconf:errors":{"error":{"error-type":"application","error-tag":"access-denied","error-severity":"error","error-message":"access denied"}}}'
 
 new "guest rpc netconf fail"
-expecteof "$clixon_netconf -U guest -qf $cfg" 0 "$DEFAULTHELLO<rpc xmlns=\"urn:ietf:params:xml:ns:netconf:base:1.0\"><example xmlns=\"urn:example:clixon\"><x>0</x></example></rpc>]]>]]>" '^<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><rpc-error><error-type>application</error-type><error-tag>access-denied</error-tag><error-severity>error</error-severity><error-message>access denied</error-message></rpc-error></rpc-reply>]]>]]>$'
+expecteof "$clixon_netconf -U guest -qf $cfg" 0 "$DEFAULTHELLO<rpc $DEFAULTNS><example xmlns=\"urn:example:clixon\"><x>0</x></example></rpc>]]>]]>" "^<rpc-reply $DEFAULTNS><rpc-error><error-type>application</error-type><error-tag>access-denied</error-tag><error-severity>error</error-severity><error-message>access denied</error-message></rpc-error></rpc-reply>]]>]]>$"
 
 #------------------ Set read-default permit
 

test/test_netconf.sh

@@ -26,6 +26,7 @@ cat <<EOF > $cfg
   <CLICON_BACKEND_DIR>/usr/local/lib/$APPNAME/backend</CLICON_BACKEND_DIR>
   <CLICON_BACKEND_REGEXP>example_backend.so$</CLICON_BACKEND_REGEXP>
   <CLICON_NETCONF_DIR>/usr/local/lib/$APPNAME/netconf</CLICON_NETCONF_DIR>
+  <CLICON_NETCONF_MESSAGE_ID_OPTIONAL>false</CLICON_NETCONF_MESSAGE_ID_OPTIONAL>
   <CLICON_RESTCONF_DIR>/usr/local/lib/$APPNAME/restconf</CLICON_RESTCONF_DIR>
   <CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
   <CLICON_CLI_MODE>$APPNAME</CLICON_CLI_MODE>
@@ -67,6 +68,9 @@ expecteof "$clixon_netconf -qf $cfg" 0 "<hello $DEFAULTNS><capabilities><capabil
 new "Frame with unknown message"
 expecteof "$clixon_netconf -qf $cfg" 0 "<xxx $DEFAULTNS></xxx>]]>]]>" "^<rpc-reply $DEFAULTNS><rpc-error><error-type>protocol</error-type><error-tag>unknown-element</error-tag><error-info><bad-element>xxx</bad-element></error-info><error-severity>error</error-severity><error-message>Unrecognized netconf operation</error-message></rpc-error></rpc-reply>]]>]]>$"
 
+new "Frame without message-id attribute"
+expecteof "$clixon_netconf -qf $cfg" 0 "$DEFAULTHELLO<rpc $DEFAULTONLY><get-config><source><candidate/></source></get-config></rpc>]]>]]>" "^<rpc-reply $DEFAULTONLY><rpc-error><error-type>rpc</error-type><error-tag>missing-attribute</error-tag><error-info><bad-attribute>message-id</bad-attribute></error-info><error-severity>error</error-severity><error-message>Incoming rpc</error-message></rpc-error></rpc-reply>]]>]]>$"
+
 new "netconf rcv hello, disable RFC7895/ietf-yang-library"
 expecteof "$clixon_netconf -f $cfg -o CLICON_MODULE_LIBRARY_RFC7895=0" 0 "$DEFAULTHELLO<rpc $DEFAULTNS><get-config><source><candidate/></source></get-config></rpc>]]>]]>" "^<hello $DEFAULTNS><capabilities><capability>urn:ietf:params:netconf:base:1.1</capability><capability>urn:ietf:params:netconf:base:1.0</capability><capability>urn:ietf:params:netconf:capability:candidate:1.0</capability><capability>urn:ietf:params:netconf:capability:validate:1.1</capability><capability>urn:ietf:params:netconf:capability:startup:1.0</capability><capability>urn:ietf:params:netconf:capability:xpath:1.0</capability><capability>urn:ietf:params:netconf:capability:notification:1.0</capability></capabilities><session-id>[0-9]*</session-id></hello>]]>]]><rpc-reply $DEFAULTNS><data/></rpc-reply>]]>]]>$"
 

test/test_openconfig.sh

@@ -97,7 +97,7 @@ for f in $files; do
     fi
 done
 
-new "Openconfig test: $clixon_cli -1f $cfg -y $f show version ($m modules)"
+new "Openconfig test: $clixon_cli -1f $cfg show version ($m modules)"
 for f in $files; do
     if [ -n "$(head -1 $f|grep '^module')" ]; then
 	new "$clixon_cli -D $DBG  -1f $cfg -y $f show version"

test/test_openconfig_interfaces.sh

@@ -1,5 +1,6 @@
 #!/usr/bin/env bash
 # Run a system around openconfig interface, ie: openconfig-if-ethernet
+# Note first variant uses ietf-interfaces, maybe remove this?
 
 # Magic line must be first in script (see README.md)
 s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
@@ -22,13 +23,11 @@ cat <<EOF > $cfg
 <clixon-config xmlns="http://clicon.org/config">
   <CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
   <CLICON_FEATURE>ietf-netconf:startup</CLICON_FEATURE>
-  <CLICON_YANG_DIR>$OPENCONFIG/third_party/ietf/</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$OCDIR</CLICON_YANG_DIR>
-  <CLICON_YANG_DIR>$OCDIR</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$OCDIR/interfaces</CLICON_YANG_DIR>
   <CLICON_YANG_DIR>$OCDIR/types</CLICON_YANG_DIR>
-  <CLICON_YANG_DIR>$OCDIR/wifi</CLICON_YANG_DIR>
+  <CLICON_YANG_DIR>$OCDIR/vlan</CLICON_YANG_DIR>
   <CLICON_YANG_MAIN_FILE>$fyang</CLICON_YANG_MAIN_FILE>	
   <CLICON_CLISPEC_DIR>/usr/local/lib/$APPNAME/clispec</CLICON_CLISPEC_DIR>
   <CLICON_CLI_DIR>/usr/local/lib/$APPNAME/cli</CLICON_CLI_DIR>
@@ -42,6 +41,7 @@ cat <<EOF > $cfg
 </clixon-config>
 EOF
 
+# First using ietf-interfaces (not openconfig-interfaces)
 # Example yang
 cat <<EOF > $fyang
 module clixon-example{
@@ -99,7 +99,7 @@ fi
 new "wait backend"
 wait_backend
 
-new "$clixon_cli -D $DBG -1f $cfg -y $f show version"
+new "$clixon_cli -D $DBG -1f $cfg show version"
 expectpart "$($clixon_cli -D $DBG -1f $cfg show version)" 0 "${CLIXON_VERSION}"
 
 new "$clixon_netconf -qf $cfg"
@@ -126,6 +126,76 @@ if [ $BE -ne 0 ]; then
     stop_backend -f $cfg
 fi
 
+# Second using openconfig-interfaces instead
+# Example yang
+cat <<EOF > $fyang
+module clixon-example{
+  yang-version 1.1;
+  namespace "urn:example:example";
+  prefix ex;
+
+  import openconfig-vlan {
+    prefix oc-vlan;
+  }
+  import openconfig-if-ethernet {
+    prefix oc-eth;
+  }
+}
+EOF
+
+# Example system
+cat <<EOF > $dir/startup_db
+<config>
+   <interfaces xmlns="http://openconfig.net/yang/interfaces">
+      <interface>
+         <name>eth1</name>
+         <config>
+            <name>eth1</name>
+            <type>ianaift:ethernetCsmacd</type>
+            <mtu>9206</mtu>
+            <enabled>true</enabled>
+            <oc-vlan:tpid xmlns:oc-vlan="http://openconfig.net/yang/vlan">oc-vlan-types:TPID_0X8100</oc-vlan:tpid>
+         </config>
+         <oc-eth:ethernet xmlns:oc-eth="http://openconfig.net/yang/interfaces/ethernet">
+            <oc-eth:config>
+               <oc-eth:mac-address>2c:53:4a:09:59:73</oc-eth:mac-address>
+            </oc-eth:config>
+         </oc-eth:ethernet>
+      </interface>
+   </interfaces>
+</config>
+EOF
+
+if [ $BE -ne 0 ]; then
+    new "kill old backend"
+    sudo clixon_backend -zf $cfg
+    if [ $? -ne 0 ]; then
+	err
+    fi
+    sudo pkill -f clixon_backend # to be sure
+    
+    new "start backend -s startup -f $cfg"
+    start_backend -s startup -f $cfg
+fi
+
+new "wait backend"
+wait_backend
+
+new "$clixon_cli -D $DBG -1f $cfg show version"
+expectpart "$($clixon_cli -D $DBG -1f $cfg show version)" 0 "${CLIXON_VERSION}"
+
+if [ $BE -ne 0 ]; then
+    new "Kill backend"
+    # Check if premature kill
+    pid=$(pgrep -u root -f clixon_backend)
+    if [ -z "$pid" ]; then
+	err "backend already dead"
+    fi
+    # kill backend
+    stop_backend -f $cfg
+fi
+
+
 rm -rf $dir
 
 new "endtest"

test/test_perf_restconf.sh

@@ -32,6 +32,7 @@ ftest=$dir/test.xml
 fconfig=$dir/large.xml
 fconfig2=$dir/large2.xml # leaf-list
 foutput=$dir/output.xml
+foutput2=$dir/output2.xml
 
 cat <<EOF > $fyang
 module scaling{
@@ -132,15 +133,20 @@ expecteof "time -p $clixon_netconf -qf $cfg" 0 "$DEFAULTHELLO<rpc $DEFAULTNS><co
 
 new "Check running-db contents"
 curl $CURLOPTS -X GET -H "Accept: application/yang-data+xml" $RCPROTO://localhost/restconf/data?content=config > $foutput
+r=$?
+if [ $r -ne 0 ]; then
+    err1 "retval 0" $r
+fi
 
 # Remove Content-Length line (depends on size)
-sed -i '/Content-Length:/d' $foutput
+# Note: do not use sed -i since it is not portable between gnu and bsd
-sed -i '/content-length:/d' $foutput
+sed '/Content-Length:/d' $foutput > $foutput2 && mv $foutput2 $foutput
+sed '/content-length:/d' $foutput > $foutput2 && mv $foutput2 $foutput
 # Remove (nginx) web-server specific lines
-sed -i '/Server:/d' $foutput
+sed '/Server:/d' $foutput > $foutput2 && mv $foutput2 $foutput
-sed -i '/Date:/d' $foutput
+sed '/Date:/d' $foutput > $foutput2 && mv $foutput2 $foutput
-sed -i '/Transfer-Encoding:/d' $foutput
+sed '/Transfer-Encoding:/d' $foutput > $foutput2 && mv $foutput2 $foutput
-sed -i '/Connection:/d' $foutput
+sed '/Connection:/d' $foutput > $foutput2 && mv $foutput2 $foutput
 
 # Create a file to compare with
 if ${HAVE_LIBNGHTTP2}; then
@@ -165,7 +171,7 @@ echo "</data>
" >> $ftest
 
 ret=$(diff -i $ftest $foutput)
 if [ $? -ne 0 ]; then
-    echo "$ret"
+    echo "diff -i $ftest $foutput"
     err1 "Matching running-db with $fconfigonly"
 fi	
 

test/test_restconf.sh

@@ -172,37 +172,86 @@ function testrun()
     echo "curl $CURLOPTS -X GET $proto://$addr/.well-known/host-meta"
     expectpart "$(curl $CURLOPTS -X GET $proto://$addr/.well-known/host-meta)" 0 "HTTP/$HVER 200" "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
 
-    if ! ${HAVE_LIBNGHTTP2}; then # http/2
+    if [ ${HAVE_LIBNGHTTP2} = true -a ${HAVE_LIBEVHTP} = false ]; then
+	echo "native + http/2 only"
+	# Important here is robustness of restconf daemon, not a meaningful reply
+	if [ $proto = http ]; then # see (2) https to http port in restconf_main_native.c
+	    # http protocol mismatch can just close the socket if assumed its http/2
+	    # everything else would guess it is http/1 which is really wrong here
+	    # The tr statement replaces null char to get rid of annoying message:
+	    # ./test_restconf.sh: line 180: warning: command substitution: ignored null byte in input
+	    new "restconf GET http/1.0  - close"
+	    expectpart "$(curl $CURLOPTS --http1.0 -X GET $proto://$addr/.well-known/host-meta | tr '\0' '\n')" 0 "" --not-- 'HTTP'
+	else
+	    new "restconf GET https/1.0  - close"
+	    expectpart "$(curl $CURLOPTS --http1.0 -X GET $proto://$addr/.well-known/host-meta)" 52 "" --not-- 'HTTP'
 
-    if [ "${WITH_RESTCONF}" = "native" ]; then # XXX does not work with nginx
+	fi
-	new "restconf GET http/1.0  - returns 1.0"
-	expectpart "$(curl $CURLOPTS --http1.0 -X GET $proto://$addr/.well-known/host-meta)" 0 'HTTP/1.0 200 OK' "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
-    fi 
-    new "restconf GET http/1.1"
-    expectpart "$(curl $CURLOPTS --http1.1 -X GET $proto://$addr/.well-known/host-meta)" 0 'HTTP/1.1 200 OK' "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
 
-    # Try http/2 - go back to http/1.1
+	if [ $proto = http ]; then
-    new "restconf GET http/2"
+	    new "restconf GET http/1.1 - close"
-    expectpart "$(curl $CURLOPTS --http2 -X GET $proto://$addr/.well-known/host-meta)" 0 "HTTP/1.1 200 OK" "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
+	    expectpart "$(curl $CURLOPTS --http1.1 -X GET $proto://$addr/.well-known/host-meta | tr '\0' '\n')" 0 --not-- 'HTTP'
+	else
+	    new "restconf GET https/1.1 - close"
+	    expectpart "$(curl $CURLOPTS --http1.1 -X GET $proto://$addr/.well-known/host-meta)" 52 --not-- 'HTTP'
+	fi
+	
+	if [ $proto = http ]; then
+	    new "restconf GET http/2 switch protocol"
+	    expectpart "$(curl $CURLOPTS --http2 -X GET $proto://$addr/.well-known/host-meta | tr '\0' '\n')" 0 --not-- 'HTTP'
+	else
+	    new "restconf GET https/2 alpn protocol"
+	    expectpart "$(curl $CURLOPTS --http2 -X GET $proto://$addr/.well-known/host-meta)" 0 "HTTP/$HVER 200" "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
+	fi
 
-    if [ $proto = http ]; then # see (2) https to http port in restconf_main_native.c
+	# Wrong protocol http when https or vice versa
-	new "restconf GET http/2 prior-knowledge (http)"
+	if [ $proto = http ]; then # see (2) https to http port in restconf_main_native.c
-	expectpart "$(curl $CURLOPTS --http2-prior-knowledge -X GET $proto://$addr/.well-known/host-meta 2>&1)" "16 52 55" # "Error in the HTTP2 framing layer" "Connection reset by peer"
+	    new "Wrong proto=https on http port, expect err 35 wrong version number"
+	    expectpart "$(curl $CURLOPTS -X GET https://$addr:80/.well-known/host-meta 2>&1)" 35 #"wrong version number" # dependent on curl version
+	else # see (1) http to https port in restconf_main_native.c
+	    new "Wrong proto=http on https port, expect bad request"
+	    expectpart "$(curl $CURLOPTS -X GET http://$addr:443/.well-known/host-meta)" "16 52 55" --not-- 'HTTP'
+	fi
     else
-    	new "restconf GET http/2 prior-knowledge(http2)"
+	echo "fcgi or native+http/1 or native+http/1+http/2"
-	expectpart "$(curl $CURLOPTS --http2-prior-knowledge -X GET $proto://$addr/.well-known/host-meta)" 0 'HTTP/1.1 200 OK' "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
+	if [ "${WITH_RESTCONF}" = "native" ]; then # XXX does not work with nginx
-    fi
+	    new "restconf GET http/1.0  - returns 1.0"
+	    expectpart "$(curl $CURLOPTS --http1.0 -X GET $proto://$addr/.well-known/host-meta)" 0 'HTTP/1.0 200 OK' "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
+	fi 
+	new "restconf GET http/1.1"
+	expectpart "$(curl $CURLOPTS --http1.1 -X GET $proto://$addr/.well-known/host-meta)" 0 'HTTP/1.1 200 OK' "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
 
-    # Negative test GET datastore
+	if ${HAVE_LIBNGHTTP2}; then
-    if [ $proto = http ]; then # see (2) https to http port in restconf_main_native.c
+	    # http/1 + http/2
-	new "Wrong proto=https on http port, expect err 35 wrong version number"
+
-	expectpart "$(curl $CURLOPTS -X GET https://$addr:80/.well-known/host-meta 2>&1)" 35 #"wrong version number" # dependent on curl version
+	    new "restconf GET http/2 switch protocol"
-    else # see (1) http to https port in restconf_main_native.c
+	    expectpart "$(curl $CURLOPTS --http2 -X GET $proto://$addr/.well-known/host-meta)" 0 "" "HTTP/2 200" "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"	    # Only if http:  HTTP/1.1 101 Switching Protocols
-	new "Wrong proto=http on https port, expect bad request"
+	else
-	expectpart "$(curl $CURLOPTS -X GET http://$addr:443/.well-known/host-meta)" 0 "HTTP/$HVER 400"
+	    # http/1 only Try http/2 - go back to http/1.1
-	# expectpart "$(curl $CURLOPTS -X GET http://$addr:443/.well-known/host-meta 2>&1)" 56 "Connection reset by peer"
+	    new "restconf GET http/2 switch protocol"
-    fi
+	    expectpart "$(curl $CURLOPTS --http2 -X GET $proto://$addr/.well-known/host-meta)" 0 "HTTP/1.1 200 OK" "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"	    
+	fi
+	
+	# http2-prior knowledge
+	if [ $proto = http ]; then # see (2) https to http port in restconf_main_native.c
+	    new "restconf GET http/2 prior-knowledge (http)"
+	    expectpart "$(curl $CURLOPTS --http2-prior-knowledge -X GET $proto://$addr/.well-known/host-meta 2>&1)" "16 52 55" # "Error in the HTTP2 framing layer" "Connection reset by peer"
+	else
+    	    new "restconf GET https/2 prior-knowledge"
+	    expectpart "$(curl $CURLOPTS --http2-prior-knowledge -X GET $proto://$addr/.well-known/host-meta)" 0 "HTTP/$HVER 200" "<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>" "<Link rel='restconf' href='/restconf'/>" "</XRD>"
+	fi
+	
+	# Wrong protocol http when https or vice versa
+	if [ $proto = http ]; then # see (2) https to http port in restconf_main_native.c
+	    new "Wrong proto=https on http port, expect err 35 wrong version number"
+	    expectpart "$(curl $CURLOPTS -X GET https://$addr:80/.well-known/host-meta 2>&1)" 35 #"wrong version number" # dependent on curl version
+	else # see (1) http to https port in restconf_main_native.c
+	    new "Wrong proto=http on https port, expect bad request"
+	    expectpart "$(curl $CURLOPTS -X GET http://$addr:443/.well-known/host-meta)" 0 "HTTP/" "400"
+	    # expectpart "$(curl $CURLOPTS -X GET http://$addr:443/.well-known/host-meta 2>&1)" 56 "Connection reset by peer"
+	fi
     fi # HTTP/2    
+
     # Exact match
     new "restconf get restconf resource. RFC 8040 3.3 (json)"
     expectpart "$(curl $CURLOPTS -X GET -H "Accept: application/yang-data+json" $proto://$addr/restconf)" 0 "HTTP/$HVER 200" '{"ietf-restconf:restconf":{"data":{},"operations":{},"yang-library-version":"2019-01-04"}}'
@@ -341,7 +390,7 @@ function testrun()
     new "restconf Check eth/0/0 GET augmented state level 2"
     expectpart "$(curl $CURLOPTS -X GET -H 'Accept: application/yang-data+json' $proto://$addr/restconf/data/ietf-interfaces:interfaces/interface=eth%2f0%2f0/clixon-example:my-status)" 0 "HTTP/$HVER 200" '{"clixon-example:my-status":{"int":42,"str":"foo"}}' 
 
-    new "restconf Check eth/0/0 added state XXXXXXX"
+    new "restconf Check eth/0/0 added state"
     expectpart "$(curl $CURLOPTS -X GET -H 'Accept: application/yang-data+json' $proto://$addr/restconf/data/clixon-example:state)" 0 "HTTP/$HVER 200" '{"clixon-example:state":{"op":\["41","42","43"\]}}'
 
     new "restconf Re-post eth/0/0 which should generate error"
@@ -393,7 +442,7 @@ function testrun()
 
     new "restconf rpc using POST xml"
     ret=$(curl $CURLOPTS -X POST -H "Content-Type: application/yang-data+json" -H "Accept: application/yang-data+xml" -d '{"clixon-example:input":{"x":42}}' $proto://$addr/restconf/operations/clixon-example:example)
-    expect='<output xmlns="urn:example:clixon"><x>42</x><y>42</y></output>'
+    expect='<output message-id="42" xmlns="urn:example:clixon"><x>42</x><y>42</y></output>'
     match=`echo $ret | grep --null -Eo "$expect"`
     if [ -z "$match" ]; then
 	err "$expect" "$ret"

test/test_restconf_internal.sh

@@ -275,10 +275,8 @@ if [ $pid0 -eq $pid3 ]; then
     err1 "A different pid" "same pid: $pid3"
 fi
 
-new "kill restconf using kill"
+new "kill restconf"
-stop_restconf_pre
+sudo kill $pid3
-
-sleep $DEMSLEEP
 
 new "Wait for restconf to stop"
 wait_restconf_stopped
@@ -378,6 +376,7 @@ cat<<EOF > $startupdb
 EOF
 
 new "kill old restconf"
+sleep $DEMSLEEP
 stop_restconf_pre
 
 new "test params: -f $cfg"
@@ -485,10 +484,9 @@ fi
 #Start backend -s none should start 
 
 new "kill restconf"
+sleep $DEMSLEEP
 stop_restconf
 
-sleep $DEMSLEEP # Lots of processes need to die before next test
-
 new "endtest"
 endtest
 

test/test_restconf_nmap.sh

@@ -116,12 +116,7 @@ fi
 new "wait backend"
 wait_backend
 
-#new "netconf edit config"
+# Explicit start of restconf for easier debugging
-#expecteof "$clixon_netconf -qf $cfg" 0 "$DEFAULTHELLO<rpc $DEFAULTNS><edit-config><target><candidate/></target><config>$RESTCONFIG</config></edit-config></rpc>]]>]]>" "^<rpc-reply $DEFAULTNS><ok/></rpc-reply>]]>]]>$"
-
-#new "netconf commit"
-#expecteof "$clixon_netconf -qf $cfg" 0 "$DEFAULTHELLO<rpc $DEFAULTNS><commit/></rpc>]]>]]>" "^<rpc-reply $DEFAULTNS><ok/></rpc-reply>]]>]]>$"
-
 if [ $RC -ne 0 ]; then
     new "kill old restconf daemon"
     stop_restconf_pre

test/test_restconf_op.sh

@@ -204,7 +204,6 @@ new "restconf DELETE whole datastore"
 expectpart "$(curl $CURLOPTS -X DELETE $RCPROTO://localhost/restconf/data)" 0 "HTTP/$HVER 204"
 
 #--------------- Multiple request in single TCP tests
-
 expectpart "$(curl $CURLOPTS -H "Accept: application/yang-data+xml" -X GET $RCPROTO://localhost/restconf/data?content=config --next $CURLOPTS -H "Content-Type: application/yang-data+json" -X POST $RCPROTO://localhost/restconf/data -d '{"example:cont1":{"interface":{"name":"local0","type":"regular"}}}')" 0 "HTTP/$HVER 200" '<data/>' "HTTP/$HVER 201"
 
 #--------------- json type tests

test/test_restconf_patch.sh

@@ -91,6 +91,9 @@ cat <<EOF > $dir/example-system.yang
         leaf enable-jukebox-streaming {
           type boolean;
         }
+        leaf extraleaf {
+          type string;
+        }
       }
    }
 EOF
@@ -250,6 +253,16 @@ expectpart "$(curl -u andy:bar $CURLOPTS -X PATCH  -H "Content-Type: application
 new "GET check" # XXX: "data" should probably be namespaced?
 expectpart "$(curl -u andy:bar $CURLOPTS -X GET $RCPROTO://localhost/restconf/data?content=config -H 'Accept: application/yang-data+xml')" 0 "HTTP/$HVER 200" '<extra xmlns="http://example.com/ns/example-jukebox">c</extra>' '<data>'
 
+new "Add empty leaf"
+expectpart "$(curl -u andy:bar $CURLOPTS -X POST $RCPROTO://localhost/restconf/data -H 'Content-Type: application/yang-data+json' -d '{"example-system:system":{"extraleaf":""}}')" 0 "HTTP/$HVER 201"
+
+new "Add entry with PATCH"
+expectpart "$(curl -u andy:bar $CURLOPTS -X PATCH $RCPROTO://localhost/restconf/data/example-system:system -H 'Content-Type: application/yang-data+json' -d '{"example-system:system":{"extraleaf":"something"}}')" 0 "HTTP/$HVER 204"
+
+new "GET check"
+expectpart "$(curl -u andy:bar $CURLOPTS -X GET $RCPROTO://localhost/restconf/data/example-system:system -H 'Accept: application/yang-data+xml')" 0 "HTTP/$HVER 200" '<system xmlns="http://example.com/ns/example-system"><extraleaf>something</extraleaf></system>'
+
+
 if [ $RC -ne 0 ]; then
     new "Kill restconf daemon"
     stop_restconf

test/test_restconf_ssl_certs.sh

@@ -288,10 +288,10 @@ EOF
 
 
     new "limited invalid cert"
-    expectpart "$(curl $CURLOPTS --key $certdir/limited.key --cert $certdir/limited.crt -X GET $RCPROTO://localhost/restconf/data/example:x 2>&1)" "35 55 56" # 55 "certificate expired"
+    expectpart "$(curl $CURLOPTS --key $certdir/limited.key --cert $certdir/limited.crt -X GET $RCPROTO://localhost/restconf/data/example:x 2>&1)" "16 35 55 56" # 55 "certificate expired"
 
     new "too weak cert (sign w md5)"
-    expectpart "$(curl $CURLOPTS --key $certdir/mymd5.key --cert $certdir/mymd5.crt -X GET $RCPROTO://localhost/restconf/data/example:x 2>&1)" 58 "md too weak"
+    expectpart "$(curl $CURLOPTS --key $certdir/mymd5.key --cert $certdir/mymd5.crt -X GET $RCPROTO://localhost/restconf/data/example:x 2>&1)" "35 58" # "md too weak"
 
 # Havent been able to generate "wrong CA"
 #    new "invalid cert from wrong CA"

test/test_restconf_yang_patch.sh

@@ -0,0 +1,205 @@
+#!/usr/bin/env bash
+# Restconf RFC8072 yang patch 
+# XXX enable YANG_PACTH in include/clixon_custom.h to run this test
+# Use nacm module in example/main/example_restconf.c hardcoded to
+# andy:bar and wilma:bar
+
+# Magic line must be first in script (see README.md)
+s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi
+
+echo "...skipped: YANG_PATCH NYI"
+if [ "$s" = $0 ]; then exit 0; else return 0; fi
+    
+APPNAME=example
+    
+cfg=$dir/conf.xml
+startupdb=$dir/startup_db
+fjukebox=$dir/example-jukebox.yang
+    
+# Define default restconfig config: RESTCONFIG
+RESTCONFIG=$(restconf_config user false)
+
+cat <<EOF > $cfg
+<clixon-config xmlns="http://clicon.org/config">
+  <CLICON_CONFIGFILE>$cfg</CLICON_CONFIGFILE>
+  <CLICON_YANG_DIR>/usr/local/share/clixon</CLICON_YANG_DIR>
+  <CLICON_YANG_DIR>$IETFRFC</CLICON_YANG_DIR>
+  <CLICON_YANG_MAIN_DIR>$dir</CLICON_YANG_MAIN_DIR>
+  <CLICON_SOCK>/usr/local/var/$APPNAME/$APPNAME.sock</CLICON_SOCK>
+  <CLICON_FEATURE>ietf-netconf:startup</CLICON_FEATURE>
+  <CLICON_RESTCONF_DIR>/usr/local/lib/$APPNAME/restconf</CLICON_RESTCONF_DIR>
+  <CLICON_BACKEND_PIDFILE>$dir/restconf.pidfile</CLICON_BACKEND_PIDFILE>
+  <CLICON_XMLDB_DIR>$dir</CLICON_XMLDB_DIR>
+  <CLICON_NACM_MODE>internal</CLICON_NACM_MODE>
+  <CLICON_NACM_DISABLED_ON_EMPTY>true</CLICON_NACM_DISABLED_ON_EMPTY>
+  $RESTCONFIG
+</clixon-config>
+EOF
+
+NACM0="<nacm xmlns=\"urn:ietf:params:xml:ns:yang:ietf-netconf-acm\">
+     <enable-nacm>true</enable-nacm>
+     <read-default>deny</read-default>
+     <write-default>deny</write-default>
+     <exec-default>permit</exec-default>
+   <groups>
+       <group>
+         <name>admin</name>
+         <user-name>andy</user-name>
+       </group>
+       <group>
+         <name>limited</name>
+         <user-name>wilma</user-name>
+       </group>
+   </groups>
+   <rule-list>
+       <name>admin</name>
+       <group>admin</group>
+       <rule>
+         <name>permit-all</name>
+         <module-name>*</module-name>
+         <access-operations>*</access-operations>
+         <action>permit</action>
+         <comment>
+             Allow the 'admin' group complete access to all operations and data.
+         </comment>
+       </rule>
+   </rule-list>
+   <rule-list>
+       <name>limited</name>
+       <group>limited</group>
+       <rule>
+         <name>limit-jukebox</name>
+         <module-name>jukebox-example</module-name>
+         <access-operations>read create delete</access-operations>
+         <action>deny</action>
+       </rule>
+   </rule-list>
+ </nacm>
+"
+
+cat<<EOF > $startupdb
+<${DATASTORE_TOP}>
+   $NACM0
+</${DATASTORE_TOP}>
+EOF
+
+# An extra testmodule that includes nacm
+cat <<EOF > $dir/example-system.yang
+   module example-system {
+      namespace "http://example.com/ns/example-system";
+      prefix "ex";
+      import ietf-netconf-acm {
+	prefix nacm;
+      }
+      container system {
+        leaf enable-jukebox-streaming {
+          type boolean;
+        }
+        leaf extraleaf {
+          type string;
+        }
+      }
+   }
+EOF
+
+# Common Jukebox spec (fjukebox must be set)
+. ./jukebox.sh
+
+new "test params: -s startup -f $cfg"
+if [ $BE -ne 0 ]; then
+    new "kill old backend"
+    sudo clixon_backend -zf $cfg
+    if [ $? -ne 0 ]; then
+	err
+    fi
+    sudo pkill -f clixon_backend # to be sure
+
+    new "start backend -s startup -f $cfg"
+    start_backend -s startup -f $cfg
+fi
+
+new "wait backend"
+wait_backend
+
+if [ $RC -ne 0 ]; then
+    new "kill old restconf daemon"
+    stop_restconf_pre
+
+    new "start restconf daemon"
+    start_restconf -f $cfg
+fi
+
+new "wait restconf"
+wait_restconf
+
+# RFC 8072 A.1.1 
+REQ='<yang-patch xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-patch">
+        <patch-id>add-songs-patch</patch-id>
+        <edit>
+          <edit-id>edit1</edit-id>
+          <operation>create</operation>
+          <target>/song=Bridge%20Burning</target>
+          <value>
+            <song xmlns="http://example.com/ns/example-jukebox">
+              <name>Bridge Burning</name>
+              <location>/media/bridge_burning.mp3</location>
+              <format>MP3</format>
+              <length>288</length>
+            </song>
+          </value>
+        </edit>
+        <edit>
+          <edit-id>edit2</edit-id>
+          <operation>create</operation>
+          <target>/song=Rope</target>
+          <value>
+            <song xmlns="http://example.com/ns/example-jukebox">
+              <name>Rope</name>
+              <location>/media/rope.mp3</location>
+              <format>MP3</format>
+              <length>259</length>
+            </song>
+          </value>
+        </edit>
+        <edit>
+          <edit-id>edit3</edit-id>
+          <operation>create</operation>
+          <target>/song=Dear%20Rosemary</target>
+          <value>
+            <song xmlns="http://example.com/ns/example-jukebox">
+              <name>Dear Rosemary</name>
+              <location>/media/dear_rosemary.mp3</location>
+              <format>MP3</format>
+              <length>269</length>
+            </song>
+          </value>
+        </edit>
+      </yang-patch>'
+
+new "RFC 8072 A.1.1 Add resources: Error."
+expectpart "$(curl -u andy:bar $CURLOPTS -X PATCH -H 'Content-Type: application/yang-patch+xml' -H 'Accept: application/yang-data+xml' $RCPROTO://localhost/restconf/data/example-jukebox:jukebox/library/artist=Foo%20Fighters/album=Wasting%20Light -d "$REQ")" 0 "HTTP/$HVER 409"
+
+
+if [ $RC -ne 0 ]; then
+    new "Kill restconf daemon"
+    stop_restconf
+fi
+
+if [ $BE -ne 0 ]; then
+    new "Kill backend"
+    # Check if premature kill
+    pid=$(pgrep -u root -f clixon_backend)
+    if [ -z "$pid" ]; then
+	err "backend already dead"
+    fi
+    # kill backend
+    stop_backend -f $cfg
+fi
+
+# Set by restconf_config
+unset RESTCONFIG
+
+rm -rf $dir
+
+new "endtest"
+endtest

test/test_rpc.sh

@@ -89,10 +89,10 @@ new "restconf example rpc xml/json"
 expectpart "$(curl $CURLOPTS -X POST -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+json' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' $RCPROTO://localhost/restconf/operations/clixon-example:example)"  0 'Content-Type: application/yang-data+json' '{"clixon-example:output":{"x":"0","y":"42"}}'
 
 new "restconf example rpc json/xml"
-expectpart "$(curl $CURLOPTS -X POST -H 'Content-Type: application/yang-data+json' -H 'Accept: application/yang-data+xml' -d '{"clixon-example:input":{"x":"0"}}' $RCPROTO://localhost/restconf/operations/clixon-example:example)" 0 'Content-Type: application/yang-data+xml' '<output xmlns="urn:example:clixon"><x>0</x><y>42</y></output>'
+expectpart "$(curl $CURLOPTS -X POST -H 'Content-Type: application/yang-data+json' -H 'Accept: application/yang-data+xml' -d '{"clixon-example:input":{"x":"0"}}' $RCPROTO://localhost/restconf/operations/clixon-example:example)" 0 'Content-Type: application/yang-data+xml' '<output message-id="42" xmlns="urn:example:clixon"><x>0</x><y>42</y></output>'
 
 new "restconf example rpc xml/xml"
-expectpart "$(curl $CURLOPTS -X POST -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+xml' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' $RCPROTO://localhost/restconf/operations/clixon-example:example)" 0 'Content-Type: application/yang-data+xml' '<output xmlns="urn:example:clixon"><x>0</x><y>42</y></output>'
+expectpart "$(curl $CURLOPTS -X POST -H 'Content-Type: application/yang-data+xml' -H 'Accept: application/yang-data+xml' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' $RCPROTO://localhost/restconf/operations/clixon-example:example)" 0 'Content-Type: application/yang-data+xml' '<output message-id="42" xmlns="urn:example:clixon"><x>0</x><y>42</y></output>'
 
 new "restconf example rpc xml in w json encoding (expect fail)"
 expectpart "$(curl $CURLOPTS -X POST -H 'Content-Type: application/yang-data+json' -H 'Accept: application/yang-data+xml' -d '<input xmlns="urn:example:clixon"><x>0</x></input>' $RCPROTO://localhost/restconf/operations/clixon-example:example)" 0 "HTTP/$HVER 400" "<errors xmlns=\"urn:ietf:params:xml:ns:yang:ietf-restconf\"><error><error-type>rpc</error-type><error-tag>malformed-message</error-tag><error-severity>error</error-severity><error-message>json_parse: line 1: syntax error at or before: '&lt;'</error-message></error></errors>"

test/test_submodule.sh

@@ -122,7 +122,7 @@ EOF
 cat <<EOF > $fsub2
 submodule sub2 {
    yang-version 1.1;
-   belongs-to sub1 {
+   belongs-to main {
       prefix ex;
    }
    import extra2{

test/test_type.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 # Advanced union types and generated code
 # and enum w values
-# The test is run twice, first with dbcache turned on, then turned off.
+# The test is run three times, with dbcache turned on, cache off and zero-copy
 # It is the only test with dbcache off.
 
 # Magic line must be first in script (see README.md)
@@ -47,9 +47,9 @@ module example3{
 EOF
 cat <<EOF > $fyang2
 module example2{
-  import example3 { prefix ex3; }
   namespace "urn:example:example2";
   prefix ex2;
+  import example3 { prefix ex3; }
   grouping gr2 {
     leaf talle{
       type ex3:t;

test/test_union.sh

@@ -50,9 +50,9 @@ module example3{
 EOF
 cat <<EOF > $fyang2
 module example2{
-  import example3 { prefix ex3; }
   namespace "urn:example:example2";
   prefix ex2;
+  import example3 { prefix ex3; }
   grouping gr2 {
     leaf talle{
       type ex3:t;

test/test_upgrade_checkold.sh

@@ -44,8 +44,8 @@ EOF
 cat <<EOF > $fyang2
 module A{
   prefix a;
-  revision 2021-01-01;
   namespace "urn:example:a";
+  revision 2021-01-01;
   container upgraded{
   }
 }
@@ -95,8 +95,8 @@ for oldyang in true false; do
     cat <<EOF > $fyang1
 module A{
   prefix a;
-  revision 2016-01-01;
   namespace "urn:example:a";
+  revision 2016-01-01;
   container old{
   }
 }

test/test_upgrade_failsafe.sh

@@ -38,8 +38,8 @@ fyangB=$dir/B@2019-01-01.yang
 cat <<EOF > $fyangA0
 module A{
   prefix a;
-  revision 0814-01-28;
   namespace "urn:example:a";
+  revision 0814-01-28;
   leaf a0{
     type string;
   }
@@ -53,9 +53,9 @@ EOF
 cat <<EOF > $fyangA1
 module A{
   prefix a;
+  namespace "urn:example:a";
   revision 2019-01-01;
   revision 0814-01-28;
-  namespace "urn:example:a";
   /*  leaf a0 has been removed */
   leaf a1{
     description "exists in both versions";
@@ -72,8 +72,8 @@ EOF
 cat <<EOF > $fyangB
 module B{
   prefix b;
-  revision 2019-01-01;
   namespace "urn:example:b";
+  revision 2019-01-01;
   leaf b{
     type string;
   }
@@ -84,8 +84,8 @@ EOF
 cat <<EOF > /dev/null
 module C{
   prefix c;
-  revision 2019-01-01;
   namespace "urn:example:c";
+  revision 2019-01-01;
   leaf c{
     type string;
   }

test/test_upgrade_module.sh

@@ -46,8 +46,8 @@ touch $log
 cat <<EOF > $fyangb
 module B{
   prefix b;
-  revision 2016-01-01;
   namespace "urn:example:b";
+  revision 2016-01-01;
   container dummy{
   }
 }
@@ -258,8 +258,8 @@ function testall()
     cat <<EOF > $fyang
 module A{
   prefix a;
-  revision 2016-01-01;
   namespace "urn:example:interfaces";
+  revision 2016-01-01;
   container dummy{
   }
 }

test/test_upgrade_repair.sh

@@ -23,8 +23,8 @@ NEWXML='<a2 xmlns="urn:example:a">new version</a2>'
 cat <<EOF > $fyangA0
 module A{
   prefix a;
-  revision 0814-01-28;
   namespace "urn:example:a";
+  revision 0814-01-28;
   leaf a0{
     type string;
   }
@@ -38,9 +38,9 @@ EOF
 cat <<EOF > $fyangA1
 module A{
   prefix a;
+  namespace "urn:example:a";
   revision 2019-01-01;
   revision 0814-01-28;
-  namespace "urn:example:a";
   /*  leaf a0 has been removed */
   leaf a1{
     description "exists in both versions";

test/test_xpath_canonical.sh

@@ -48,13 +48,13 @@ new "xpath canonical form (other)"
 expectpart "$($clixon_util_xpath -c -y $ydir -p /i:x/j:y -n i:urn:example:a -n j:urn:example:b)" 0 '/a:x/b:y' '0 : a = "urn:example:a"' '1 : b = "urn:example:b"'
 
 new "xpath canonical form predicate 1"
-expectpart "$($clixon_util_xpath -c -y $ydir -p "/i:x[j:y='e1']" -n i:urn:example:a -n j:urn:example:b)" 0 "/a:x\[b:y = 'e1'\]" '0 : a = "urn:example:a"' '1 : b = "urn:example:b"'
+expectpart "$($clixon_util_xpath -c -y $ydir -p "/i:x[j:y='e1']" -n i:urn:example:a -n j:urn:example:b)" 0 "/a:x\[b:y='e1'\]" '0 : a = "urn:example:a"' '1 : b = "urn:example:b"'
 
 new "xpath canonical form predicate self"
-expectpart "$($clixon_util_xpath -c -y $ydir -p "/i:x[.='42']" -n i:urn:example:a -n j:urn:example:b)" 0 "/a:x\[. = '42'\]" '0 : a = "urn:example:a"'
+expectpart "$($clixon_util_xpath -c -y $ydir -p "/i:x[.='42']" -n i:urn:example:a -n j:urn:example:b)" 0 "/a:x\[.='42'\]" '0 : a = "urn:example:a"'
 
 new "xpath canonical form descendants"
-expectpart "$($clixon_util_xpath -c -y $ydir -p "//x[.='42']" -n null:urn:example:a -n j:urn:example:b)" 0 "//a:x\[. = '42'\]" '0 : a = "urn:example:a"'
+expectpart "$($clixon_util_xpath -c -y $ydir -p "//x[.='42']" -n null:urn:example:a -n j:urn:example:b)" 0 "//a:x\[.='42'\]" '0 : a = "urn:example:a"'
 
 new "xpath canonical form (no default should fail)"
 expectpart "$($clixon_util_xpath -c -y $ydir -p /x/j:y -n i:urn:example:a -n j:urn:example:b 2> /dev/null)" 255

test/test_yang_load.sh

@@ -22,9 +22,9 @@ fyang3=$dir/other.yang
 cat <<EOF > $fyang1
 module example{
   prefix ex;
+  namespace "urn:example:clixon";
   revision $NEWDATE;
   revision $OLDDATE;
-  namespace "urn:example:clixon";
   leaf newex{
     type string;
   }
@@ -35,8 +35,8 @@ EOF
 cat <<EOF > $fyang2
 module example{
   prefix ex;
-  revision $OLDDATE;
   namespace "urn:example:clixon";
+  revision $OLDDATE;
   leaf oldex{
     type string;
   }
@@ -47,8 +47,8 @@ EOF
 cat <<EOF > $fyang3
 module other{
   prefix oth;
-  revision $NEWDATE;
   namespace "urn:example:clixon2";
+  revision $NEWDATE;
   leaf other{
     type string;
   }

test/vagrant/vagrant.sh

@@ -169,7 +169,7 @@ case $release in
 	    native)
 		$sshcmd sudo yum install -y libevent openssl
 		$sshcmd sudo yum install -y libevent-devel openssl-devel
-		$sshcmd sudo yum-config-manager --enable powertools
+		$sshcmd sudo dnf config-manager --set-enabled powertools
 		$sshcmd sudo yum install -y libnghttp2-devel
 		;;
 	esac
@@ -215,7 +215,7 @@ case $release in
 		;;
 	    native)
 #		$sshcmd sudo apt install -y libevent-2.1
-		$sshcmd sudo apt install -y libevent-dev libssl-dev libnghttp2-dev
+		$sshcmd sudo apt install -y libevent-dev libssl-dev
 		;;
 	esac
 	;;

util/Makefile.in

@@ -91,7 +91,7 @@ APPSRC   += clixon_util_path.c
 APPSRC   += clixon_util_datastore.c
 APPSRC   += clixon_util_regexp.c
 APPSRC   += clixon_util_socket.c
-APPSRC   += clixon_util_validate.c
+APPSRC   += clixon_util_validate.c 
 APPSRC   += clixon_netconf_ssh_callhome.c
 APPSRC   += clixon_netconf_ssh_callhome_client.c
 ifdef with_restconf

yang/clixon/Makefile.in

@@ -41,7 +41,7 @@ datarootdir     = @datarootdir@
 # See also OPT_YANG_INSTALLDIR for the standard yang files
 YANG_INSTALLDIR   = @YANG_INSTALLDIR@
 
-YANGSPECS	 = clixon-config@2021-05-20.yang   # 5.2
+YANGSPECS	 = clixon-config@2021-07-11.yang   # 5.3
 YANGSPECS	+= clixon-lib@2021-03-08.yang      # 5.1
 YANGSPECS	+= clixon-rfc5277@2008-07-01.yang
 YANGSPECS	+= clixon-xml-changelog@2019-03-21.yang

yang/clixon/clixon-config.yang

@@ -1 +0,0 @@
-clixon-config@2021-03-08.yang

yang/clixon/clixon-config@2021-05-20.yang

@@ -50,6 +50,7 @@ module clixon-config {
 	            CLICON_RESTCONF_PRIVILEGES
 	            CLICON_RESTCONF_INSTALLDIR
 	            CLICON_RESTCONF_STARTUP_DONTUPDATE
+                    CLICON_NETCONF_MESSAGE_ID_OPTIONAL
              Released in Clixon 5.2";
     }
     revision 2021-03-08 {
@@ -465,6 +466,16 @@ module clixon-config {
                  is returned, which conforms to the RFC.
                  Note this applies only to external NETCONF, not the internal (IPC) netconf";
 	}
+	leaf CLICON_NETCONF_MESSAGE_ID_OPTIONAL {
+	    type boolean;
+	    default false;
+	    description
+		"This option relates to RFC 6241 Sec 4.1 <rpc> Element
+                 The <rpc> element has a mandatory attribute 'message-id', which is a
+                 string chosen by the sender of the RPC.
+                 If true, an RPC can be sent without a message-id.
+                 This applies to both  external NETCONF and internal (IPC) netconf";
+	}
 	leaf CLICON_RESTCONF_DIR {
 	    type string;
 	    description

yang/clixon/clixon-config@2021-07-11.yang

@@ -43,12 +43,33 @@ module clixon-config {
 
        ***** END LICENSE BLOCK *****";
 
+    revision 2021-07-11 {
+	description
+	    "Added option:
+                    CLICON_SYSTEM_CAPABILITIES
+             Removed default value:
+	            CLICON_RESTCONF_INSTALLDIR
+             Marked as obsolete:
+                     CLICON_YANG_LIST_CHECK
+             (Will be) Released in Clixon 5.3";
+    }
+    revision 2021-05-20 {
+	description
+	    "Added option:
+	            CLICON_RESTCONF_USER
+	            CLICON_RESTCONF_PRIVILEGES
+	            CLICON_RESTCONF_INSTALLDIR
+	            CLICON_RESTCONF_STARTUP_DONTUPDATE
+                    CLICON_NETCONF_MESSAGE_ID_OPTIONAL
+             Released in Clixon 5.2";
+    }
     revision 2021-03-08 {
 	description
 	    "Added option:
                    CLICON_NETCONF_HELLO_OPTIONAL
 		   CLICON_CLI_AUTOCLI_EXCLUDE
-	           CLICON_XMLDB_UPGRADE_CHECKOLD";
+	           CLICON_XMLDB_UPGRADE_CHECKOLD
+	     Released in Clixon 5.1";
     }
     revision 2020-12-30 {
 	description
@@ -171,6 +192,10 @@ module clixon-config {
 		"Commit startup configuration into running state
                  After reboot when no persistent running db exists";
 	    }
+	    enum running-startup{
+		description
+		    "First try running db, if it is empty try startup db.";
+	    }
 	}
     }
     typedef datastore_format{
@@ -406,7 +431,11 @@ module clixon-config {
 		"If false, skip Yang list check sanity checks from RFC 7950, Sec 7.8.2: 
                  The 'key' statement, which MUST be present if the list represents configuration.
                  Some yang specs seem not to fulfil this. However, if you reset this, there may
-                 be follow-up errors due to code that assumes a configuration list has keys";
+                 be follow-up errors due to code that assumes a configuration list has keys
+                 Marked as obsolete since the observation above seemed to be related to the
+                 yang-data extension in RFC8040 allows non-key lists. This has been implemented
+                 by a YANG_FLAG_NOKEY yang flag mechanism";
+	    status obsolete;
 	}
 	leaf CLICON_YANG_UNKNOWN_ANYDATA{
 	    type boolean;
@@ -421,6 +450,18 @@ module clixon-config {
                  only loading from startup but may occur in other circumstances as well. This
                  means that sanity checks of erroneous XML/JSON may not be properly signalled.";
 	}
+	leaf CLICON_SYSTEM_CAPABILITIES {
+	    type boolean;
+	    default false;
+	    description
+		"Enable module ietf-system-capabilities and ietf-notification-capabilities
+                 Note: There are several dependencies:
+                 - ietf-yang-library revision 2019-01-04 is REQUIRED 
+                 - nacm
+                 - ietf-yang-structure-ext.yang, 
+                 - ietf-yang-instance-data
+                 see draft-ietf-netconf-notification-capabilities-17";
+	}
 	leaf CLICON_BACKEND_DIR {
 	    type string;
 	    description
@@ -451,6 +492,16 @@ module clixon-config {
                  is returned, which conforms to the RFC.
                  Note this applies only to external NETCONF, not the internal (IPC) netconf";
 	}
+	leaf CLICON_NETCONF_MESSAGE_ID_OPTIONAL {
+	    type boolean;
+	    default false;
+	    description
+		"This option relates to RFC 6241 Sec 4.1 <rpc> Element
+                 The <rpc> element has a mandatory attribute 'message-id', which is a
+                 string chosen by the sender of the RPC.
+                 If true, an RPC can be sent without a message-id.
+                 This applies to both  external NETCONF and internal (IPC) netconf";
+	}
 	leaf CLICON_RESTCONF_DIR {
 	    type string;
 	    description
@@ -470,7 +521,33 @@ module clixon-config {
                  Note: Obsolete, use fcgi-socket in clixon-restconf.yang instead";
 	    status obsolete;
 	}
-
+	leaf CLICON_RESTCONF_INSTALLDIR {
+	    type string;
+	    description
+		"If set, path to dir of clixon-restconf daemon binary as used by backend if 
+                 started internally (run-time).
+                 If this path is not set, clixon_restconf will be looked for according to
+                 configured installdir: $(sbindir) (install-time)
+                 Since programs can be moved around at install/cross-compile time the installed
+                 dir may be difficult to know at install time, which is the reason why 
+                 CLICON_RESTCONF_INSTALLDIR exists, in order to override the Makefile 
+                 installdir.
+                 Note on the installdir, DESTDIR is not included since according to man pages:
+                     by specifying DESTDIR should not change the operation of the software in 
+                     any way, so its value should not be included in any file contents. ";
+	}
+	leaf CLICON_RESTCONF_STARTUP_DONTUPDATE {
+	    type boolean;
+	    default false;
+	    description
+		"According to RFC 8040 Sec 1.4:
+                    If the NETCONF server supports :startup, the RESTCONF server MUST automatically
+                    update the [...] startup configuration [...] as a consequence of a RESTCONF
+                    edit operation.
+                 Setting this option disables this behaviour, ie the startup configuration is NOT
+                 automatically updated.
+                 If this option is false, the startup is autoamtically updated following the RFC";
+	}
 	leaf CLICON_RESTCONF_PRETTY {
 	    type boolean;
 	    default true;
@@ -486,6 +563,26 @@ module clixon-config {
                  Note: Obsolete, use pretty in clixon-restconf.yang instead";
 	    status obsolete;
 	}
+	leaf CLICON_RESTCONF_USER {
+	    type string;
+	    description 
+		"Run clixon_daemon as this user
+                 When drop privileges is used, the daemon will drop privileges to this user.
+                 In pre-5.2 code this was configured as compile-time constant WWWUSER with
+                 default value www-data
+                 See also CLICON_PRIVILEGES setting";
+	    default www-data;
+	}
+	leaf CLICON_RESTCONF_PRIVILEGES {
+	    type priv_mode;
+	    default drop_perm;
+	    description 
+		"Restconf privileges mode. 
+                 If drop_perm or drop_temp then drop privileges to CLICON_RESTCONF_USER.
+                 If the platform does not support getresuid and accompanying functions, the mode
+                 must be set to 'none'.
+                 ";
+	}
 	leaf CLICON_CLI_DIR {
 	    type string;
 	    description
@@ -706,7 +803,7 @@ module clixon-config {
                  user (eg datastores).
                  It also sets the backend unix socket owner to this user, but its group
                  is set by CLICON_SOCK_GROUP.
-                 See also CLICON_PRIVILEGES setting";
+                 See also CLICON_BACKEND_PRIVILEGES setting";
 	}
 	leaf CLICON_BACKEND_PRIVILEGES {
 	    type priv_mode;

yang/clixon/clixon-lib@2020-12-30.yang

@@ -1,180 +0,0 @@
-module clixon-lib {
-    yang-version 1.1;
-    namespace "http://clicon.org/lib";
-    prefix cl;
-
-    organization
-	"Clicon / Clixon";
-
-    contact
-	"Olof Hagsand <olof@hagsand.se>";
-
-    description
-      "Clixon Netconf extensions for communication between clients and backend.
-      
-       ***** BEGIN LICENSE BLOCK *****
-       Copyright (C) 2009-2019 Olof Hagsand
-       Copyright (C) 2020-2021 Olof Hagsand and Rubicon Communications, LLC(Netgate)
-       
-       This file is part of CLIXON
-
-       Licensed under the Apache License, Version 2.0 (the \"License\");
-       you may not use this file except in compliance with the License.
-       You may obtain a copy of the License at
-            http://www.apache.org/licenses/LICENSE-2.0
-       Unless required by applicable law or agreed to in writing, software
-       distributed under the License is distributed on an \"AS IS\" BASIS,
-       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-       See the License for the specific language governing permissions and
-       limitations under the License.
-
-       Alternatively, the contents of this file may be used under the terms of
-       the GNU General Public License Version 3 or later (the \"GPL\"),
-       in which case the provisions of the GPL are applicable instead
-       of those above. If you wish to allow use of your version of this file only
-       under the terms of the GPL, and not to allow others to
-       use your version of this file under the terms of Apache License version 2, 
-       indicate your decision by deleting the provisions above and replace them with
-       the notice and other provisions required by the GPL. If you do not delete
-       the provisions above, a recipient may use your version of this file under
-       the terms of any one of the Apache License version 2 or the GPL.
-
-       ***** END LICENSE BLOCK *****";
-
-    revision 2020-12-30 {
-	description
-	    "Changed: RPC process-control output parameter status to pid";
-    }
-    revision 2020-12-08 {
-	description
-	    "Added: autocli-op extension.
-                    rpc process-control for process/daemon management
-             Released in clixon 4.9";
-    }
-    revision 2020-04-23 {
-	description
-	    "Added: stats RPC for clixon XML and memory statistics.
-             Added: restart-plugin RPC for restarting individual plugins without restarting backend.";
-    }
-    revision 2019-08-13 {
-	description
-	    "No changes (reverted change)";
-    }
-    revision 2019-06-05 {
-	description
-	    "ping rpc added for liveness";
-    }
-    revision 2019-01-02 {
-	description
-	    "Released in Clixon 3.9";
-    }
-    typedef service-operation {
-        type enumeration {
-            enum start {
-                description
-                    "Start if not already running";
-            }
-            enum stop {
-                description
-                    "Stop if running";
-            }
-            enum restart {
-                description
-                    "Stop if running, then start";
-            }
-            enum status {
-                description
-                    "Check status";
-            }
-        }
-        description
-            "Common operations that can be performed on a service";
-    }
-    extension autocli-op {
-      description 
-        "Takes an argument an operation defing how to modify the clispec at 
-         this point in the YANG tree for the automated generated CLI.
-         Note that this extension is only used in clixon_cli.
-         Operations is expected to be extended, but the following operations are defined:
-         - hide  This command is active but not shown by ? or TAB";
-      argument cliop;
-   }
-   rpc debug {
-	description "Set debug level of backend.";
-	input {
-	    leaf level {
-		type uint32;
-	    }
-	}
-    }
-    rpc ping {
-        description "Check aliveness of backend daemon.";
-    }
-    rpc stats {
-        description "Clixon XML statistics.";
-	output {
-	    container global{
-		description "Clixon global statistics";
-		leaf xmlnr{
-		    description "Number of XML objects: number of residing xml/json objects
-                             in the internal 'cxobj' representation.";
-		    type uint64;
-		}
-	    }
-	    list datastore{
-		description "Datastore statistics";
-		key "name";
-		leaf name{
-		    description "name of datastore (eg running).";
-		    type string;
-		}
-		leaf nr{
-		    description "Number of XML objects. That is number of residing xml/json objects
-                             in the internal 'cxobj' representation.";
-		    type uint64;
-		}
-		leaf size{
-		    description "Size in bytes of internal datastore cache of datastore tree.";
-		    type uint64;
-		}
-	    }
-
-	}
-    }
-    rpc restart-plugin {
-	description "Restart specific backend plugins.";
-	input {
-	    leaf-list plugin {
-		description "Name of plugin to restart";
-		type string;
-	    }
-	}
-    }
-
-    rpc process-control {
-	description
-	    "Control a specific process or daemon: start/stop, etc.
-             This is for direct managing of a process by the backend. 
-             Alternatively one can manage a daemon via systemd, containerd, kubernetes, etc.";
-	input {
-	    leaf name {
-		description "Name of process";
-		type string;
-		mandatory true;
-	    }
-	    leaf operation {
-		type service-operation;
-		mandatory true;
-		description
-		    "One of the strings 'start', 'stop', 'restart', or 'status'.";
-	    }
-	}
-	output {
-	    leaf pid {
-		description "Process-id of running process or 0 if not running
-                             Value is only valid for operation status";
-		type uint32;
-	    }
-	}
-    }
-}

yang/clixon/clixon-restconf@2021-03-15.yang

@@ -1,221 +0,0 @@
-module clixon-restconf {
-    yang-version 1.1;
-    namespace "http://clicon.org/restconf";
-    prefix "clrc";
-
-    import ietf-inet-types {
-	prefix inet;
-    }
-
-    organization
-	"Clixon";
-
-    contact
-	"Olof Hagsand <olof@hagsand.se>";
-
-    description
-	"This YANG module provides a data-model for the Clixon RESTCONF daemon.
-       ***** BEGIN LICENSE BLOCK *****
-       Copyright (C) 2020 Olof Hagsand and Rubicon Communications, LLC(Netgate)
-
-       This file is part of CLIXON
-
-       Licensed under the Apache License, Version 2.0 (the \"License\");
-       you may not use this file except in compliance with the License.
-       You may obtain a copy of the License at
-            http://www.apache.org/licenses/LICENSE-2.0
-       Unless required by applicable law or agreed to in writing, software
-       distributed under the License is distributed on an \"AS IS\" BASIS,
-       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-       See the License for the specific language governing permissions and
-       limitations under the License.
-
-       Alternatively, the contents of this file may be used under the terms of
-       the GNU General Public License Version 3 or later (the \"GPL\"),
-       in which case the provisions of the GPL are applicable instead
-       of those above. If you wish to allow use of your version of this file only
-       under the terms of the GPL, and not to allow others to
-       use your version of this file under the terms of Apache License version 2,
-       indicate your decision by deleting the provisions above and replace them with
-       the notice and other provisions required by the GPL. If you do not delete
-       the provisions above, a recipient may use your version of this file under
-       the terms of any one of the Apache License version 2 or the GPL.
-
-       ***** END LICENSE BLOCK *****";
-
-    revision 2021-03-15 {
-	description
-	    "make authentication-type none a feature
-	     Added flag to enable core dumps";
-    }
-    revision 2020-12-30 {
-	description
-	    "Added: debug field
-             Added 'none' as default value for auth-type
-             Changed http-auth-type enum from 'password' to 'user'";
-    }
-    revision 2020-10-30 {
-	description
-	    "Initial release";
-    }
-
-    feature fcgi {
-	description
-	    "This feature indicates that the restconf server supports the fast-cgi reverse
-             proxy solution.
-             That is, a reverse proxy is the HTTP front-end and the restconf daemon listens
-             to a fcgi socket.
-             The alternative is the internal HTTP solution using evhtp.";
-    }
-
-    feature allow-auth-none {
-        description
-	  "This feature allows the use of authentication-type none.";
-    }
-
-    typedef http-auth-type {
-	type enumeration {
-	    enum none {
-		if-feature "allow-auth-none";
-		description
-		    "Incoming message are set to authenticated by default. No ca-auth callback is called,
-                     Authenticated user is set to special user 'none'.
-                     Typically assumes NACM is not enabled.";
-	    }
-	    enum client-certificate {
-		description
-		    "TLS client certificate validation is made on each incoming message. If it passes
-                    the authenticated user is extracted from the SSL_CN parameter
-                     The ca-auth callback can be used to revise this behavior.";
-	    }
-	    enum user {
-		description
-		    "User-defined authentication as defined by the ca-auth callback.
-                     One example is some form of password authentication, such as basic auth.";
-	    }
-	}
-	description
-	    "Enumeration of HTTP authorization types.";
-    }
-    grouping clixon-restconf{
-	description
-	    "HTTP RESTCONF configuration.";
-	leaf enable {
-	    type boolean;
-	    default "false";
-	    description
-		"Enables RESTCONF functionality.
-                 Note that starting/stopping of a restconf daemon is different from it being
-                 enabled or not.
-                 For example, if the restconf daemon is under systemd management, the restconf
-                 daemon will only start if enable=true.";
-	}
-	leaf auth-type {
-	    type http-auth-type;
-	    description
-		"The authentication type.
-                 Note client-certificate applies only if ssl-enable is true and socket has ssl";
-	    default user;
-	}
-	leaf debug {
-	    description
-		"Set debug level of restconf daemon.
-                 0 is no debug, 1 is debugging, more is detailed debug.
-                 Debug logs will be directed to syslog with
-                    ident: clixon_restconf and PID
-                    facility: LOG_USER
-                    level: LOG_DEBUG";
-	    type uint32;
-	    default 0;
-	}
-	leaf enable-core-dump {
-	    description
-	        "enable core dumps.
-                 this is a no-op on systems that don't support it.";
-	    type boolean;
-	    default false;
-	}
-	leaf pretty {
-	    type boolean;
-	    default true;
-	    description
-		"Restconf return value pretty print.
-                 Restconf clients may add HTTP header:
-                      Accept: application/yang-data+json, or
-                      Accept: application/yang-data+xml
-                 to get return value in XML or JSON.
-                 RFC 8040 examples print XML and JSON in pretty-printed form.
-                 Setting this value to false makes restconf return not pretty-printed
-                 which may be desirable for performance or tests
-                 This replaces the CLICON_RESTCONF_PRETTY option in clixon-config.yang";
-	}
-	/* From this point only specific options
-	 * First fcgi-specific options
-	 */
-	leaf fcgi-socket {
-	    if-feature fcgi; /* Set by default by fcgi clixon_restconf daemon */
-	    type string;
-	    default "/www-data/fastcgi_restconf.sock";
-	    description
-		"Path to FastCGI unix socket. Should be specified in webserver
-         	 Eg in nginx: fastcgi_pass unix:/www-data/clicon_restconf.sock
-                 Only if with-restconf=fcgi, NOT evhtp
-                 This replaces CLICON_RESTCONF_PATH option in clixon-config.yang";
-	}
-	/* Second, evhtp-specific options */
-	leaf server-cert-path {
-	    type string;
-	    description
-		"Path to server certificate file.
-                 Note only applies if socket has ssl enabled";
-	}
-	leaf server-key-path {
-	    type string;
-	    description
-		"Path to server key file
-                 Note only applies if socket has ssl enabled";
-	}
-	leaf server-ca-cert-path {
-	    type string;
-	    description
-		"Path to server CA cert file
-	         Note only applies if socket has ssl enabled";
-	}
-	list socket {
-	    description
-		"List of server sockets that the restconf daemon listens to";
-	    key "namespace address port";
-	    leaf namespace {
-		type string;
-		description
-		    "Network namespace.
-                     On platforms where namespaces are not suppported, 'default'
-                     Default value can be changed by RESTCONF_NETNS_DEFAULT";
-	    }
-	    leaf address {
-		type inet:ip-address;
-		description "IP address to bind to";
-	    }
-	    leaf port {
-		type inet:port-number;
-		description "TCP port to bind to";
-	    }
-	    leaf ssl {
-		type boolean;
-		default true;
-		description "Enable for HTTPS otherwise HTTP protocol";
-	    }
-	}
-    }
-    container restconf {
-	description
-	    "This presence is strictly not necessary since the enable flag
-             in clixon-restconf is the flag bearing the actual semantics.
-             However, removing the presence leads to default config in all
-             clixon installations, even those which do not use backend-started restconf.
-             One could see this as mostly cosmetically annoying.
-             Alternative would be to make the inclusion of this yang conditional.";
-	presence "Enables RESTCONF";
-	uses clixon-restconf;
-    }
-}

yang/mandatory/Makefile.in

@@ -50,6 +50,7 @@ YANGSPECS += ietf-restconf-monitoring@2017-01-26.yang
 YANGSPECS += ietf-yang-library@2019-01-04.yang
 YANGSPECS += ietf-yang-types@2013-07-15.yang
 YANGSPECS += ietf-datastores@2018-02-14.yang
+YANGSPECS += ietf-yang-patch@2017-02-22.yang
 
 all:	
 

yang/mandatory/ietf-yang-patch@2017-02-22.yang

@@ -0,0 +1,390 @@
+module ietf-yang-patch {
+  yang-version 1.1;
+  namespace "urn:ietf:params:xml:ns:yang:ietf-yang-patch";
+  prefix "ypatch";
+
+  import ietf-restconf { prefix rc; }
+
+  organization
+    "IETF NETCONF (Network Configuration) Working Group";
+
+  contact
+    "WG Web:   <https://datatracker.ietf.org/wg/netconf/>
+     WG List:  <mailto:netconf@ietf.org>
+
+     Author:   Andy Bierman
+               <mailto:andy@yumaworks.com>
+
+     Author:   Martin Bjorklund
+               <mailto:mbj@tail-f.com>
+
+     Author:   Kent Watsen
+               <mailto:kwatsen@juniper.net>";
+
+  description
+    "This module contains conceptual YANG specifications
+     for the YANG Patch and YANG Patch Status data structures.
+
+     Note that the YANG definitions within this module do not
+     represent configuration data of any kind.
+     The YANG grouping statements provide a normative syntax
+     for XML and JSON message-encoding purposes.
+
+     Copyright (c) 2017 IETF Trust and the persons identified as
+     authors of the code.  All rights reserved.
+
+     Redistribution and use in source and binary forms, with or
+     without modification, is permitted pursuant to, and subject
+     to the license terms contained in, the Simplified BSD License
+     set forth in Section 4.c of the IETF Trust's Legal Provisions
+     Relating to IETF Documents
+     (http://trustee.ietf.org/license-info).
+
+     This version of this YANG module is part of RFC 8072; see
+     the RFC itself for full legal notices.";
+
+  revision 2017-02-22 {
+    description
+      "Initial revision.";
+    reference
+      "RFC 8072: YANG Patch Media Type.";
+  }
+
+  typedef target-resource-offset {
+    type string;
+    description
+      "Contains a data resource identifier string representing
+       a sub-resource within the target resource.
+       The document root for this expression is the
+       target resource that is specified in the
+       protocol operation (e.g., the URI for the PATCH request).
+
+       This string is encoded according to the same rules as those
+       for a data resource identifier in a RESTCONF request URI.";
+    reference
+       "RFC 8040, Section 3.5.3.";
+  }
+
+  rc:yang-data "yang-patch" {
+    uses yang-patch;
+  }
+
+  rc:yang-data "yang-patch-status" {
+    uses yang-patch-status;
+  }
+
+  grouping yang-patch {
+
+    description
+      "A grouping that contains a YANG container representing the
+       syntax and semantics of a YANG Patch edit request message.";
+
+    container yang-patch {
+      description
+        "Represents a conceptual sequence of datastore edits,
+         called a patch.  Each patch is given a client-assigned
+         patch identifier.  Each edit MUST be applied
+         in ascending order, and all edits MUST be applied.
+         If any errors occur, then the target datastore MUST NOT
+         be changed by the YANG Patch operation.
+
+         It is possible for a datastore constraint violation to occur
+         due to any node in the datastore, including nodes not
+         included in the 'edit' list.  Any validation errors MUST
+         be reported in the reply message.";
+
+      reference
+        "RFC 7950, Section 8.3.";
+
+      leaf patch-id {
+        type string;
+        mandatory true;
+        description
+          "An arbitrary string provided by the client to identify
+           the entire patch.  Error messages returned by the server
+           that pertain to this patch will be identified by this
+           'patch-id' value.  A client SHOULD attempt to generate
+           unique 'patch-id' values to distinguish between
+           transactions from multiple clients in any audit logs
+           maintained by the server.";
+      }
+
+      leaf comment {
+        type string;
+        description
+          "An arbitrary string provided by the client to describe
+           the entire patch.  This value SHOULD be present in any
+           audit logging records generated by the server for the
+           patch.";
+      }
+
+      list edit {
+        key edit-id;
+        ordered-by user;
+
+        description
+          "Represents one edit within the YANG Patch request message.
+           The 'edit' list is applied in the following manner:
+
+             - The first edit is conceptually applied to a copy
+               of the existing target datastore, e.g., the
+               running configuration datastore.
+             - Each ascending edit is conceptually applied to
+               the result of the previous edit(s).
+             - After all edits have been successfully processed,
+               the result is validated according to YANG constraints.
+             - If successful, the server will attempt to apply
+               the result to the target datastore.";
+
+        leaf edit-id {
+          type string;
+          description
+            "Arbitrary string index for the edit.
+             Error messages returned by the server that pertain
+             to a specific edit will be identified by this value.";
+        }
+
+        leaf operation {
+          type enumeration {
+            enum create {
+              description
+                "The target data node is created using the supplied
+                 value, only if it does not already exist.  The
+                 'target' leaf identifies the data node to be
+                 created, not the parent data node.";
+            }
+            enum delete {
+              description
+                "Delete the target node, only if the data resource
+                 currently exists; otherwise, return an error.";
+            }
+
+            enum insert {
+              description
+                "Insert the supplied value into a user-ordered
+                 list or leaf-list entry.  The target node must
+                 represent a new data resource.  If the 'where'
+                 parameter is set to 'before' or 'after', then
+                 the 'point' parameter identifies the insertion
+                 point for the target node.";
+            }
+            enum merge {
+              description
+                "The supplied value is merged with the target data
+                 node.";
+            }
+            enum move {
+              description
+                "Move the target node.  Reorder a user-ordered
+                 list or leaf-list.  The target node must represent
+                 an existing data resource.  If the 'where' parameter
+                 is set to 'before' or 'after', then the 'point'
+                 parameter identifies the insertion point to move
+                 the target node.";
+            }
+            enum replace {
+              description
+                "The supplied value is used to replace the target
+                 data node.";
+            }
+            enum remove {
+              description
+                "Delete the target node if it currently exists.";
+            }
+          }
+          mandatory true;
+          description
+            "The datastore operation requested for the associated
+             'edit' entry.";
+        }
+
+        leaf target {
+          type target-resource-offset;
+          mandatory true;
+          description
+            "Identifies the target data node for the edit
+             operation.  If the target has the value '/', then
+             the target data node is the target resource.
+             The target node MUST identify a data resource,
+             not the datastore resource.";
+        }
+
+        leaf point {
+          when "(../operation = 'insert' or ../operation = 'move')"
+             + "and (../where = 'before' or ../where = 'after')" {
+            description
+              "This leaf only applies for 'insert' or 'move'
+               operations, before or after an existing entry.";
+          }
+          type target-resource-offset;
+          description
+            "The absolute URL path for the data node that is being
+             used as the insertion point or move point for the
+             target of this 'edit' entry.";
+        }
+
+        leaf where {
+          when "../operation = 'insert' or ../operation = 'move'" {
+            description
+              "This leaf only applies for 'insert' or 'move'
+               operations.";
+          }
+          type enumeration {
+            enum before {
+              description
+                "Insert or move a data node before the data resource
+                 identified by the 'point' parameter.";
+            }
+            enum after {
+              description
+                "Insert or move a data node after the data resource
+                 identified by the 'point' parameter.";
+            }
+
+            enum first {
+              description
+                "Insert or move a data node so it becomes ordered
+                 as the first entry.";
+            }
+            enum last {
+              description
+                "Insert or move a data node so it becomes ordered
+                 as the last entry.";
+            }
+          }
+          default last;
+          description
+            "Identifies where a data resource will be inserted
+             or moved.  YANG only allows these operations for
+             list and leaf-list data nodes that are
+             'ordered-by user'.";
+        }
+
+        anydata value {
+          when "../operation = 'create' "
+             + "or ../operation = 'merge' "
+             + "or ../operation = 'replace' "
+             + "or ../operation = 'insert'" {
+            description
+              "The anydata 'value' is only used for 'create',
+               'merge', 'replace', and 'insert' operations.";
+          }
+          description
+            "Value used for this edit operation.  The anydata 'value'
+             contains the target resource associated with the
+             'target' leaf.
+
+             For example, suppose the target node is a YANG container
+             named foo:
+
+                 container foo {
+                   leaf a { type string; }
+                   leaf b { type int32; }
+                 }
+
+             The 'value' node contains one instance of foo:
+
+                 <value>
+                    <foo xmlns='example-foo-namespace'>
+                       <a>some value</a>
+                       <b>42</b>
+                    </foo>
+                 </value>
+              ";
+        }
+      }
+    }
+
+  } // grouping yang-patch
+
+  grouping yang-patch-status {
+
+    description
+      "A grouping that contains a YANG container representing the
+       syntax and semantics of a YANG Patch Status response
+       message.";
+
+    container yang-patch-status {
+      description
+        "A container representing the response message sent by the
+         server after a YANG Patch edit request message has been
+         processed.";
+
+      leaf patch-id {
+        type string;
+        mandatory true;
+        description
+          "The 'patch-id' value used in the request.";
+      }
+
+      choice global-status {
+        description
+          "Report global errors or complete success.
+           If there is no case selected, then errors
+           are reported in the 'edit-status' container.";
+
+        case global-errors {
+          uses rc:errors;
+          description
+            "This container will be present if global errors that
+             are unrelated to a specific edit occurred.";
+        }
+        leaf ok {
+          type empty;
+          description
+            "This leaf will be present if the request succeeded
+             and there are no errors reported in the 'edit-status'
+             container.";
+        }
+      }
+
+      container edit-status {
+        description
+          "This container will be present if there are
+           edit-specific status responses to report.
+           If all edits succeeded and the 'global-status'
+           returned is 'ok', then a server MAY omit this
+           container.";
+
+        list edit {
+          key edit-id;
+
+          description
+            "Represents a list of status responses,
+             corresponding to edits in the YANG Patch
+             request message.  If an 'edit' entry was
+             skipped or not reached by the server,
+             then this list will not contain a corresponding
+             entry for that edit.";
+
+          leaf edit-id {
+            type string;
+             description
+               "Response status is for the 'edit' list entry
+                with this 'edit-id' value.";
+          }
+
+          choice edit-status-choice {
+            description
+              "A choice between different types of status
+               responses for each 'edit' entry.";
+            leaf ok {
+              type empty;
+              description
+                "This 'edit' entry was invoked without any
+                 errors detected by the server associated
+                 with this edit.";
+            }
+            case errors {
+              uses rc:errors;
+              description
+                "The server detected errors associated with the
+                 edit identified by the same 'edit-id' value.";
+            }
+          }
+        }
+      }
+    }
+  }  // grouping yang-patch-status
+
+}