* Preparations for Clixon 5.4 release, docs, readme:s, etc

CHANGELOG.md

@@ -34,7 +34,7 @@
 ## 5.4.0
 Expected: November, 2021
 
-Thanks netgate for providing the dispatcher code!
+Thanks Netgate for providing the dispatcher code (used in the pagination)!
 
 ### New features
 
@@ -44,11 +44,22 @@ Thanks netgate for providing the dispatcher code!
     * Use accessor functions `pagination_offset()`, `pagination_limit()`, etc
   * Reverted state data callback API to pre-5.3 (see C/CLI API changes below)
   * See https://clixon-docs.readthedocs.io/en/latest/pagination.html
+* Added support for XPATH function `bit-is-set()`
+* Added: [Recursive search CLIXON_YANG_DIR](https://github.com/clicon/clixon/issues/284)
+* Added statistics for YANG: number of objects and memory used
+  * See clixon-lib: stats rpc
 
 ### API changes on existing protocol/config features
 
 Users may have to change how they access the system
  
+* Optional yangs for testing have been removed from the Clixon repo
+  * These were included for testing
+  * If you want to run the Clixon test suite you need to point `YANGMODELS`, see test/README.md
+  * The following configure options have been removed:
+    * `configure --with-opt-yang-installdir=DIR`
+    * `configure   --enable-optyangs`
+  * You may need to specify standard YANGs using configure option `--with-yang-standard-dir=DIR`
 * RPC replies now verified with YANG
   * Stricter checking of outgoing RPC replies from server
   * See [RPC output not verified by yang](https://github.com/clicon/clixon/issues/283)
@@ -63,7 +74,8 @@ Users may have to change how they access the system
   * Modified option: RPC stats extended with YANG stats
 * New `clixon-config@2021-11-11.yang` revision
   * Added option:
-    * CLICON_PLUGIN_CALLBACK_CHECK
+    * `CLICON_PLUGIN_CALLBACK_CHECK`
+    * `CLICON_YANG_AUGMENT_ACCEPT_BROKEN`
   * Modified options:
     * CLICON_CLI_GENMODEL_TYPE: added OC_COMPRESS enum
     * CLICON_YANG_DIR: recursive search
@@ -79,6 +91,7 @@ Users may have to change how they access the system
   * See also updated [https://clixon-docs.readthedocs.io/en/latest/pagination.html]
 * NETCONF hello errors, such as wrong session-id, prefix, namespace terminates session
   * Instead of returning an rpc-error reply
+  * This conforms to RFC 6241
 
 ### C/CLI-API changes on existing features
 
@@ -90,28 +103,30 @@ Developers may need to change their code
   ```
   int statedata(clicon_handle     h,
                 cvec             *nsc,
-	        char             *xpath,
-	        cxobj            *xstate)
+                char             *xpath,
+                cxobj            *xstate)
   ```
 
 ### Minor features
 
-* Added statistics for YANG: number of objects and memory used
-  * See clixon-lib: stats rpc
+* Added configure option `--with-yang-standard-dir=DIR`
+  * Directory of standard IETF/IEEE YANG specs
+* Added option `CLICON_YANG_AUGMENT_ACCEPT_BROKEN` to accept broken yangmodels.
+  * This is a debug option for CI testcases where standard YANG models are broken
 * Performance improvement
   * Added ancestor config cache indicating wether the node or an ancestor is config false or true
-  * Improved yang cardinality lookup
+  * Improved performance of yang cardinality lookup
 * Added sorting of YANG statements
-  * Some openconfig specs seem to have use/when before a "config" which it depends on. This leads to XML encoding being in the "wrong order.
+  * Some openconfig specs seem to have use/when before a "config" which it depends on. This leads to XML encoding being in the "wrong" order.
   * When parsing, clixon now sorts container/list statements so that sub-statements with WHEN are put last.
   * See [Statements given in "load set" are order dependent](https://github.com/clicon/clixon/issues/287)
-* Added: [Recursive search CLIXON_YANG_DIR](https://github.com/clicon/clixon/issues/284)
 * Plugin context check before and after all callbacks.
   * Check blocked signals and signal handlers
   * Check termios settings
   * Any changes to context are logged at loglevel WARNING
-  * New option: CLICON_PLUGIN_CALLBACK_CHECK: enable it to for checks (default false)
-* [OpenConfig path compression](https://github.com/clicon/clixon/pull/276)
+  * New option: `CLICON_PLUGIN_CALLBACK_CHECK`: enable it to for checks (default false)
+* Added: [OpenConfig Path Compression Support](https://github.com/clicon/clixon/issues/274)
+  * PR: [OpenConfig path compression](https://github.com/clicon/clixon/pull/276)
 * C API: Added set/get pointer API to clixon_data:
    * Changed signature of `rpc_callback_call()`
    * Added json/cli support for cli save/load
@@ -124,6 +139,7 @@ Developers may need to change their code
 
 ### Corrected Bugs
 
+* [JSON leaf-list output single element leaf-list does not use array](https://github.com/clicon/clixon/issues/289)
 * [very slow execution of load_set_file #288](https://github.com/clicon/clixon/issues/288)
 * [RPC output not verified by yang](https://github.com/clicon/clixon/issues/283)
 * [Statements given in "load set" are order dependent](https://github.com/clicon/clixon/issues/287)

README.md

@@ -17,6 +17,6 @@ Latest release is 5.3.0 from September 2021. See [CHANGELOG.md](CHANGELOG.md) re
 
 Clixon interaction is best done posting issues, pull requests, or joining the
 [slack channel](https://clixondev.slack.com).
-[Slack invite](https://join.slack.com/t/clixondev/shared_invite/zt-wegsemtw-u~VRdvYPtSRKXqULDLlWJQ) (updated 26/9 2021)
+[Slack invite](https://join.slack.com/t/clixondev/shared_invite/zt-za7p40vq-l_Ib1Cq9rX~cHCkLCdu~Aw) (updated 26/11 2021)
 
 Clixon is sponsored by [Rubicon Communications LLC(Netgate)](https://www.netgate.com/)

docker/base/Dockerfile

@@ -76,7 +76,7 @@ COPY clixon .
 RUN adduser -D -H www-data
 
 # Configure, build and install clixon
-RUN ./configure --prefix=/clixon/build --with-cligen=/clixon/build --with-wwwuser=www-data --enable-optyangs --with-restconf=native
+RUN ./configure --prefix=/clixon/build --with-cligen=/clixon/build --with-wwwuser=www-data --with-restconf=native
 
 RUN make
 RUN make install

lib/src/clixon_file.c

@@ -138,7 +138,7 @@ clicon_files_recursive(const char *dir,
     int     res = 0;
     char    errbuf[128];
  
-    clicon_debug(1, "%s dir:%s", __FUNCTION__, dir);
+    clicon_debug(2, "%s dir:%s", __FUNCTION__, dir);
     if (regexp && (res = regcomp(&re, regexp, REG_EXTENDED)) != 0) {
         regerror(res, &re, errbuf, sizeof(errbuf));
         clicon_err(OE_DB, 0, "regcomp: %s", errbuf);

lib/src/clixon_xml.c

@@ -170,7 +170,7 @@ struct xml{
     struct xml       *x_up_candidate; /* Candidate parent node for special cases (when+xpath) */
 #endif
     int              _x_vector_i;   /* internal use: xml_child_each */
-    int              _x_i;          /* internal use for sorting: 
+    int              _x_i;          /* internal use for stable sorting: 
 				       see xml_enumerate and xml_cmp */
     /*----- next is body/attribute only */
     cbuf             *x_value_cb;  /* attribute and body nodes have values (XXX: this consumes 

lib/src/clixon_xml_sort.c

@@ -416,7 +416,7 @@ xml_sort(cxobj *x)
     if ((ys = xml_spec(x)) != 0	&& yang_config(ys)==0)
 	return 1;
 #endif
-    xml_enumerate_children(x);
+    xml_enumerate_children(x); /* This is to make sorting "stable", ie not change existing order */
     qsort(xml_childvec_get(x), xml_child_nr(x), sizeof(cxobj *), xml_cmp_qsort);
     return 0;
 }

lib/src/clixon_yang.c

@@ -2690,6 +2690,7 @@ yang_if_feature(clicon_handle h,
     int         opand = -1; /* -1:not set, 0:or, 1:and */
     int         enabled = 0;
     cg_var     *cv;
+
     if ((vec = clicon_strsep(ys->ys_argument, " \t\r\n", &nvec)) == NULL)
 	goto done;
     /* Two steps: first detect operators
@@ -3026,7 +3027,8 @@ schema_nodeid_iterate(yang_stmt    *yn,
 		goto done;
 	}
 	if (ys == NULL){
-	    clicon_debug(1, "%s: %s not found", __FUNCTION__, id);
+	    clicon_debug(1, "%s: %s not found, last id found:%s",
+			 __FUNCTION__, id, yang_argument_get(yp));
 	    goto ok;
 	}
 	yp = ys; /* ys is matched */

test/fuzz/README.md

@@ -15,7 +15,9 @@ On ubuntu this may be enough:
 ```
   sudo apt install afl
 ```
-Or get source: `https://github.com/google/AFL`
+
+Or get source: `https://github.com/google/AFL`, see
+docs/QuickStartGuide.txt for instructions, essentially type `make`
 
 You may have to change cpu frequency:
 ```

test/fuzz/cli/README.md

@@ -11,6 +11,8 @@ Build and install a clixon system (in particular the backend, the CLI will be re
 Build and install CLIgen statically:
 ```
   ./configure LINKAGE=static INSTALLFLAGS="" CC=/usr/bin/afl-clang-fast
+  make
+  sudo make install
 ```
 
 ## Build
@@ -39,7 +41,7 @@ Below is an example of how to do this for the main example. You can replace the
   make
   sudo make install
   
-  cd example # Compile and install application plugins (here main example)
+  cd example/main # Compile and install application plugins (here main example)
   make clean
   make
   sudo make install