Commit graph

53 commits

Author SHA1 Message Date
Olof hagsand
3d10c3bfcb evhtp debugging: memleaks, test errors, etc 2020-11-19 18:06:20 +01:00
Olof hagsand
9016dde319 Modify tests so they all work with evhtp and backend config, eg that CLICON_RESTCONF_CONFIG is default true.
Note that some tests are forced to CLICON_RESTCONF_CONFIG is false due to test complexities.
2020-11-18 13:57:25 +01:00
Olof hagsand
794d51a365 * Top-level default leafs assigned.
* Enforcing RFC 7950 Sec 7.6.1 means unassigned top-level leafs (or leafs under non-presence containers) are assigned default values.
* NACM default behaviour is read-only (empty configs are dead-lockedd)
  * This applies if NACM is loaded and `CLICON_NACM_MODE` is `internal`
* Fixed: [default values don't show up in datastores #111](https://github.com/clicon/clixon/issues/111)
2020-08-06 15:20:57 +02:00
Olof hagsand
5dc2a6387d unified curl test options 2020-07-03 14:13:15 +02:00
Olof hagsand
3af184c655 Modify restconf tests to run for both evhtp and fcgi: , curl -sik, etc 2020-06-21 15:49:46 +02:00
Olof hagsand
6e714beea5 * Restconf FCGI (eg via nginx) have changed reply message syntax slightly
* native http: new restconf_err files, generic data input, restconf_methods generalized.
* test: expecteq removed.
2020-06-16 21:51:00 +02:00
Olof hagsand
1c99bd6a9b * xml-stats moved from clixon-config.yang as state data to an rpc datastatsin clixon-lib.yang
* Two new plugin callbacks added
  * ca_daemon: Called just after a server has "daemonized", ie put in background.
  * ca_trans_commit_done: Called when all plugin commits have been done.
2020-04-23 22:58:57 +02:00
Olof hagsand
662d6b0a3f Clixon 4.4 three compile-time options: MOVE_TRANS_END for changing transaction-end callbacks; STATE_ORDERED_BY_SYSTEM to make state data system ordered; XML_MERGE_TWO_ROUNDS to improve xml_merge performance. 2020-04-02 14:28:58 +02:00
Olof hagsand
adbb683329 C-API:
* All uses of `api_path2xpath_cvv()` should be replaced by `api_path2xpath()`
  * `api_path2xpath()` added an `xerr` argument.
2020-03-10 22:37:16 +01:00
Olof hagsand
19e21be0bc Very large commit for upcoming 4.4 release
Major New features

* New and updated search functions using xpath, api-path and instance-id
  * New search functions using api-path and instance_id:
    * C search functions: `clixon_find_instance_id()` and `clixon_find_api_path()`
  * Binary search optimization in lists for indexed leafs in all three formats.
    * This improves search performance to O(logN) which is drastical improvements for large lists.
  * You can also register explicit indexes for making binary search (not only list keys)
  * For more info, see docs at [paths](https://clixon-docs.readthedocs.io/en/latest/paths.html) and
[search](https://clixon-docs.readthedocs.io/en/latest/xml.html#searching-in-xml)

API changes on existing features (you may need to change your code)
* On failed validation of leafrefs, error message changed from: `No such leaf` to `No leaf <name> matching path <path>`.
* CLI Error message (clicon_rpc_generate_error()) changed when backend returns netconf error to be more descriptive:
  * Original: `Config error: Validate failed. Edit and try again or discard changes: Invalid argument`
  * New (example): `Netconf error: application operation-failed Identityref validation failed, undefined not derived from acl-base . Validate failed. Edit and try again or discard changes"

Minor changes

* Test framework
  * Added `-- -S <file>` command-line to main example to be able to return any state to main example.
  * Added `test/cicd` test scripts for running on a set of other hosts
* C-code restructuring
  * clixon_yang.c partitioned and moved code into clixon_yang_parse_lib.c and clixon_yang_module.c and move back some code from clixon_yang_type.c.
    * partly to reduce size, but most important to limit code that accesses internal yang structures, only clixon_yang.c does this now.
2020-02-02 15:52:30 +01:00
Olof hagsand
05ce6f38a3 test kill valgrind process after run 2019-12-09 21:12:39 +01:00
Olof Hagsand
82356b7450 test scripts: separated wait of backend and restconf starts 2019-10-27 15:54:16 +00:00
Olof hagsand
279614d64f * The backend socket has now support of credentials of peer clients
* Added: CLICON_NACM_CREDENTIALS and CLICON_NACM_RECOVERY_USER
2019-10-18 19:33:23 +02:00
Olof Hagsand
734f65e25a freebsd test script modifications 2019-10-13 07:08:53 -10:00
Olof hagsand
fe1aeda4f2 test env bash; freebsd config 2019-09-17 22:44:01 +02:00
Olof hagsand
27fd99e7cd * Implemented backend daemon drop privileges after initialization to
run as non-privileged user
2019-09-14 18:34:32 +02:00
Olof hagsand
3d5abb77f9 * Backend daemon drops privileges after initialization (to not run as root)
* New config option `CLICON_USER` with default value `clicon`
  * Can also be set with `-U <user>` clixon_backend command-line option
2019-09-11 21:24:14 +02:00
Olof hagsand
aa14f8ac2c RESTCONF PUT/POST -d {} media is enforced 2019-08-06 15:47:46 +02:00
Olof hagsand
e7b60619da * Pushed tag to 4.0.1.PRE
* Restconf RFC 8040 increased feature compliance
  * Cache-Control: no-cache added in HTTP responses (RFC Section 5.5)
  * Restconf monitoring capabilities (RFC Section 9.1)
* Added support for Yang extensions
  * New plugin callback: ca_extension
  * Main backend example includes example code on how to implement a Yang extension in a plugin.
* JSON changes
  * Non-pretty-print output removed all extra spaces.
    * Example: `{"nacm-example:x": 42}` --> {"nacm-example:x":42}`
  * Empty JSON container changed from `null` to `{}`.
    * Empty list and leafs remain as `null`
* Removed unnecessary configure dependencies
  * libnsl, libcrypt, libm, if_vlan,...
* pseudo-plugin added, to enable callbacks also for main programs. Useful for extensions
* Yang Unique statements with multiple schema identifiers did not work on some platforms due to memory error.
2019-07-23 22:11:14 +02:00
Olof hagsand
67b8685bab The Clixon API has been extended with namespaces, or namespace contexts in the following cases:
* CLIspec functions have added namespace parameter:
    * `cli_show_config <db> <format> <xpath>` --> `cli_show_config <db> <format> <xpath> <namespace>`
    * `cli_copy_config <db> <xpath> ...` --> `cli_copy_config <db> <xpath> <namespace> ...`
  * Xpath API
    * `xpath_first(x, format, ...)` --> `xpath_first(x, nsc, format, ...)`
    * `xpath_vec(x, format, vec, veclen, ...)` --> `xpath_vec(x, nsc, format, vec, veclen, ...)`
    * `xpath_vec_flag(x, format, flags, vec, veclen, ...)` --> `xpath_vec_flag(x, format, flags, vec, veclen, ...)`
    * `xpath_vec_bool(x, format, ...)` --> `xpath_vec_bool(x, nsc, format, ...)`
    * `xpath_vec_ctx(x, xpath, xp)` --> `xpath_vec_ctx(x, nsc, xpath, xp)`
  * xmldb_get0 has an added `nsc` parameter:
    * `xmldb_get0(h, db, xpath, copy, xret, msd)` --> `xmldb_get0(h, db, nsc, xpath, copy, xret, msd)`
  * The plugin statedata callback (ca_statedata) has been extended with an nsc parameter:
    * `int example_statedata(clicon_handle h, cvec *nsc, char *xpath, cxobj *xstate);`
  * rpc get and get-config api function has an added namespace argument:
    * `clicon_rpc_get_config(clicon_handle h, char *db, char *xpath, char *namespace, cxobj **xt);`
    * `int clicon_rpc_get(clicon_handle h, char *xpath, char *namespace, cxobj **xt);`
2019-07-08 10:36:37 +02:00
Olof hagsand
98cc62eace Fixed a problem with some netconf error messages caused restconf daemon to exit due to no XML encoding 2019-06-12 18:15:44 +02:00
Olof hagsand
de15b2bf80 * RESTCONF strict namespace validation of data in POST and PUT.
* Accepted:
  ```
    curl -X PUT http://localhost/restconf/data/mod:a -d {"mod:a":"x"}
  ```
  * Not accepted (must prefix "a" with module):
  ```
    curl -X PUT http://localhost/restconf/data/mod:a -d {"a":"x"}
  ```
  * Undefine `RESTCONF_NS_DATA_CHECK` in include/clixon_custom.h to disable strict check.
2019-06-10 16:15:02 +02:00
Olof hagsand
bc54f2d04c * Regexp improvements
* Added check for libxml in configure';
  * Added clixon_util_regexp utility function
* Yang state get improvements
  * Integrated state and config into same tree on retrieval, not separate trees
  * Added cli functions `cli_show_config_state()` and `cli_show_auto_state()` for showing combined config and state info.
  * Added integrated state in the main example: `interface/oper-state`.
  * Added performance tests for getting state, see [test/test_perf_state.sh].
2019-05-20 16:03:29 +02:00
Olof hagsand
cf7675dfac Experimental customized error output strings, see [lib/clixon/clixon_err_string.h] 2019-04-11 12:45:31 +02:00
Olof hagsand
e29cd7cfb9 * Optimized validation by making xml_diff work on raw cache tree (not copies)
* xmldb_get() removed unnecessary config option
2019-04-07 15:55:53 +02:00
Olof hagsand
606245ef02 * Added flags to example backend to control its behaviour:
* Start with `-- -r` to run the reset plugin
  * Start with `-- -s` to run the state callback
* Rewrote yang dir load algorithm to follow the algorithm in [FAQ](FAQ(doc/FAQ.md#how-are-yang-files-found) with more precise timestamp checks, etc.
2019-03-18 16:31:34 +01:00
Olof hagsand
98fb6eceef * Clixon configuration file top-level symbols has changed to clixon-configand namespace check is enforced.
* clixon-config YAML file has new revision: 2019-03-05.
  * New URN and changed top-level symbol to `clixon-config`
* Removed obsolete `_CLICON_XML_NS_STRICT` variable and `CLICON_XML_NS_STRICT` config option.
2019-03-06 09:42:55 +01:00
Olof hagsand
83edd29678 Revert "Revert "* Added valgrind memory leak tests in testmem.sh for restconf""
This reverts commit a73d2bd242.
2019-02-20 16:25:42 +01:00
Olof hagsand
a73d2bd242 Revert "* Added valgrind memory leak tests in testmem.sh for restconf"
String error
This reverts commit 6e63f0a77b.
2019-02-20 16:19:07 +01:00
Olof Hagsand
6e63f0a77b * Added valgrind memory leak tests in testmem.sh for restconf
* Added clicon_socket_set() and clicon_socket_get() functions for cleaning up backend server and restconf FCGI socket on termination.
* Removed obsolete `-x` command-line option to clixon_cli.
2019-02-20 13:06:18 +01:00
Olof hagsand
4c0b412e9f Added valgrind memory leak tests for backend. Fixed some minor leaks and file descriptor closes. 2019-02-19 13:16:59 +01:00
Olof Hagsand
d64d433468 * Changed base OS in clixon-system container to Alpine
* Moved datastore/text/ code to datastore/ since there is only one type of datastore.
* Added "magic script line" to test script for sourced and scriped exits
2019-02-16 17:45:39 +01:00
Olof Hagsand
860c0c042d Added RCLOG for redirecting restconf logs at tests; with-restconf error in example 2019-02-12 13:19:52 +01:00
Olof Hagsand
2b596591f1 Added Clixon example full system docker container, see [docker/system]. 2019-02-07 22:23:58 +01:00
Olof hagsand
02d725b2c0 * xml_cmp() respects 'ordered-by user' for state nodes, which violates RFC 7950 [https://github.com/clicon/clixon/issues/63. (Thanks JDL) 2019-02-03 16:19:33 +01:00
Olof hagsand
936ae389e8 Test script change: merged new2+new functions 2019-02-02 12:20:47 +01:00
Olof hagsand
1e4022e73c NACM cleanup, uniform rule function, change of function names, etc. 2019-02-02 11:35:50 +01:00
Olof hagsand
8bf5cb0de5 NACM module access control point for edit-config 2019-02-01 14:56:45 +01:00
Olof hagsand
ffecebf32a * NACM Data node READ access module support (RFC8341 3.4.5)
* Access control points added for `get` and `get-config` in addition to incoming rpc.
   * RFC 8341 Example A.2 implemented, see: [test/test_nacm_module.sh]
* Added `username` argument on `xmldb_put()` datastore function for NACM data-node write checks
* Added `xml_rootchild_node()` lib function as variant of `xml_rootchild()`
2019-01-27 13:26:15 +01:00
Olof hagsand
04bb05c83f Upgrade testcases for nacm 2019-01-22 13:41:03 +01:00
Olof hagsand
d55787d8ec Merge branch 'develop' into nacm 2019-01-21 16:45:24 +01:00
Olof hagsand
0267afcb21 * Added -o "<option>=<value>" command-line option to all programs: backend, cli, netconf, restconf.
* Added -p <dir> command-line option to all programs: backend, cli, netconf, restconf.
* Moved and updated all standard ietf and iana yang files from example and yang/ to `yang/standard`.
* Renamed example yang from example.yang -> clixon-example.yang
2019-01-13 17:34:15 +01:00
Olof hagsand
90545b05cd nacm testcases for modules and datastore 2019-01-12 13:52:35 +01:00
Olof hagsand
5b6af82e29 Validation of mandatory choice and recursive mandatory containers. 2019-01-05 11:08:16 +01:00
Olof hagsand
32ea957a52 CLICON_XML_SORT option (in clixon-config.yang) has been removed and set to true permanently since setting it to false is obsolete. 2019-01-02 17:43:58 +01:00
Olof hagsand
0baebc93fd * Strict namespace setting can be a problem when upgrading existing database files, such as startup-db or persistent running-db, or any other saved XML file.
* For backward compatibility, load of startup and running set CLICON_XML_NS_STRICT to false temporarily.
* Added three-valued return values for several validate functions where -1 is fatal error, 0 is validation failed and 1 is validation OK.
  * This includes: `xmldb_put`, `xml_yang_validate_all`, `xml_yang_validate_add`, `xml_yang_validate_rpc`, `api_path2xml`, `api_path2xpath`
* Added new xml functions for specific types: `xml_child_nr_notype`, `xml_child_nr_notype`, `xml_child_i_type`, `xml_find_type`.
2019-01-02 15:18:29 +01:00
Olof hagsand
861300d6c0 netconf error handling and test summary script 2018-12-21 14:44:59 +01:00
Olof hagsand
f872c7e295 * More precise Yang validation and better error messages
* For Example, adding bad-, missing-, or unknown-element error messages, etc instead of operation-failed
* Removed delete-config support for candidate db since it is not supported in RFC6241.
* Switched the order of `error-type` and `error-tag` in all netconf and restconf error messages to comply to RFC order.
* Added example_rpc RPC to example backend
* Renamed xml_namespace[_set]() to xml_prefix[_set]()
* Some restconf error messages contained "rpc-reply" or "rpc-error" which have now been removed.
* Netconf/Restconf RPC extra input arguments are ignored (https://github.com/clicon/clixon/issues/47)
2018-12-21 01:33:41 +01:00
Olof hagsand
ae1af8da9e * NACM extension (RFC8341)
* NACM module support (RFC8341 A1+A2)
   * Recovery user "_nacm_recovery" added.
     * Example use is restconf PUT when NACM edit-config is permitted, then automatic commit and discard are permitted using recovery user.
   * Example user changed adm1 to andy to comply with RFC8341 example

 * Yang code upgrade (RFC7950)
   * RPC method input parameters validated
     * see https://github.com/clicon/clixon/issues/4
* Correct XML namespace handling
   * XML multiple modules was based on "loose" semantics so that yang modules were found by iterating thorugh namespaces until a match was made. This did not adhere to proper [XML namespace handling](https://www.w3.org/TR/2009/REC-xml-names-20091208), and causes problems with overlapping names and false positives. Below see XML accepted (but wrong), and correct namespace declaration:
```
      <rpc><my-own-method></rpc> # Wrong but accepted
      <rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> # Correct
        <my-own-method xmlns="http://example.net/me/my-own/1.0">
      </rpc>
```
   * To keep old loose semantics set config option CLICON_XML_NS_ITERATE (true by default)
   * XML to JSON translator support for mapping xmlns attribute to module name prefix.
   * Default namespace is still "urn:ietf:params:xml:ns:netconf:base:1.0"
   * See https://github.com/clicon/clixon/issues/49
* Changed all make tags --> make TAGS
* Keyvalue datastore removed (it has been disabled since 3.3.3)
* debug rpc added in example application (should be in clixon-config).
2018-12-16 19:49:19 +01:00
Olof hagsand
39a5086218 * Yang Support of submodule, include and belongs-to.
* Improved unknown handling
* Configure option `CLICON_YANG_DIR` is changed from a single directory to a path of directories
    * Note CLIXON_DATADIR (=/usr/local/share/clixon) need to be in the list
2018-12-01 18:17:42 +01:00