Commit graph

59 commits

Author SHA1 Message Date
Olof hagsand
a674af6f2c Cleaning up code for xml insignificant whitespace removal
Experimenal explicit index search code
2020-02-07 14:59:57 +01:00
Olof hagsand
19e21be0bc Very large commit for upcoming 4.4 release
Major New features

* New and updated search functions using xpath, api-path and instance-id
  * New search functions using api-path and instance_id:
    * C search functions: `clixon_find_instance_id()` and `clixon_find_api_path()`
  * Binary search optimization in lists for indexed leafs in all three formats.
    * This improves search performance to O(logN) which is drastical improvements for large lists.
  * You can also register explicit indexes for making binary search (not only list keys)
  * For more info, see docs at [paths](https://clixon-docs.readthedocs.io/en/latest/paths.html) and
[search](https://clixon-docs.readthedocs.io/en/latest/xml.html#searching-in-xml)

API changes on existing features (you may need to change your code)
* On failed validation of leafrefs, error message changed from: `No such leaf` to `No leaf <name> matching path <path>`.
* CLI Error message (clicon_rpc_generate_error()) changed when backend returns netconf error to be more descriptive:
  * Original: `Config error: Validate failed. Edit and try again or discard changes: Invalid argument`
  * New (example): `Netconf error: application operation-failed Identityref validation failed, undefined not derived from acl-base . Validate failed. Edit and try again or discard changes"

Minor changes

* Test framework
  * Added `-- -S <file>` command-line to main example to be able to return any state to main example.
  * Added `test/cicd` test scripts for running on a set of other hosts
* C-code restructuring
  * clixon_yang.c partitioned and moved code into clixon_yang_parse_lib.c and clixon_yang_module.c and move back some code from clixon_yang_type.c.
    * partly to reduce size, but most important to limit code that accesses internal yang structures, only clixon_yang.c does this now.
2020-02-02 15:52:30 +01:00
Olof hagsand
2d9d204f69 * RESTCONF "insert" and "point" query parameters supported
* Yang Netconf leaf/leaf-list insert support
  * For "ordered-by user" leafs and leaf-lists, the insert and value/key attributes are supported according to RFC7950 Sections 7.7.9 and 7.8.6

* Fixed RESTCONF api-path leaf-list selection was not made properly
2019-07-31 14:04:27 +02:00
Olof hagsand
70ebfa4d80 * Identity/identityref mapped between XML and JSON 2019-07-28 18:09:55 +02:00
Olof hagsand
e7b60619da * Pushed tag to 4.0.1.PRE
* Restconf RFC 8040 increased feature compliance
  * Cache-Control: no-cache added in HTTP responses (RFC Section 5.5)
  * Restconf monitoring capabilities (RFC Section 9.1)
* Added support for Yang extensions
  * New plugin callback: ca_extension
  * Main backend example includes example code on how to implement a Yang extension in a plugin.
* JSON changes
  * Non-pretty-print output removed all extra spaces.
    * Example: `{"nacm-example:x": 42}` --> {"nacm-example:x":42}`
  * Empty JSON container changed from `null` to `{}`.
    * Empty list and leafs remain as `null`
* Removed unnecessary configure dependencies
  * libnsl, libcrypt, libm, if_vlan,...
* pseudo-plugin added, to enable callbacks also for main programs. Useful for extensions
* Yang Unique statements with multiple schema identifiers did not work on some platforms due to memory error.
2019-07-23 22:11:14 +02:00
Olof hagsand
e44685a100 Replaced strchr with nodeid_split
Experimental code for replacing identityref lists with module:id instead of prefix:id
2019-07-10 17:18:34 +02:00
Olof hagsand
67b8685bab The Clixon API has been extended with namespaces, or namespace contexts in the following cases:
* CLIspec functions have added namespace parameter:
    * `cli_show_config <db> <format> <xpath>` --> `cli_show_config <db> <format> <xpath> <namespace>`
    * `cli_copy_config <db> <xpath> ...` --> `cli_copy_config <db> <xpath> <namespace> ...`
  * Xpath API
    * `xpath_first(x, format, ...)` --> `xpath_first(x, nsc, format, ...)`
    * `xpath_vec(x, format, vec, veclen, ...)` --> `xpath_vec(x, nsc, format, vec, veclen, ...)`
    * `xpath_vec_flag(x, format, flags, vec, veclen, ...)` --> `xpath_vec_flag(x, format, flags, vec, veclen, ...)`
    * `xpath_vec_bool(x, format, ...)` --> `xpath_vec_bool(x, nsc, format, ...)`
    * `xpath_vec_ctx(x, xpath, xp)` --> `xpath_vec_ctx(x, nsc, xpath, xp)`
  * xmldb_get0 has an added `nsc` parameter:
    * `xmldb_get0(h, db, xpath, copy, xret, msd)` --> `xmldb_get0(h, db, nsc, xpath, copy, xret, msd)`
  * The plugin statedata callback (ca_statedata) has been extended with an nsc parameter:
    * `int example_statedata(clicon_handle h, cvec *nsc, char *xpath, cxobj *xstate);`
  * rpc get and get-config api function has an added namespace argument:
    * `clicon_rpc_get_config(clicon_handle h, char *db, char *xpath, char *namespace, cxobj **xt);`
    * `int clicon_rpc_get(clicon_handle h, char *xpath, char *namespace, cxobj **xt);`
2019-07-08 10:36:37 +02:00
Olof hagsand
2aeb925521 * JSON parse and print improvements
* Integrated parsing with namespace translation and yang spec lookup
2019-06-03 16:40:54 +02:00
Olof hagsand
2fe185d683 * Support for multiple patterns as described in RFC7950 Section 9.4.7
* Added regex cache to type resolution
* Added compiled regexp parameter as part of internal yang type resolution functions
* All internal `ys_populate_*()` functions (except ys_populate()) have switched parameters: `clicon_handle, yang_stmt *)`
2019-05-29 11:39:09 +02:00
Olof hagsand
d775eb5374 * Improved submodule implementation, as part of [Yang submodule import prefix restrictions #60](https://github.com/clicon/clixon/issues/60). 2019-05-14 14:09:02 +02:00
Olof hagsand
6bf2a74e24 * Restconf with startup feature will now copy all edit changes to startup db (as it should according to RFC 8040)
* See [Restconf does not handle startup datastore according to the RFC](https://github.com/clicon/clixon/issues/74)
* Netconf Startup feature is no longer hardcoded, you need to explicitly enable it (See RFC 6241, Section 8.7)
  * Enable in config file with: `<CLICON_FEATURE>ietf-netconf:startup</CLICON_FEATURE>`, or use `*:*`
2019-04-26 12:12:55 +02:00
Olof hagsand
e4239496a8 Yang access function. Mem leak in USE_XML_INSERT 2019-04-25 13:46:23 +02:00
Olof hagsand
16de5f47ba Merge branch 'master' of https://github.com/clicon/clixon 2019-04-24 15:35:57 +02:00
Olof hagsand
65c809b1c3 Removed external direct access to the yang_stmt struct. 2019-04-24 15:13:19 +02:00
Olof hagsand
7847e74c5e yang_cvec_set(y, cvv) added 2019-04-24 09:56:07 +02:00
Olof hagsand
8c36083e16 * A yang type regex cache added, this helps the performance by avoiding re-running the regcomp command on every iteration.
* An XML namespace cache added (see `xml2ns()`)
* Better performance of XML whitespace parsing/scanning.
2019-04-19 16:01:39 +02:00
Olof hagsand
cef0dc5a22 * Added yang access functions
* Change all y->ys_parent to yang_parent_get(y)
    * Change all y->ys_keyword to yang_keyword_get(y)
    * Change all y->ys_argument to yang_argument_get(y)
    * Change all y->ys_cv to yang_cv_get(y)
    * Change all y->ys_cvec to yang_cvec_get(y)
2019-04-11 15:53:25 +02:00
Olof hagsand
b5f1f7346e Structural C-code change: Merged yang_spec and yang_node types into yang_stmt 2019-04-02 12:27:33 +02:00
Olof hagsand
3f68cca06c * New yang changelog experimental feature for automatic upgrade
* Added modules-state diff parameter to xmldb_get datastore function for startup scenarios.
* Allowed Yang extended Xpath functions (syntax only):
  * re-match, deref, derived-from, derived-from-or-self, enum-value, bit-is-set
* XSD regular expression handling of dash(`-`)
  *: Translate XDS `[xxx\-yyy]` to POSIX `[xxxyyy-]`.
* YANG Anydata treated same as Anyxml
2019-03-21 18:05:55 +01:00
Olof hagsand
01c8b0b836 * [Identityref inside augment statement](https://github.com/clicon/clixon/issues/77)
* Yang-stmt enhanced with "shortcut" to original module
2019-03-05 22:42:13 +01:00
Olof hagsand
b182e8666f [yang type range statement does not support multiple values](https://github.com/clicon/clixon/issues/59)
[Multiple ranges support](https://github.com/clicon/clixon/issues/78)
Restructured "cb" parameter in cli_generate.c
2019-03-04 14:36:43 +01:00
Olof Hagsand
f035c3ca23 * Removed all dependency on strverscmp
* Added libgen.h for baseline()
2019-02-27 10:46:05 +01:00
Olof hagsand
207858e20d * Support of yangmodels supported, see test_yangmodels.sh
* Added -o "<option>=<value>" command-line option to all programs: backend, cli, netconf, restconf.
* Ignore CR(\r) in yang files for DOS files
2019-01-10 20:52:19 +01:00
Olof hagsand
c7e847cd24 Keyword "min" (not only "max") can be used in built-in types "range" and "length" statements. 2019-01-06 22:11:59 +01:00
Olof hagsand
058a14579f Yang choice functionality improved and stricter validation for CLI generation, mandatory flags, etc. 2019-01-04 11:37:26 +01:00
Olof hagsand
0103d58994 2019 2019-01-02 15:48:30 +01:00
Olof hagsand
0baebc93fd * Strict namespace setting can be a problem when upgrading existing database files, such as startup-db or persistent running-db, or any other saved XML file.
* For backward compatibility, load of startup and running set CLICON_XML_NS_STRICT to false temporarily.
* Added three-valued return values for several validate functions where -1 is fatal error, 0 is validation failed and 1 is validation OK.
  * This includes: `xmldb_put`, `xml_yang_validate_all`, `xml_yang_validate_add`, `xml_yang_validate_rpc`, `api_path2xml`, `api_path2xpath`
* Added new xml functions for specific types: `xml_child_nr_notype`, `xml_child_nr_notype`, `xml_child_i_type`, `xml_find_type`.
2019-01-02 15:18:29 +01:00
Olof hagsand
ae1af8da9e * NACM extension (RFC8341)
* NACM module support (RFC8341 A1+A2)
   * Recovery user "_nacm_recovery" added.
     * Example use is restconf PUT when NACM edit-config is permitted, then automatic commit and discard are permitted using recovery user.
   * Example user changed adm1 to andy to comply with RFC8341 example

 * Yang code upgrade (RFC7950)
   * RPC method input parameters validated
     * see https://github.com/clicon/clixon/issues/4
* Correct XML namespace handling
   * XML multiple modules was based on "loose" semantics so that yang modules were found by iterating thorugh namespaces until a match was made. This did not adhere to proper [XML namespace handling](https://www.w3.org/TR/2009/REC-xml-names-20091208), and causes problems with overlapping names and false positives. Below see XML accepted (but wrong), and correct namespace declaration:
```
      <rpc><my-own-method></rpc> # Wrong but accepted
      <rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> # Correct
        <my-own-method xmlns="http://example.net/me/my-own/1.0">
      </rpc>
```
   * To keep old loose semantics set config option CLICON_XML_NS_ITERATE (true by default)
   * XML to JSON translator support for mapping xmlns attribute to module name prefix.
   * Default namespace is still "urn:ietf:params:xml:ns:netconf:base:1.0"
   * See https://github.com/clicon/clixon/issues/49
* Changed all make tags --> make TAGS
* Keyvalue datastore removed (it has been disabled since 3.3.3)
* debug rpc added in example application (should be in clixon-config).
2018-12-16 19:49:19 +01:00
Olof hagsand
ac1aa44fc4 * Yang Configure options changed
* `CLICON_YANG_DIR` is changed from a single directory to a path of directories
    * Note CLIXON_DATADIR (=/usr/local/share/clixon) need to be in the list
  * CLICON_YANG_MAIN_FILE Provides a filename with a single module filename.
  * CLICON_YANG_MAIN_DIR Provides a directory where all yang modules should be loaded.
* Change all @datamodel:tree to @datamodel in all CLI specification files
  * If you generate CLI code from the model (CLIXON_CLI_GENMODEL).
  * For backward compatibility, define CLICON_CLI_MODEL_TREENAME_PATCH in clixon_custom.h
* Removed return value ymodp from yang parse functions (eg yang_parse()).
* New config option: CLICON_CLI_MODEL_TREENAME defining name of generated syntax tree if CLIXON_CLI_GENMODEL is set.
2018-12-03 21:16:35 +01:00
Olof hagsand
39a5086218 * Yang Support of submodule, include and belongs-to.
* Improved unknown handling
* Configure option `CLICON_YANG_DIR` is changed from a single directory to a path of directories
    * Note CLIXON_DATADIR (=/usr/local/share/clixon) need to be in the list
2018-12-01 18:17:42 +01:00
Olof hagsand
a8f0aad411 * Yang parser is stricter (see cardinality below) which may break parsing of slack yang specs.
* YANG parser cardinality checked (only modules level yet)
  * See https://github.com/clicon/clixon/issues/48
2018-11-18 20:55:57 +01:00
Olof hagsand
a4e29bcdb7 * All yang modules are stored in the clicon_dbspec_yang() option.
* Except clixon-config module due to bug reported below
* Bug: Top-level Yang symbol cannot be called "config" in any imported yang file.
  * datastore uses "config" as reserved keyword for storing any XML whoich collides with code for detecting Yang sanit
2018-10-19 15:26:22 +02:00
Olof Hagsand
a23c1ec68d Added CLIXON_DEFAULT_CONFIG=/usr/local/etc/clixon.xml as option and in exampl\
e (so you dont need to provide -f command-line option).
2018-10-11 19:03:35 +00:00
Olof hagsand
507e03742d Yang 1.1 action syntax added (but function is not supported) 2018-10-10 22:27:46 +02:00
Olof hagsand
ea77e7f02d * YANG Features
* Yang 1.1 feature and if-feature according to RFC 7950 7.20.1 and 7.20.2.
  * See https://github.com/clicon/clixon/issues/41
  * Features are declared via CLICON_FEATURE in the configuration file.
  * logical combination of features not implemented, eg if-feature "not foo or
* Identity without any identityref:s caused SEGV
2018-10-10 20:11:20 +02:00
Olof hagsand
acb8748470 * Generilized top-level yang parsing functions
* Clarified semantics of main yang module:
    * -y option to commands MUST specify filename
    * CLICON_YANG_MODULE_MAIN MUST specify a module
    * yang_parse() changed to take either filename or module name and revision.
  * Removed clicon_dbspec_name[_set]().
    * Use yang_main_module_name() instead.
  * Replaced yang_spec_main with yang_spec_parse_module
  * Added yang_spec_parse_file
2018-10-07 11:04:33 +02:00
Olof hagsand
0631be19cb * Notification event streams enhancements
* Yang 1.1 notification support
  * Event stream discovery support according to RFC 5277 Sec 3.2.5.1 (netconf) and RFC 8040 (restconf)
2018-09-18 22:49:29 +02:00
Olof Hagsand
de2d5be6dd export populate 2018-08-27 19:07:52 +00:00
Olof hagsand
719ea93398 Conformance to RFC-8040 operations where prefix was used instead of module name.
* Proper specification for an operation is POST /restconf/operations/<module_name>:<rpc_procedure> HTTP/1.1
  * See https://github.com/clicon/clixon/issues/31, https://github.com/clicon/clixon/pull/32 and https://github.com/clicon/clixon/issues/30
  * Thanks David Cornejo and Dmitry Vakhrushev of Netgate for pointing this out
2018-07-13 16:31:39 +02:00
Olof hagsand
85c4782e36 Dedicated xml,json,yang and xsl parser utility programs added
Sanity check of stdarg (...) added
Cleanup of error messages.
2018-06-17 19:40:06 +02:00
Olof hagsand
7e4e1d6deb * Support for YANG identity and identityref according to RFC 7950 Sec 7.18 and 9.10
* Previous support did no validation of values.
  * Validation of types and CLI expansion
  * Example extended with inclusion of iana-if-type RFC 7224 interface identities
2018-06-03 15:36:05 +02:00
Olof Hagsand
3df4a8a190 Added keyword arg to find schemanode functions 2018-04-27 19:15:11 +02:00
Olof hagsand
1913407e52 Replacing remaining badrequest() with proper restconf error msg. 2018-04-22 20:13:54 +02:00
Olof hagsand
990700b68d https://github.com/clicon/clixon/issues/16
Added restconf/operations get, see RFC8040 Sec 3.3.2:
2018-03-04 21:24:38 +01:00
Olof hagsand
35b142cf4d * Added Yang "extension" statement. This includes parsing unknown
statements and identifying them as extensions or not. However,
  semantics for specific extensions must still be added.
* Renamed ytype_id and ytype_prefix to yarg_id and yarg_prefix, respectively

* Added cli_show_version()
2018-02-25 20:01:53 +01:00
Olof hagsand
38f0b446fa 2017->2018 2018-01-01 12:25:33 +01:00
Olof hagsand
4b92dbdc10 Optimized search performance for large lists by sorting and binary search 2017-12-27 11:34:47 +01:00
Olof hagsand
056b5c97dd experimental netconf yang spec 2017-12-08 19:37:09 +01:00
Olof hagsand
fd91bb2933 restconf RPC 2017-07-30 16:16:10 +02:00
Olof hagsand
d0660bf611 Added xpath support for predicate: current(); added by-reference return for schema_nodeid functions; prototype enum2int function 2017-07-30 11:00:02 +02:00