Commit graph

171 commits

Author SHA1 Message Date
Olof hagsand
dd8883420c * Made backend transaction and commit/validate API available to plugin code.
* This enables that RPOC handles can call commit and validate via lib
  * The commit/validate API is now: `candidate_validate()` and `candidate_commit()`
2021-06-17 19:20:27 +02:00
Olof hagsand
b40dafb691 * Event exit API changed to a single decrementing counter where 1 means exit.
* Removed: `clicon_exit_reset()`
  * Changed: `clicon_exit_set()` --> `clixon_exit_set(int nr)`
  * Changed: `clicon_exit_get()` --> `clixon_exit_get()`
* native restconf special case upgrade handling from http/1 -> http/2 and restconf restart: delay the restart using event exit counter
2021-06-16 11:58:13 +02:00
Olof hagsand
e0c3f5467c * Added new startup-mode: running-startup: First try running db, if it is empty try startup db.
* See [Can startup mode to be extended to support running-startup mode? #234](https://github.com/clicon/clixon/issues/234)
* Improved error message on failed MUST condition
2021-05-27 13:18:01 +02:00
Olof hagsand
c20c672d83 * Changed config and install options for Restconf
* clixon_restconf daemon is installed in /usr/local/sbin (as clixon_backend), instead of /www-data
    * `configure --with-wwwdir=<dir>` remains but only applies to fcgi socket and log
    * New option `CLICON_RESTCONF_INSTALL_DIR` is set to where clixon_restconf is installed, with default `/usr/local/sbin/`
  * Restconf drop privileges user is defined by `CLICON_RESTCONF_USER`
    * `configure --with-wwwuser=<user>` is removed
  * clixon_restconf drop of privileges is defined by `CLICON_RESTCONF_PRIVILEGES` option
* New clixon-restconf@2020-05-20.yang revision
  * Added: restconf `log-destination`
2021-05-23 17:14:18 +02:00
Olof hagsand
05009aed67 Add restconf get list extra =restval test 2021-04-28 11:31:41 +02:00
Olof hagsand
b88722fa25 - Restructure plugin module using a "module struct" rather than global variables.
This include plugin module init and exit functions
  New type: clixon_plugin_t exposed via public API while struct clixon_plugin is private
- Makefile changes for static linkage using "LINKAGE=static"
- Moved nacm external init to later stage in backend_main
2021-04-19 11:06:40 +02:00
Olof hagsand
8469a0962e * Add multiple yang support also for obsolete versions
* This means that files and datastores supporting modstate also looks for deleted or updated yang modules.
  * A stricter binding which gives error if loading outdated YANG file does not exist.
* Stricter yang checks: you cannot do get-config on datastores that have obsolete YANG
* Added xerr output parameter to `xmldb_get0()`
2021-04-12 12:42:26 +02:00
Olof hagsand
477059f33d Fixed again: [backend start resconf failed due to path string truncated #192](https://github.com/clicon/clixon/issues/192) 2021-03-24 12:23:00 +01:00
Olof hagsand
43db8dd5c5 Fixed: [state showing error in cli with CLICON_STREAM_DISCOVERY_RFC8040 #191](https://github.com/clicon/clixon/issues/191) 2021-03-18 10:41:08 +01:00
Olof hagsand
afdf6ffcf8 Fixed: [state showing error in cli with CLICON_STREAM_DISCOVERY_RFC8040 #191](https://github.com/clicon/clixon/issues/191) 2021-03-18 09:43:13 +01:00
Olof hagsand
fb9917e81e * Removed endtag parameter of clixon_xml_parse_file() 2021-03-15 13:24:27 +01:00
Olof hagsand
7762b10cbb * Changed signal handling
* Moved clixon-proc sigchild handling	from handler to clixon_events
2021-03-11 18:12:43 +01:00
Olof hagsand
b2f9c59a34 * Reverted blocked signal behavior introduced in 5.0.
* Introduced a delay before making process start/stop/restart processes for race conditions when configuring eg restconf
* For restconf `CLICON_BACKEND_RESTCONF_PROCESS`, restart restconf if restconf is edited.
2021-03-02 12:24:07 +01:00
Olof hagsand
0fe8acd301 wait.h->sys/wait.h, restconf_rpc wait in test 2021-02-26 11:50:15 +01:00
Olof hagsand
d5edd64257 If a signal handler runs during select() loop in clixon_event_loop() and unless the signal handler sets clixon_exit, the select will be restarted. 2021-02-25 22:35:13 +01:00
Olof hagsand
9840e248c6 * Augment target node check strict, instead of printing a warning, it will terminate with error.
* Fixed: [Augment that reference a submodule as target node fails #178](https://github.com/clicon/clixon/issues/178)
* Fixed a memory error that was reported in slack by Pawel Maslanka
  * The crash printout was: `realloc(): invalid next size Aborted`
2021-02-19 15:46:55 +01:00
Olof hagsand
710fc76887 * Restconf authentication callback (ca_auth) signature changed
* Not backward compatible: All uses of the ca-auth callback in restconf plugins must be changed
  * New version is: `int ca_auth(h, req, auth_type, authp, userp)`
    * where `auth_type` is the requested authentication-type (none, client-cert or user-defined)
    * `authp` is the returned authentication flag
    * `userp` is the returned associated authenticated user
    * and the return value is three-valued: -1: Error, 0: ignored, 1: OK
  * For more info see [clixon-docs](https://clixon-docs.readthedocs.io/en/latest/restconf.html)
* New clixon-restconf@2020-12-30.yang revision
2021-02-09 21:26:35 +01:00
Olof hagsand
40008f182e Fix restart of restconf processes 2021-02-01 10:44:50 +01:00
Olof hagsand
f65efa3e5e Add ca_reset plugin also when backend starts as -s none 2021-01-31 20:47:41 +01:00
Olof hagsand
f5f013c739 * Corrected client session handling to make internal IPC socket persistent
* Applies to cli/netconf/restconf/client-api code
  * Previous behaviour:
    * Close socket after each rpc, but now keeps the socket open until the client terminates
    * Kept locks over socket life-cycle, but according to RFC 6241 7.5 a lock should be relaeased when session ends
2021-01-27 15:07:27 +01:00
Olof hagsand
2d402b7ba5 Added validation of clixon-restconf.yang: server-key-path and server-cert-path must be present if ssl enabled. 2021-01-21 12:11:26 +01:00
Olof hagsand
96b50b88e8 Copyright 2021 2021-01-13 14:40:34 +01:00
Olof hagsand
cf63d0f761 * New clixon-config@2020-12-30.yang revision
* New clixon-lib@2020-12-30.yang revision
* Added callback to process-control RPC feature in clixon-lib.yang to manage processes
* Changed behavior of starting restconf internally using `CLICON_BACKEND_RESTCONF_PROCESS` monitoring changes in enable flag, not only the RPC.
* Changed: RPC process-control output parameter status to pid
2021-01-02 17:27:44 +01:00
Olof hagsand
b8f5d1dcae * Added CLICON_BACKEND_RESTCONF_PROCESS to clixon-config.yang 2020-12-17 12:13:06 +01:00
Olof hagsand
22adc58187 * New process-control RPC feature in clixon-lib.yang to manage processes
* This is an alternative to manage a clixon daemon via sudtemd, containerd or other
  * One important special case is starting the clixon-restconf daemon internally
  * This is how it works:
    * Register a process via `clixon_process_register(h, name, namespace, argv, argc)`
    * Use process-control RPC defined in clixon-lib.yang to start/stop/restart or query status on that process
  * Example code in the main example
2020-12-15 15:48:42 +01:00
Olof hagsand
a71e8aaceb * minor edits make datastore=nocache pass tests, and ensure generic copy-config works 2020-12-04 14:47:55 +01:00
Olof hagsand
9016dde319 Modify tests so they all work with evhtp and backend config, eg that CLICON_RESTCONF_CONFIG is default true.
Note that some tests are forced to CLICON_RESTCONF_CONFIG is false due to test complexities.
2020-11-18 13:57:25 +01:00
Olof hagsand
c31b1c471b * Changed first parameter from int fd to FILE *f in the following functions:
* clixon_xml_parse_file(), clixon_json_parse_file(), yang_parse_file()
  * See [Bytewise read() of files is slow #146](https://github.com/clicon/clixon/issues/146)
2020-11-04 22:21:42 +01:00
Olof hagsand
5c5e45d10a Added new backend plugin: ca_pre-demon called if backend is daemonized just prior to forking. 2020-10-31 17:08:59 +01:00
Olof hagsand
f9cc26582a minor: strncpy -> strcpy and formatting 2020-10-10 15:32:19 +02:00
Olof hagsand
6f2c4a076d * Configuration directory
* A new configuration option `CLICON_CONFIGDIR` has been added for loading of extra config files
  * If not given, only the main configfile is loaded.
  * If given, and if the directory exists, the files in this directory will be loaded alphabetically AFTER the main config file in the following way:
    * leaf values are overwritten
    * leaf-list values are appended
  * You can override file setting with `-E <dir>` command-line option.
* New clixon-config@2020-10-01.yang revision
  * Added option for configuration directory: `CLICON_CONFIGDIR`
2020-10-01 11:04:25 +02:00
Olof hagsand
5ee6283526 Fixed: [xmldb_get0 returns invalid candidate on startup transaction callbacks #126](https://github.com/clicon/clixon/issues/126). Always clear candidate-db before db initialization. 2020-09-07 15:09:19 +02:00
Olof hagsand
81fc7f742b * Netconf as default namespace has been disabled by default.
* Only requests on the form: `<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><edit-config>...` will be accepted
  * All replies will be on the form: `<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">...`
  * Requests such as: `<rpc><edit-config>...` will not  be accepted.
  * You can revert this behaviour (to clixon pre-4.6 behaviour) by enabling `CLICON_NAMESPACE_NETCONF_DEFAULT`
  * This API change is a consequence of: [copy-config's RPC cxobj parameter does not contain namespace #131](https://github.com/clicon/clixon/issues/131)
2020-09-02 15:44:10 +02:00
Olof hagsand
5b81c43d96 New backend switch: -q : Quit startup directly after upgrading and print result on stdout 2020-08-03 12:49:33 +02:00
Olof hagsand
23bcee8185 * Netconf lock/unlock behaviour changed to adhere to RFC 6241
* Changed commit lock error tag from "lock denied" to "in-use".
  * Changed unlock error message from "lock is already held" to #lock not active" or "lock held by other session".
  * Fixed [lock candidate succeeded even though it is modified #110](https://github.com/clicon/clixon/issues/110)
2020-07-28 16:13:34 +02:00
Olof hagsand
36576143b5 Revised CLICON_YANG_UNKNOWN_ANYDATA patch: dont allow new nodes after initial startup 2020-07-23 17:04:12 +02:00
Olof hagsand
e2d9c046af further restconf refactoring 2020-06-11 21:54:09 +02:00
Olof hagsand
ff5462ecac Replaced the global variable debug with access function: clicon_debug_get(). 2020-06-09 16:04:49 +02:00
Olof hagsand
14b03f63a3 all clixon event functions prepended with clixon_
Replaced fcgi getparam code with clixon-data parameters
2020-06-05 16:19:03 +02:00
Olof hagsand
1a1aa58249 CLIgen pt_expand API change and mem problems 2020-05-20 12:00:52 +02:00
Olof hagsand
dafc6d10e0 Added option to treat unknown XML (wrt YANG) as anydata. 2020-05-03 22:03:33 +02:00
Olof hagsand
b8ec6a4957 * Optimized prefix checks at xml parse time: using many prefixes slowed down parsing considerably
* Cleared startup-db cache after restart
2020-04-29 23:11:48 +02:00
Olof hagsand
9a8c6cf3e6 * New clixon-lib@2020-04-23.yang revision
* Added: stats RPC for clixon XML and memory statistics.
  * Added: restart-plugin RPC for restarting individual plugins without restarting backend.
* xml-stats moved from clixon-config.yang as state data to an rpc `datastats` in clixon-lib.yang
* Experimental: restart_plugin
* Two new plugin callbacks added
  * ca_daemon: Called just after a server has "daemonized", ie put in background.
  * ca_trans_commit_done: Called when all plugin commits have been done.
    * Note: If you have used "end" callback and usign transaction data, you should probably use this instead.
2020-04-28 22:10:06 +02:00
Olof hagsand
1c99bd6a9b * xml-stats moved from clixon-config.yang as state data to an rpc datastatsin clixon-lib.yang
* Two new plugin callbacks added
  * ca_daemon: Called just after a server has "daemonized", ie put in background.
  * ca_trans_commit_done: Called when all plugin commits have been done.
2020-04-23 22:58:57 +02:00
Olof hagsand
65806f1ef2 Copyright change: LLC -> LLC(Netgate) 2020-04-23 16:55:56 +02:00
Olof hagsand
09a2e09848 * New XML parsing API:
* `clixon_xml_parse_string()`
   * `clixon_xml_parse_file()`
* New JSON parsing API, with same signature as XML parsing:
   * `clixon_json_parse_string()`
   * `clixon_xml_parse_file()`
* XML YANG binding API have been rearranged as follows:
   * `xml_bind_yang_rpc()`
   * `xml_bind_yang_rpc_reply()`
   * `xml_bind_yang()`
   * `xml_bind_yang0()`
2020-03-19 21:32:27 +01:00
Olof hagsand
0f54899ae4 * Added option: CLICON_CLI_BUF_START and CLICON_CLI_BUF_THRESHOLD so you can change the start and
threshold of quadratic and linear growth of CLIgen buffers (cbuf:s)
* Memory footprint
  * Do not autopopulate namespace cache, instead use on-demand, see `xml2ns()`.
  * Set CBUF start level to 256 (`CLICON_CLI_BUF_START` option)
  * Reduced xml child vector default size from 4 to 1 with quadratic growoth to 64K then linear
2020-03-04 11:27:43 +01:00
Olof hagsand
9fa5e216c4 * New "general-purpose" datastore upgrade callback added which i called once on startup, intended for lo
w-level general upgrades and as a complement to module-specific upgrade.
  * Called on startup after initial XML parsing, but before module-specific upgrades
  * Enabled by definign the `.ca_datastore_upgrade`
  * [General-purpose upgrade documentation](https://clixon-docs.readthedocs.io/en/latest/backend.html#ge
neral-purpose)
* JSON parse error messages change from ` on line x: syntax error,..` to `json_parse: line x: syntax err
or`
* Unknown-element error message is more descriptive, eg from `namespace is: urn:example:clixon` to: `Fai
led to find YANG spec of XML node: x with parent: xp in namespace urn:example:clixon`.
* C-API parse and validation API more capable
  * `xml_spec_populate` family of functions extended with three-value return values
    * -1: error, 0: parse OK, 1: parse and YANG binding OK.
  * `xml_parse` and `json_parse` API changes
    * Three value returns: -1: error, 0: parse OK, 1: parse and YANG binding OK.
    * Extended `xml_parse_file2` and `xml_parse_string2` extended API functions with all options available.
      * New concept called `yang_bind` that defines how XML symbols are bound to YANG after parsing
    * Existing API same except `xml_parse_file` `endtag` argument moved to `xml_parse_file2`
* C-API: Added instrumentation: `xml_size` and `xml_stats_get`.
* Fixed: Enabling modstate (CLICON_XMLDB_MODSTATE), changing a revision on a yang, and restarting made the backend daemon exit at start (thanks Matt)
  * Also: ensure to load `ietf-yang-library.yang ` if CLICON_XMLDB_MODSTATE is set
2020-02-20 14:00:01 +01:00
Olof hagsand
19e21be0bc Very large commit for upcoming 4.4 release
Major New features

* New and updated search functions using xpath, api-path and instance-id
  * New search functions using api-path and instance_id:
    * C search functions: `clixon_find_instance_id()` and `clixon_find_api_path()`
  * Binary search optimization in lists for indexed leafs in all three formats.
    * This improves search performance to O(logN) which is drastical improvements for large lists.
  * You can also register explicit indexes for making binary search (not only list keys)
  * For more info, see docs at [paths](https://clixon-docs.readthedocs.io/en/latest/paths.html) and
[search](https://clixon-docs.readthedocs.io/en/latest/xml.html#searching-in-xml)

API changes on existing features (you may need to change your code)
* On failed validation of leafrefs, error message changed from: `No such leaf` to `No leaf <name> matching path <path>`.
* CLI Error message (clicon_rpc_generate_error()) changed when backend returns netconf error to be more descriptive:
  * Original: `Config error: Validate failed. Edit and try again or discard changes: Invalid argument`
  * New (example): `Netconf error: application operation-failed Identityref validation failed, undefined not derived from acl-base . Validate failed. Edit and try again or discard changes"

Minor changes

* Test framework
  * Added `-- -S <file>` command-line to main example to be able to return any state to main example.
  * Added `test/cicd` test scripts for running on a set of other hosts
* C-code restructuring
  * clixon_yang.c partitioned and moved code into clixon_yang_parse_lib.c and clixon_yang_module.c and move back some code from clixon_yang_type.c.
    * partly to reduce size, but most important to limit code that accesses internal yang structures, only clixon_yang.c does this now.
2020-02-02 15:52:30 +01:00
Olof Hagsand
a87e9cb246 clang warnings 2019-12-23 14:03:42 +00:00