From be59bd48d8a231c8b67a8c8634ba28e9f7c3b51b Mon Sep 17 00:00:00 2001 From: Olof hagsand Date: Wed, 27 Mar 2019 18:04:01 +0100 Subject: [PATCH] upgrade example and test --- doc/FAQ.md | 6 ++ doc/startup.md | 118 ++++++++++++-------------------- example/example_backend.c | 36 +++++----- lib/clixon/clixon_plugin.h | 2 +- lib/src/clixon_plugin.c | 4 +- test/test_feature.sh | 23 +++++-- test/test_upgrade_interfaces.sh | 2 +- 7 files changed, 88 insertions(+), 103 deletions(-) diff --git a/doc/FAQ.md b/doc/FAQ.md index 3e24a4c7..ab023a96 100644 --- a/doc/FAQ.md +++ b/doc/FAQ.md @@ -37,6 +37,7 @@ * [I want to add a hook to an existing operation, can I do that?](#i-want-to-add-a-hook-to-an-existing-operation-can-i-do-that) * [How do I write an authentication callback?](#how-do-i-write-an-authentication-callback) * [What about access control?](#what-about-access-control) + * [Does Clixon support upgrade?](#does-clixon-support-upgrade) * [How do I write a CLI translator function?](#how-do-i-write-a-cli-translator-function) ## What is Clixon? @@ -570,6 +571,11 @@ Control Model defined in [RFC8341](https://tools.ietf.org/html/rfc8341) Incoming RPC and data node access points are supported with some limitations. See the (README)(../README.md) for more information. +## Does Clixon support upgrade? + +Yes. Clixon provides a callback interface where datastores can be +upgraded. This is described in [the startup doc](startup.md). + ## How do I write a CLI translator function? The CLI can perform variable translation. This is useful if you want to diff --git a/doc/startup.md b/doc/startup.md index 8e972e00..19d3ad89 100644 --- a/doc/startup.md +++ b/doc/startup.md @@ -120,7 +120,7 @@ module-state of the backend daemon, a set of _upgrade_ callbacks are made. This allows a user to program upgrade funtions in the backend plugins to automatically upgrade the XML to the current version. -Clixon also has a system-provided [automatic upgrading method](#automatic-upgrades) based on Yang changelogs covered in a separate chapter. +Clixon also has an experimental [automatic upgrading method](#automatic-upgrades) based on Yang changelogs covered in a separate chapter. A user registers upgrade callbacks based on module and revision ranges. A user can register many callbacks, or choose wildcards. @@ -138,37 +138,36 @@ callback per module and revision range that will be called in a series. A user registers upgrade callbacks in the backend `clixon_plugin_init()` function. The signature of upgrade callback is as follows: ``` -int upgrade_callback_register(clicon_handle h, - clicon_upgrade_cb cb, - char *namespace, - uint32_t from, - uint32_t to, - void *arg) + upgrade_callback_register(h, cb, namespace, from, revision, arg); ``` where: * `h` is the Clicon handle, * `cb` is the name of the callback function, -* `namespace` defines a Yang module. NULL denotes all modules. Note that module `name` is not used. -* `revision` is the revision date to where the upgrade is made. It is either the same revision as the Clixon system module, or an older version. In the latter case, it i recommended to provide an upgrade function to the most recent revision. If revision is `0` it means that this module is not present in the system and is _obsolete_. -* `from` is a revision date indicated an optional start date of the upgrade. This allows for defining a partial upgrade. It can also be `0` to denote any old version. +* `namespace` defines a Yang module. NULL denotes all modules. Note that module `name` is not used (XML uses namespace, whereas JSON uses name, XML is more common). +* `from` is a revision date indicated an optional start date of the upgrade. This allows for defining a partial upgrade. It can also be `0` to denote any version. +* `revision` is the revision date "to" where the upgrade is made. It is either the same revision as the Clixon system module, or an older version. In the latter case, you can provide another upgrade callback to the most recent revision. * `arg` is a user defined argument which can be passed to the callback. One example of registering a "catch-all" upgrade: ``` - upgrade_callback_register(h, yang_changelog_upgrade, NULL, 0, 0, NULL); + upgrade_callback_register(h, xml_changelog_upgrade, NULL, 0, 0, NULL); ``` -Another example are fine-grained stepwise upgrades of a single module: +Another example are fine-grained stepwise upgrades of a singlemodule [upgrade example](#example-upgrade): ``` - upgrade_callback_register(h, upgrade_a_1, "urn:example:a", - 20171201, 20180101, NULL); - upgrade_callback_register(h, upgrade_a_2, "urn:example:a", - 20180101, 20190101, NULL); + upgrade_callback_register(h, upgrade_2016, "urn:example:interfaces", + 20140508, 20160101, NULL); + upgrade_callback_register(h, upgrade_2018, "urn:example:interfaces", + 20160101, 20180220, NULL); ``` - -In the latter case, one upgrade function updates data modelled by `A` -from revision 2017-12-01 to 2018-01-01; a separate one updates from -2018-01-01 to 2019-01-01. These are run in series. +``` + 20140508 20160101 20180220 +------+--------------+--------------+--------> + upgrade_2016 upgrade_2018 +``` +In the latter case, the first callback upgrades +from revision 2014-05-08 to 2016-01-01; while the second makes upgrades from +2016-01-01 to 2018-02-20. These are run in series. ### Upgrade callback @@ -181,61 +180,15 @@ Note the following: * Upgrade callbacks _will_ be called if the datastore contains a version of a module that is older than the module loaded in Clixon. * Upgrade callbacks _will_ also be called if the datastore contains a version of a module that is not present in Clixon - an obsolete module. -An example upgrade callback: +Re-using the previous stepwise example, if a datastore is loaded based on revision 20140508 by a system supporting revision 20180220, the following two callbacks will be made: ``` - /*! Automatic upgrade of module A - * @param[in] h Clicon handle - * @param[in] xn XML tree to be updated - * @param[in] namespace Namespace of module (for info) - * @param[in] from From revision on the form YYYYMMDD - * @param[in] to To revision on the form YYYYMMDD (0 not in system) - * @param[in] arg User argument given at rpc_callback_register() - * @param[out] cbret Return xml tree, eg ..., , "urn:example:interfaces", 20140508, 20180220, NULL, cbret); + upgrade_2018(h, , "urn:example:interfaces", 20140508, 20180220, NULL, cbret); ``` - Note that the example shown is a template for an upgrade function. It gets the nodes of an yang module given by `namespace` and the (outdated) `from` revision, and iterates through them. Actual -upgrading code may be implemented by a user. +upgrading code is in [the example](../example/example_backend.c). If no action is made by the upgrade calback, and thus the XML is not upgraded, the next step is XML/Yang validation. @@ -247,6 +200,19 @@ However, if the validation fails, the backend will try to enter the failsafe mode so that the user may perform manual upgarding of the configuration. +### Example upgrade + +[The example](../example/example_backend.c) and [test](../test/test_upgrade_interfaces.sh) shos the code for upgrading of an interface module. The example is inspired by the ietf-interfaces module that made a subset of the upgrades shown in the examples. + +The code is split in two steps. The `upgrade_2016` callback does the following transforms: + * Move /if:interfaces-state/if:interface/if:admin-status to /if:interfaces/if:interface/ + * Move /if:interfaces-state/if:interface/if:statistics to if:interfaces/if:interface/ + * Rename /interfaces/interface/description to /interfaces/interface/descr + +While the `upgrade_2018` callback does the following transforms: + * Delete /if:interfaces-state + * Wrap /interfaces/interface/descr to /interfaces/interface/docs/descr + * Change type /interfaces/interface/statistics/in-octets to decimal64 and divide all values with 1000 ## Extra XML @@ -358,10 +324,10 @@ The example shown in this Section is also available as a regression [test script ## Automatic upgrades -Clixon supports an experimental yang changelog feature based on +Clixon supports an EXPERIMENTAL xml changelog feature based on "draft-wang-netmod-module-revision-management-01" (Zitao Wang et al) where changes to the Yang model are documented and loaded into -Clixon. +Clixon. The implementation is not complete. When upgrading, the system parses the changelog and tries to upgrade the datastore automatically. This featire is experimental and has @@ -369,10 +335,10 @@ several limitations. You enable the automatic upgrading by registering the changelog upgrade method in `clixon_plugin_ini()` using wildcards: ``` - upgrade_callback_register(h, yang_changelog_upgrade, NULL, 0, 0, NULL); + upgrade_callback_register(h, xml_changelog_upgrade, NULL, 0, 0, NULL); ``` -The transformation is defined by a list of changelogs. Each changelog defined how a module (defined by a namespace) is transformed from an old revision to a nnew. Example: +The transformation is defined by a list of changelogs. Each changelog defined how a module (defined by a namespace) is transformed from an old revision to a nnew. Example from [test_upgrade_auto.sh](../test/test_upgrade_auto.sh) ``` @@ -389,12 +355,12 @@ Each changelog consists of set of (orderered) steps: 1 insert /a:system - <y>created</y> + created 2 delete - /a:system/a:x + /a:system/a:x ``` Each step has an (atomic) operation: diff --git a/example/example_backend.c b/example/example_backend.c index 6a8e329a..caaacfd1 100644 --- a/example/example_backend.c +++ b/example/example_backend.c @@ -267,7 +267,7 @@ example_statedata(clicon_handle h, * @retval -1 Error * @see clicon_upgrade_cb * @see test_upgrade_interfaces.sh - * @see upgrade_interfaces_2016 + * @see upgrade_2016 * This example shows a two-step upgrade where the 2014 function does: * - Move /if:interfaces-state/if:interface/if:admin-status to * /if:interfaces/if:interface/ @@ -276,13 +276,13 @@ example_statedata(clicon_handle h, * - Rename /interfaces/interface/description to descr */ static int -upgrade_interfaces_2014(clicon_handle h, - cxobj *xt, - char *ns, - uint32_t from, - uint32_t to, - void *arg, - cbuf *cbret) +upgrade_2016(clicon_handle h, + cxobj *xt, + char *ns, + uint32_t from, + uint32_t to, + void *arg, + cbuf *cbret) { int retval = -1; yang_spec *yspec; @@ -367,7 +367,7 @@ upgrade_interfaces_2014(clicon_handle h, * @retval -1 Error * @see clicon_upgrade_cb * @see test_upgrade_interfaces.sh - * @see upgrade_interfaces_2014 + * @see upgrade_2016 * The 2016 function does: * - Delete /if:interfaces-state * - Wrap /interfaces/interface/descr to /interfaces/interface/docs/descr @@ -375,13 +375,13 @@ upgrade_interfaces_2014(clicon_handle h, * fraction-digits 3 and divide all values with 1000 */ static int -upgrade_interfaces_2016(clicon_handle h, - cxobj *xt, - char *ns, - uint32_t from, - uint32_t to, - void *arg, - cbuf *cbret) +upgrade_2018(clicon_handle h, + cxobj *xt, + char *ns, + uint32_t from, + uint32_t to, + void *arg, + cbuf *cbret) { int retval = -1; yang_spec *yspec; @@ -624,9 +624,9 @@ clixon_plugin_init(clicon_handle h) * test interface example. Otherwise the auto-upgrade feature is enabled. */ if (_upgrade){ - if (upgrade_callback_register(h, upgrade_interfaces_2014, "urn:example:interfaces", 0, 0, NULL) < 0) + if (upgrade_callback_register(h, upgrade_2016, "urn:example:interfaces", 20140508, 20160101, NULL) < 0) goto done; - if (upgrade_callback_register(h, upgrade_interfaces_2016, "urn:example:interfaces", 0, 0, NULL) < 0) + if (upgrade_callback_register(h, upgrade_2018, "urn:example:interfaces", 20160101, 20180220, NULL) < 0) goto done; } else diff --git a/lib/clixon/clixon_plugin.h b/lib/clixon/clixon_plugin.h index 5886cf58..5e9dc68c 100644 --- a/lib/clixon/clixon_plugin.h +++ b/lib/clixon/clixon_plugin.h @@ -204,7 +204,7 @@ struct clixon_plugin_api{ /* * Macros */ -#define upgrade_callback_register(h, cb, namespace, from, to, arg) upgrade_callback_reg_fn((h), (cb), #cb, (namespace), (from), (to), (arg)) +#define upgrade_callback_register(h, cb, namespace, from, rev, arg) upgrade_callback_reg_fn((h), (cb), #cb, (namespace), (from), (rev), (arg)) typedef struct clixon_plugin_api clixon_plugin_api; diff --git a/lib/src/clixon_plugin.c b/lib/src/clixon_plugin.c index 8f4bed82..45b97e88 100644 --- a/lib/src/clixon_plugin.c +++ b/lib/src/clixon_plugin.c @@ -559,8 +559,8 @@ static upgrade_callback_t *upgrade_cb_list = NULL; * @param[in] fnstr Stringified function for debug * @param[in] arg Domain-specific argument to send to callback * @param[in] namespace Module namespace (if NULL all modules) - * @param[in] rev To module revision (0 means module obsoleted) * @param[in] from From module revision (0 from any revision) + * @param[in] revision To module revision (0 means module obsoleted) * @retval 0 OK * @retval -1 Error * @see upgrade_callback_call which makes the actual callback @@ -570,8 +570,8 @@ upgrade_callback_reg_fn(clicon_handle h, clicon_upgrade_cb cb, const char *fnstr, char *namespace, - uint32_t revision, uint32_t from, + uint32_t revision, void *arg) { upgrade_callback_t *uc; diff --git a/test/test_feature.sh b/test/test_feature.sh index 7c6622eb..d3145263 100755 --- a/test/test_feature.sh +++ b/test/test_feature.sh @@ -1,5 +1,21 @@ #!/bin/bash -# Yang features. if-feature. and schema resources according to RFC7895 +# Yang features. if-feature. +# The test has a example module with FEATURES A and B, where A is enabled and +# B is disabled. +# It also uses an ietf-router module where (only) router-id is enabled +# Also check modules-state (RFC7895) announces the enabled features. +# +# From RFC7950: +# 7.20.1 Schema nodes tagged with an "if-feature" statement are _ignored_ by +# the server unless the server supports the given feature expression. +# 8.1: There MUST be no nodes tagged with "if-feature" present if the +# "if-feature" expression evaluates to "false" in the server. +# - Should the server just "ignore" these nodes or actively reject them? +# +# Clixon has a strict implementation of the features so that setting +# data with disabled features is same as if they are not present in the Yang. +# Which means no cli syntax or edit operations were syntactically allowed +# (and therefore invalid). # Magic line must be first in script (see README.md) s="$_" ; . ./lib.sh || if [ "$s" = $0 ]; then exit 0; else return 0; fi @@ -31,7 +47,7 @@ cat < $cfg EOF cat < $fyang -module $APPNAME{ +module example{ yang-version 1.1; namespace "urn:example:clixon"; prefix ex; @@ -44,9 +60,6 @@ module $APPNAME{ feature B{ description "This test feature is disabled"; } - feature C{ - description "This test feature is disabled"; - } leaf x{ if-feature A; type "string"; diff --git a/test/test_upgrade_interfaces.sh b/test/test_upgrade_interfaces.sh index ebbcb8e4..3b641eb6 100755 --- a/test/test_upgrade_interfaces.sh +++ b/test/test_upgrade_interfaces.sh @@ -270,7 +270,7 @@ cat < $cfg $cfg /usr/local/share/clixon - *:* + interfaces:if-mib $dir /usr/local/var/$APPNAME/$APPNAME.sock /usr/local/lib/example/backend