Starting 3.10 release cycle

Moved docs to doc/

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Clixon Changelog
 
+## 3.10.0 (Upcoming)
+
+### Major New features
+
+### API changes on existing features (you may need to change your code)
+
+### Minor changes
+
+### Corrected Bugs
+
 ## 3.9.0 (21 Feb 2019)
 
 Thanks for all bug reports, feature requests and support! Thanks to [Netgate](https://www.netgate.com) and other sponsors for making Clixon a better tool!

README.md

@@ -27,7 +27,7 @@ support.
   * [Clixon project page](http://www.clicon.org)
   * [Tests and CI](test/README.md)
   * [Containers](docker/README.md)
-  * [Roadmap](ROADMAP.md)
+  * [Roadmap](doc/ROADMAP.md)
   * [Reference manual](#reference) 
   
 ## Background

configure

@@ -2167,9 +2167,9 @@ ac_compiler_gnu=$ac_cv_c_compiler_gnu
 : ${INSTALLFLAGS="-s"}
 
 CLIXON_VERSION_MAJOR="3"
-CLIXON_VERSION_MINOR="9"
+CLIXON_VERSION_MINOR="10"
 CLIXON_VERSION_PATCH="0"
-CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}\""
+CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}.PRE\""
 
 # Fix to specific CLIgen version (eg 3.5) or head (3)
 CLIGEN_VERSION="3"

configure.ac

@@ -43,9 +43,9 @@ AC_INIT(lib/clixon/clixon.h.in)
 : ${INSTALLFLAGS="-s"}
 
 CLIXON_VERSION_MAJOR="3"
-CLIXON_VERSION_MINOR="9"
+CLIXON_VERSION_MINOR="10"
 CLIXON_VERSION_PATCH="0"
-CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}\""
+CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}.PRE\""
 
 # Fix to specific CLIgen version (eg 3.5) or head (3) 
 CLIGEN_VERSION="3" 

doc/DEVELOP.md

@@ -50,6 +50,8 @@ configure.ac --.
                     +--> config.status* -+               +--> make*
      Makefile.in ---'                    `-> Makefile ---'
 ```
+Note: remember to run autoheader sometimes (when?)
+And when you do note (https://github.com/olofhagsand/cligen/issues/17) which states that cligen_custom.h should be in quote. 
 
 ## Debug
 How to debug
@@ -96,9 +98,24 @@ EOF
 
 ## New release
 What to think about when doing a new release.
-* run test/mem.sh
+* Ensure all tests run OK
-* New clixon-config.yang revision?
+* New yang/clicon/clixon-config@XXX.yang revision?
-Tagging:
+* In configure.ac, for minor releases change CLIXON_VERSION in configure.ac to eg: (minor should have been bumped):
-* change CLIXON_VERSION in configure.ac
+```
-* git tag -a <version"
+  CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}\""
-* git push origin <version>
+```
+* For patch releases change CLIXON_VERSION_PATCH
+* Run autoconf
+* Git stuff:
+```
+  git tag -a <version"
+  git push origin <version>
+```
+
+After release:
+* Bump minor version and add a "PRE":
+```
+  CLIXON_VERSION_MINOR="10" ++
+  CLIXON_VERSION="\"${CLIXON_VERSION_MAJOR}.${CLIXON_VERSION_MINOR}.${CLIXON_VERSION_PATCH}.PRE\""
+```
+* Run autoconf

doc/ROADMAP.md

@@ -3,8 +3,10 @@
 ## High prio
 - Special handling of the initial startup transaction to avoid exit at startup
   - Possibly - draft-wu-netconf-restconf-factory-restore-03
+  - See (startup.md)
 - Handle revisions to data model.
   - Possibly draft-wang-netmod-module-revision-management-01
+  - See (startup.md)
 - (DONE) NACM (RFC 8341) 
   - NACM support for create, read, update, delete operations 
   - ACM support for specifying a module name other than '*' 
@@ -33,12 +35,12 @@ Not prioritized:
   - Query parameters
 - Streams (netconf and restconf)
   - Extend native stream mode with external persistent timeseries database, eg influxdb.
-- Jenkins CI/CD and webhooks
+- (DONE)Jenkins/Travis CI/CD and webhooks
 - YANG
   - RFC 6022 [NETCONF monitoring](https://github.com/clicon/clixon/issues/39)
   - Deviation, min/max-elements, action, unique
 - Containers
-  - [Docker improvements](https://github.com/clicon/clixon/issues/44)
+  - (DONE)[Docker improvements](https://github.com/clicon/clixon/issues/44)
   - Kubernetes Helm chart definition
 - [gRPC](https://github.com/clicon/clixon/issues/43)
 

doc/startup.md

@@ -0,0 +1,62 @@
+# New Clixon Startup functionality
+
+## Objectives
+When Clixon 3.9 backend starts, it assumes a well-formed initial XML
+configuration which it parses and validates. Depending on starting
+mode (-s command-line) this is the "startup" or "running"
+configuration.
+
+If this initial configuration fails, clixon backend exits. This has
+the consequence that an operator cannot manage the system unless with
+out-of-band mechanisms.
+
+## Objectives
+This document describes a new startup mechanism with the following goals:
+* An operator should be notified of the startup status
+* The backend should remain up in case of errors but may enter a "failsafe" mode.
+* XML syntax errors should be detected and reported
+* Yang module info is added to (startup) datastore database
+* Yang module mismatch should be detected and reported
+* Validation failures should be detected and reported, specifically of mismatching modules.
+
+## Proposal
+
+A new user callback is introduced:
+```
+  int startup-cb(h, status, module-state-diff, *valid)
+```
+which is called once at startup to report startup state to application:
+- status is one of: OK, INVALID and ERROR.
+- module-state-diff contains a list of RFC7895 differences between the yang modules running in the system, and the ones in the startup config.
+- valid is a return value that if set to 0 forces the status to INVALID (if OK on entry).
+
+A new read-only datastore is introduced:
+```
+  CLICON_XMLDB_FAILSAFE If set, a failsafe read-only datastore is expected,
+                         in CLICON_XMLDB_DIR, called failsafe_db
+```
+
+Datastore databases are optionally extended with modules state according to
+RFC7895. A new config option is introduced to control this:
+```
+  CLICON_XMLDB_MODSTATE If set, tag datastores with RFC 7895 YANG Module Library 
+ info. When loaded at startup, a check is made if the system yang modules match
+```
+
+Proposed algoritm:
+0. Backend starts with a set of yang module revisions as of RFC7895.
+1. Parse startup XML (or JSON)
+2. If syntax failure, call startup-cb(ERROR), copy failsafe db to candidate and commit. Done
+3. Check yang module versions between backend and init config XML. (msdiff)
+4. Validate startup db. (valid)
+5. If valid fails, call startup-cb(Invalid, msdiff), keep startup in candidate and commit failsafe db. Done.
+6. Call startup-cb(OK, msdiff) and commit. 
+
+Note:
+
+1. If done in step 2) the failsafe db is in both candidate and running. The operator need to repair the XML file before reloading.
+2. If done in step 5) The operator has the non-valid database in candidate and can edit it, and when ready can commit it. During this time, the failsafe db is running.
+3. If done in steps 5 and 6, the module-state-diff contains the (potential) differences in the modules-state diff.
+
+## Thanks
+Thanks matt smith and dave cornejo for input