diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a8628ed..799fff1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,17 +21,21 @@ * Option CLICON_MODULE_SET_ID is set and changed when modules change. * Notification not supported * Yang 1.1 notification support (RFC 7950: Sec 7.16) -* Major rewrite of event streams +* Restconf stream notification support - two variants. + * Both a "native" stream support and one using nginx/nchan pub/sub. + * See (apps/restconf/README.md) for details. +* New event streams implementation * See clicon_stream.[ch] for details * Added stream discovery according to RFC 5277 for netconf and RFC 8040 for restconf * Enabled by CLICON_STREAM_DISCOVERY_RFC5277 and CLICON_STREAM_DISCOVERY_RFC8040. - * Set access/subscribe base URL with: CLICON_STREAM_URL_PREFIX (default https://localhost/streams). + * Set access/subscribe base URL with: CLICON_STREAM_URL (default "https://localhost") and CLICON_STREAM_PATH (default "streams") * Example: new stream "foo" will get access URL: https://localhost/streams/foo * Optional pub/sub support enabled by ./configure --enable-publish - * Set publish URL base with: CLICON_STREAM_PUB_PREFIX (default http://localhost/pub) + * Set publish URL base with: CLICON_STREAM_PUB (default http://localhost/pub) * Example: new stream "foo" will get pub URL: https://localhost/pub/foo ### API changes on existing features (you may need to change your code) +* clixon-config YAML file has new revision: 2018-10-21. * Netconf hello capability updated to YANG 1.1 RFC7950 Sec 5.6.4 * Added urn:ietf:params:netconf:capability:yang-library:1.0 * Thanks @SCadilhac for helping out, see https://github.com/clicon/clixon/issues/39 diff --git a/README.md b/README.md index b821d658..440f57af 100644 --- a/README.md +++ b/README.md @@ -140,11 +140,11 @@ run with NGINX. The implementatation is based on [RFC 8040: RESTCONF Protocol](https://tools.ietf.org/html/rfc8040). The following features are supported: - OPTIONS, HEAD, GET, POST, PUT, DELETE +- stream notifications + The following are not implemented - PATCH - query parameters (section 4.9) -- notifications (sec 6) -- schema resource See [more detailed instructions](apps/restconf/README.md). @@ -183,7 +183,6 @@ according to [RFC8341(NACM)](https://tools.ietf.org/html/rfc8341), at least a subset of the functionality. See more information here: [NACM](README_NACM.md). - Runtime ======= diff --git a/apps/backend/backend_client.c b/apps/backend/backend_client.c index 70517440..96a372e2 100644 --- a/apps/backend/backend_client.c +++ b/apps/backend/backend_client.c @@ -850,6 +850,11 @@ from_client_create_subscription(clicon_handle h, goto done; } } + if ((stream_find(h, stream)) == NULL){ + if (netconf_invalid_value(cbret, "application", "No such stream") < 0) + goto done; + goto ok; + } if (stream_cb_add(h, stream, selector, ce_event_cb, (void*)ce) < 0) goto done; cprintf(cbret, ""); diff --git a/apps/netconf/netconf_rpc.c b/apps/netconf/netconf_rpc.c index 2da03ef7..838f0c73 100644 --- a/apps/netconf/netconf_rpc.c +++ b/apps/netconf/netconf_rpc.c @@ -728,6 +728,14 @@ netconf_discard_changes(clicon_handle h, major + * @see rfc5277: + * An event notification is sent to the client who initiated a + * command asynchronously when an event of + * interest... + * Parameters: eventTime type dateTime and compliant to [RFC3339] + * Also contains notification-specific tagged content, if any. With + * the exception of , the content of the notification is + * beyond the scope of this document. */ static int netconf_notification_cb(int s, @@ -822,6 +830,8 @@ netconf_create_subscription(clicon_handle h, } if (clicon_rpc_netconf_xml(h, xml_parent(xn), xret, &s) < 0) goto done; + if (xpath_first(*xret, "rpc-reply/rpc-error") != NULL) + goto ok; if (event_reg_fd(s, netconf_notification_cb, NULL, diff --git a/apps/restconf/README.md b/apps/restconf/README.md index dcce7567..14c2dcf4 100644 --- a/apps/restconf/README.md +++ b/apps/restconf/README.md @@ -22,14 +22,9 @@ server { fastcgi_pass unix:/www-data/fastcgi_restconf.sock; include fastcgi_params; } - location /stream { # for restconf notifications - fastcgi_pass unix:/www-data/fastcgi_restconf.sock; - include fastcgi_params; - proxy_http_version 1.1; - proxy_set_header Connection ""; - } } ``` + Start nginx daemon ``` sudo /etc/init.d nginx start @@ -73,12 +68,77 @@ olof@vandal> curl -G http://127.0.0.1/restconf/data/interfaces/interface/name=et curl -sX POST -d '{"interfaces":{"interface":{"name":"eth1","type":"eth","enabled":"true"}}}' http://localhost/restconf/data ``` -### Nginx Nchan for streams -Restconf notification event streams needs a server-side push -package. Clixon has used Nchan (nchan.io) for this +### Event streams -Download and install nchan, see nchan.io, Install section. +Clixon have two experimental restconf event stream implementations following +RFC8040 Section 6 using SSE. One native and one using Nginx +nchan. The two variants to subscribe to the stream is described in the +next section. + +The example [../../example/README.md] creates and EXAMPLE stream. + +Set the Clixon configuration options if they differ from default values - if they are OK you do not need to modify them: +``` +streams +https://example.com +http://localhost/pub +``` +where +- https://example.com/streams is the public fronting subscription base URL. A specific stream NAME can be accessed as https://example.com/streams/NAME +- http://localhost/pub is the local internal base publish stream. + +You access the streams using curl, but they differ slightly in behaviour as described in the following two sections. + +### Native event streams + +Add the following to extend the nginx configuration file with the following statements: +``` + location /streams { + fastcgi_pass unix:/www-data/fastcgi_restconf.sock; + include fastcgi_params; + proxy_http_version 1.1; + proxy_set_header Connection ""; + } +``` + +You access a native stream as follos: +``` + curl -H "Accept: text/event-stream" -s -X GET http://localhost/streams/EXAMPLE + curl -H "Accept: text/event-stream" -s -X GET http://localhost/streams/EXAMPLE?start-time=2014-10-25T10%3A02%3A00Z&stop-time=2014-10-25T12%3A31%3A00Z +``` +where the first command retrieves only new notifications, and the second receives a range of messages. + +### Nginx Nchan streams + +Nginx uses pub/sub channels and can be configured in a variety of +ways. The following uses a simple variant with one generic subscription +channel (streams) and one publication channel (pub). + +Configure clixon with `--enable-publish` which enables curl code for publishing streams to nchan. + +Download and install nchan, see (https://nchan.io/#install). + +Add the following to extend the nginx configuration file with the following statements: +``` + location ~ /streams/(\w+)$ { + nchan_subscriber; + nchan_channel_id $1; #first capture of the location match + } + location ~ /pub/(\w+)$ { + nchan_publisher; + nchan_channel_id $1; #first capture of the location match + } +``` + +Access the event stream EXAMPLE using curl: +``` + curl -H "Accept: text/event-stream" -s -X GET http://localhost/streams/EXAMPLE + curl -H "Accept: text/event-stream" -H "Last-Event-ID: 1539961709:0" -s -X GET http://localhost/streams/EXAMPLE +``` +where the first command retrieves the whole stream history, and the second only retreives the most recent messages given by the ID. + +See (https://nchan.io/#eventsource) on more info on how to access an SSE sub endpoint. ### Debugging diff --git a/apps/restconf/restconf_lib.c b/apps/restconf/restconf_lib.c index ab5db7b5..0ac75dc3 100644 --- a/apps/restconf/restconf_lib.c +++ b/apps/restconf/restconf_lib.c @@ -221,6 +221,26 @@ notfound(FCGX_Request *r) return 0; } +/*! HTTP error 406 Not acceptable + * @param[in] r Fastcgi request handle + */ +int +notacceptable(FCGX_Request *r) +{ + char *path; + + clicon_debug(1, "%s", __FUNCTION__); + path = FCGX_GetParam("DOCUMENT_URI", r->envp); + FCGX_FPrintF(r->out, "Status: 406\r\n"); /* 406 not acceptible */ + + FCGX_FPrintF(r->out, "Content-Type: text/html\r\n\r\n"); + FCGX_FPrintF(r->out, "

Not Acceptable

\n"); + FCGX_FPrintF(r->out, "Not Acceptable\n"); + FCGX_FPrintF(r->out, "The target resource does not have a current representation that would be acceptable to the user agent.\n", + path); + return 0; +} + /*! HTTP error 409 * @param[in] r Fastcgi request handle */ diff --git a/apps/restconf/restconf_lib.h b/apps/restconf/restconf_lib.h index c01ae7f6..0c7f6830 100644 --- a/apps/restconf/restconf_lib.h +++ b/apps/restconf/restconf_lib.h @@ -40,7 +40,6 @@ * Constants */ #define RESTCONF_API "restconf" -#define RESTCONF_STREAM "stream" /* * Prototypes (also in clixon_restconf.h) @@ -52,6 +51,7 @@ int badrequest(FCGX_Request *r); int unauthorized(FCGX_Request *r); int forbidden(FCGX_Request *r); int notfound(FCGX_Request *r); +int notacceptable(FCGX_Request *r); int conflict(FCGX_Request *r); int internal_server_error(FCGX_Request *r); int notimplemented(FCGX_Request *r); diff --git a/apps/restconf/restconf_main.c b/apps/restconf/restconf_main.c index 298c67bb..2b3949de 100644 --- a/apps/restconf/restconf_main.c +++ b/apps/restconf/restconf_main.c @@ -529,6 +529,7 @@ main(int argc, yang_spec *yspec = NULL; yang_spec *yspecfg = NULL; /* For config XXX clixon bug */ char *yang_filename = NULL; + char *stream_path; /* In the startup, logs to stderr & debug flag set later */ clicon_log_init(__PROGRAM__, LOG_INFO, logdst); @@ -561,7 +562,6 @@ main(int argc, goto done; break; } /* switch getopt */ - /* * Logs, error and debug to stderr or syslog, set debug level */ @@ -583,7 +583,7 @@ main(int argc, /* Find and read configfile */ if (clicon_options_main(h, yspecfg) < 0) goto done; - + stream_path = clicon_option_str(h, "CLICON_STREAM_PATH"); /* Now rest of options, some overwrite option file */ optind = 1; opterr = 0; @@ -652,10 +652,6 @@ main(int argc, if (clicon_option_bool(h, "CLICON_STREAM_DISCOVERY_RFC5277") && yang_spec_parse_module(h, "ietf-netconf-notification", CLIXON_DATADIR, NULL, yspec, NULL)< 0) goto done; - - - if (stream_register(h, "NETCONF", "default NETCONF event stream") < 0) - goto done; /* Call start function in all plugins before we go interactive Pass all args after the standard options to plugin_start */ @@ -692,8 +688,8 @@ main(int argc, clicon_debug(1, "path: %s", path); if (strncmp(path, "/" RESTCONF_API, strlen("/" RESTCONF_API)) == 0) api_restconf(h, r); /* This is the function */ - else if (strncmp(path, "/" RESTCONF_STREAM, strlen("/" RESTCONF_STREAM)) == 0) { - api_stream(h, r); + else if (strncmp(path+1, stream_path, strlen(stream_path)) == 0) { + api_stream(h, r, stream_path); } else if (strncmp(path, RESTCONF_WELL_KNOWN, strlen(RESTCONF_WELL_KNOWN)) == 0) { api_well_known(h, r); /* */ diff --git a/apps/restconf/restconf_stream.c b/apps/restconf/restconf_stream.c index 71482434..e2c3615a 100644 --- a/apps/restconf/restconf_stream.c +++ b/apps/restconf/restconf_stream.c @@ -35,6 +35,26 @@ See RFC 8040 RESTCONF Protocol Sections 3.8, 6, 9.3 + RFC8040: + A RESTCONF server MAY send the "retry" field, and if it does, RESTCONF + clients SHOULD use it. A RESTCONF server SHOULD NOT send the "event" + or "id" fields, as there are no meaningful values. RESTCONF + servers that do not send the "id" field also do not need to support + the HTTP header field "Last-Event-ID" + + The RESTCONF client can then use this URL value to start monitoring + the event stream: + + GET /streams/NETCONF HTTP/1.1 + Host: example.com + Accept: text/event-stream + Cache-Control: no-cache + Connection: keep-alive + + The server MAY support the "start-time", "stop-time", and "filter" + query parameters, defined in Section 4.8. Refer to Appendix B.3.6 + for filter parameter examples. + */ #ifdef HAVE_CONFIG_H @@ -62,31 +82,136 @@ /* clicon */ #include -#include /* Need to be after clixon_xml-h due to attribute format */ +#include /* Need to be after clixon_xml.h due to attribute format */ +#include "restconf_lib.h" +#include "restconf_stream.h" + +/*! Callback when stream notifications arrive from backend + */ static int -restconf_stream(clicon_handle h, - FCGX_Request *r, - event_stream_t *es) +restconf_stream_cb(int s, + void *arg) { - int retval = -1; - + int retval = -1; + FCGX_Request *r = (FCGX_Request *)arg; + int eof; + struct clicon_msg *reply = NULL; + cxobj *xtop = NULL; /* top xml */ + cxobj *xn; /* notification xml */ + cbuf *cb; + int pretty = 0; /* XXX should be via arg */ + clicon_debug(1, "%s", __FUNCTION__); + /* get msg (this is the reason this function is called) */ + if (clicon_msg_rcv(s, &reply, &eof) < 0){ + clicon_debug(1, "%s msg_rcv error", __FUNCTION__); + goto done; + } + clicon_debug(1, "%s msg: %s", __FUNCTION__, reply->op_body); + /* handle close from remote end: this will exit the client */ + if (eof){ + clicon_debug(1, "%s eof", __FUNCTION__); + clicon_err(OE_PROTO, ESHUTDOWN, "Socket unexpected close"); + close(s); + errno = ESHUTDOWN; + event_unreg_fd(s, restconf_stream_cb); + FCGX_FPrintF(r->out, "SHUTDOWN\r\n"); + FCGX_FPrintF(r->out, "\r\n"); + FCGX_FFlush(r->out); + clicon_exit_set(); + goto done; + } + if (clicon_msg_decode(reply, &xtop) < 0) + goto done; + /* create event */ + if ((cb = cbuf_new()) == NULL){ + clicon_err(OE_PLUGIN, errno, "cbuf_new"); + goto done; + } + if ((xn = xpath_first(xtop, "notification")) == NULL) + goto ok; +#ifdef notused + xt = xpath_first(xn, "eventTime"); + if ((xe = xpath_first(xn, "event")) == NULL) /* event can depend on yang? */ + goto ok; + if (xt) + FCGX_FPrintF(r->out, "M#id: %s\r\n", xml_body(xt)); + else{ /* XXX */ + gettimeofday(&tv, NULL); + FCGX_FPrintF(r->out, "M#id: %02d:0\r\n", tv.tv_sec); + } +#endif + if (clicon_xml2cbuf(cb, xn, 0, pretty) < 0) + goto done; + FCGX_FPrintF(r->out, "data: %s\r\n", cbuf_get(cb)); + FCGX_FPrintF(r->out, "\r\n"); + FCGX_FFlush(r->out); + + ok: + retval = 0; + done: + clicon_debug(1, "%s retval: %d", __FUNCTION__, retval); + if (xtop != NULL) + xml_free(xtop); + if (reply) + free(reply); + if (cb) + cbuf_free(cb); + return retval; +} + +/*! Send subsctription to backend + * @param[in] h Clicon handle + * @param[in] r Fastcgi request handle + * @param[in] name Stream name + * @param[out] sp Socket -1 if not set + */ +static int +restconf_stream(clicon_handle h, + FCGX_Request *r, + char *name, + int pretty, + int use_xml, + int *sp) +{ + int retval = -1; + cxobj *xret = NULL; + cxobj *xe; + cbuf *cb = NULL; + int s; /* socket */ + + *sp = -1; + clicon_debug(1, "%s", __FUNCTION__); + if ((cb = cbuf_new()) == NULL){ + clicon_err(OE_XML, errno, "cbuf_new"); + goto done; + } + cprintf(cb, "%s]]>]]>", name); + if (clicon_rpc_netconf(h, cbuf_get(cb), &xret, &s) < 0) + goto done; + if ((xe = xpath_first(xret, "rpc-reply/rpc-error")) != NULL){ + if (api_return_err(h, r, xe, pretty, use_xml) < 0) + goto done; + goto ok; + } + /* Setting up stream */ FCGX_SetExitStatus(201, r->out); /* Created */ FCGX_FPrintF(r->out, "Content-Type: text/event-stream\r\n"); FCGX_FPrintF(r->out, "Cache-Control: no-cache\r\n"); FCGX_FPrintF(r->out, "Connection: keep-alive\r\n"); FCGX_FPrintF(r->out, "X-Accel-Buffering: no\r\n"); FCGX_FPrintF(r->out, "\r\n"); - FCGX_FPrintF(r->out, "Here is output\r\n"); - FCGX_FPrintF(r->out, "\r\n"); FCGX_FFlush(r->out); - sync(); - sleep(1); - FCGX_FPrintF(r->out, "Here is output 2\r\n"); + *sp = s; + ok: retval = 0; - // done: + done: + if (xret) + xml_free(xret); + if (cb) + cbuf_free(cb); return retval; } @@ -94,12 +219,33 @@ restconf_stream(clicon_handle h, #include "restconf_lib.h" #include "restconf_stream.h" +int +stream_timeout(int s, + void *arg) +{ + struct timeval t; + struct timeval t1; + FCGX_Request *r = (FCGX_Request *)arg; + + clicon_debug(1, "%s", __FUNCTION__); + if (FCGX_GetError(r->out) != 0) /* break loop */ + clicon_exit_set(); + else{ + gettimeofday(&t, NULL); + t1.tv_sec = 1; t1.tv_usec = 0; + timeradd(&t, &t1, &t); + event_reg_timeout(t, stream_timeout, arg, "Stream timeout"); + } + return 0; +} + /*! Process a FastCGI request * @param[in] r Fastcgi request handle */ int api_stream(clicon_handle h, - FCGX_Request *r) + FCGX_Request *r, + char *streampath) { int retval = -1; char *path; @@ -113,28 +259,18 @@ api_stream(clicon_handle h, cbuf *cb = NULL; char *data; int authenticated = 0; - char *media_accept; - char *media_content_type; int pretty; - int parse_xml = 0; /* By default expect and parse JSON */ - int use_xml = 0; /* By default use JSON */ + int use_xml = 1; /* default */ cbuf *cbret = NULL; cxobj *xret = NULL; cxobj *xerr; - event_stream_t *es; + int s=-1; clicon_debug(1, "%s", __FUNCTION__); path = FCGX_GetParam("REQUEST_URI", r->envp); query = FCGX_GetParam("QUERY_STRING", r->envp); pretty = clicon_option_bool(h, "CLICON_RESTCONF_PRETTY"); - /* get xml/json in put and output */ - media_accept = FCGX_GetParam("HTTP_ACCEPT", r->envp); - if (media_accept && strcmp(media_accept, "application/yang-data+xml")==0) - use_xml++; - media_content_type = FCGX_GetParam("HTTP_CONTENT_TYPE", r->envp); - if (media_content_type && - strcmp(media_content_type, "application/yang-data+xml")==0) - parse_xml++; + test(r, 1); if ((pvec = clicon_strsep(path, "/", &pn)) == NULL) goto done; /* Sanity check of path. Should be /stream/ */ @@ -146,11 +282,11 @@ api_stream(clicon_handle h, retval = notfound(r); goto done; } - if (strcmp(pvec[1], RESTCONF_STREAM)){ + if (strcmp(pvec[1], streampath)){ retval = notfound(r); goto done; } - test(r, 1); + if ((method = pvec[2]) == NULL){ retval = notfound(r); goto done; @@ -158,6 +294,7 @@ api_stream(clicon_handle h, clicon_debug(1, "%s: method=%s", __FUNCTION__, method); if (str2cvec(query, '&', '=', &qvec) < 0) goto done; + if (str2cvec(path, '/', '=', &pcvec) < 0) /* rest url eg /album=ricky/foo */ goto done; /* data */ @@ -167,7 +304,6 @@ api_stream(clicon_handle h, clicon_debug(1, "%s DATA=%s", __FUNCTION__, data); if (str2cvec(data, '&', '=', &dvec) < 0) goto done; - /* If present, check credentials. See "plugin_credentials" in plugin * See RFC 8040 section 2.5 */ @@ -191,12 +327,22 @@ api_stream(clicon_handle h, goto ok; } clicon_debug(1, "%s auth2:%d %s", __FUNCTION__, authenticated, clicon_username_get(h)); - if ((es = stream_find(h, method)) == NULL){ - retval = notfound(r); + if (restconf_stream(h, r, method, pretty, use_xml, &s) < 0) goto done; + if (s != -1){ + /* Listen to backend socket */ + if (event_reg_fd(s, + restconf_stream_cb, + (void*)r, + "stream socket") < 0) + goto done; + /* Poll upstream errors */ + stream_timeout(0, (void*)r); + /* Start loop */ + event_loop(); + event_unreg_fd(s, restconf_stream_cb); + clicon_exit_reset(); } - if (restconf_stream(h, r, es) < 0) - goto done; ok: retval = 0; done: diff --git a/apps/restconf/restconf_stream.h b/apps/restconf/restconf_stream.h index 7fd5ac4a..bae59da6 100644 --- a/apps/restconf/restconf_stream.h +++ b/apps/restconf/restconf_stream.h @@ -39,6 +39,6 @@ /* * Prototypes */ -int api_stream(clicon_handle h, FCGX_Request *r); +int api_stream(clicon_handle h, FCGX_Request *r, char *streampath); #endif /* _RESTCONF_STREAM_H_ */ diff --git a/doc/FAQ.md b/doc/FAQ.md index f5f5ba9f..b21a623f 100644 --- a/doc/FAQ.md +++ b/doc/FAQ.md @@ -180,20 +180,25 @@ Example: ## How do I use notifications? -The example has a prebuilt notification stream called "NETCONF" that triggers every 5s. -You enable the notification either via the cli: +The example has a prebuilt notification stream called "EXAMPLE" that triggers every 5s. +You enable the notification via the CLI: ``` cli> notify cli> ``` -or via netconf: +or via NETCONF: ``` clixon_netconf -qf /usr/local/etc/example.xml -NETCONF]]>]]> +EXAMPLE]]>]]> ]]>]]> 2018-09-30T12:44:59.657276faultEthernet0major]]>]]> ... ``` +or via restconf: +``` + curl -H "Accept: text/event-stream" -s -X GET http://localhost/streams/EXAMPLE +``` +Consult (../apps/restconf/README.md) on more information on how to setup a reverse proxy for restconf streams. It is also possible to configure a pub/sub system such as (Nginx Nchan)[https://nchan.io]. ## How should I start the backend daemon? diff --git a/example/README.md b/example/README.md index 11ed13cf..a9f782c3 100644 --- a/example/README.md +++ b/example/README.md @@ -76,10 +76,10 @@ Send restconf command ]]>]]> ``` -## Creating notification +## Streams -The example has an example notification triggering every 10s. To start a notification -stream in the session, create a subscription: +The example has an EXAMPLE stream notification triggering every 5s. To start a notification +stream in the session using netconf, create a subscription: ``` ROUTING]]>]]> ]]>]]> @@ -95,6 +95,8 @@ Routing notification ... ``` +Restconf support is also supported, see [../apps/restconf/README.md]. + ## Initializing a plugin The example includes a restonf, netconf, CLI and two backend plugins. diff --git a/lib/clixon/clixon_event.h b/lib/clixon/clixon_event.h index 626a4b27..99b598e2 100644 --- a/lib/clixon/clixon_event.h +++ b/lib/clixon/clixon_event.h @@ -43,6 +43,8 @@ */ int clicon_exit_set(void); +int clicon_exit_reset(void); + int clicon_exit_get(void); int event_reg_fd(int fd, int (*fn)(int, void*), void *arg, char *str); diff --git a/lib/src/clixon_event.c b/lib/src/clixon_event.c index 1569f69f..d15e12c5 100644 --- a/lib/src/clixon_event.c +++ b/lib/src/clixon_event.c @@ -75,6 +75,7 @@ struct event_data{ /* * Internal variables + * XXX consider use handle variables instead of global */ static struct event_data *ee = NULL; static struct event_data *ee_timers = NULL; @@ -86,7 +87,7 @@ static int _clicon_exit = 0; /*! For signal handlers: instead of doing exit, set a global variable to exit * Status is then checked in event_loop. - * Note it maybe would be better to do use on a handle basis, bit a signal + * Note it maybe would be better to do use on a handle basis, but a signal * handler is global */ int @@ -96,6 +97,15 @@ clicon_exit_set(void) return 0; } +/*! Set exit to 0 + */ +int +clicon_exit_reset(void) +{ + _clicon_exit = 0; + return 0; +} + /*! Get the status of global exit variable, usually set by signal handlers */ int @@ -274,6 +284,8 @@ event_poll(int fd) /*! Dispatch file descriptor events (and timeouts) by invoking callbacks. * There is an issue with fairness that timeouts may take over all events * One could try to poll the file descriptors after a timeout? + * @retval 0 OK + * @retval -1 Error: eg select, callback, timer, */ int event_loop(void) diff --git a/lib/src/clixon_proto_client.c b/lib/src/clixon_proto_client.c index 2798e42b..9de878c0 100644 --- a/lib/src/clixon_proto_client.c +++ b/lib/src/clixon_proto_client.c @@ -151,8 +151,14 @@ clicon_rpc_msg(clicon_handle h, * Want to go over to use netconf directly between client and server,... * @param[in] h clicon handle * @param[in] xmlstr XML netconf tree as string - * @param[out] xret Return XML netconf tree, error or OK + * @param[out] xret Return XML netconf tree, error or OK (need to be freed) * @param[out] sp Socket pointer for notification, otherwise NULL + * @code + * cxobj *xret = NULL; + * if (clicon_rpc_netconf(h, "", &xret, NULL) < 0) + * err; + * xml_free(xret); + * @endcode * @see clicon_rpc_netconf_xml xml as tree instead of string */ int @@ -181,6 +187,14 @@ clicon_rpc_netconf(clicon_handle h, * @param[in] xml XML netconf tree * @param[out] xret Return XML netconf tree, error or OK * @param[out] sp Socket pointer for notification, otherwise NULL + * @code + * cxobj *xret = NULL; + * int s; + * if (clicon_rpc_netconf_xml(h, x, &xret, &s) < 0) + * err; + * xml_free(xret); + * @endcode + * @see clicon_rpc_netconf xml as string instead of tree */ int diff --git a/lib/src/clixon_stream.c b/lib/src/clixon_stream.c index 158bbe97..784a5556 100644 --- a/lib/src/clixon_stream.c +++ b/lib/src/clixon_stream.c @@ -149,6 +149,7 @@ stream_get_xml(clicon_handle h, { event_stream_t *es = NULL; char *url_prefix; + char *stream_path; cprintf(cb, ""); for (es=clicon_stream(h); es; es=es->es_next){ @@ -160,9 +161,10 @@ stream_get_xml(clicon_handle h, if (access){ cprintf(cb, ""); cprintf(cb, "xml"); - url_prefix = clicon_option_str(h, "CLICON_STREAM_URL_PREFIX"); - cprintf(cb, "%s/%s", - url_prefix, es->es_name); + url_prefix = clicon_option_str(h, "CLICON_STREAM_URL"); + stream_path = clicon_option_str(h, "CLICON_STREAM_PATH"); + cprintf(cb, "%s/%s/%s", + url_prefix, stream_path, es->es_name); cprintf(cb, ""); } cprintf(cb, ""); @@ -190,8 +192,7 @@ stream_del() * @param[in] fn Callback when event occurs * @param[in] arg Argument to use with callback. Also handle when deleting * @retval 0 OK - * @retval -1 Error - * XXX: from subscription_add and client_subscription_add + * @retval -1 Error, ie no such stream */ int stream_cb_add(clicon_handle h, @@ -502,8 +503,8 @@ stream_publish_cb(clicon_handle h, clicon_err(OE_XML, errno, "cbuf_new"); goto done; } - if ((pub_prefix = clicon_option_str(h, "CLICON_STREAM_PUB_PREFIX")) == NULL){ - clicon_err(OE_CFG, ENOENT, "CLICON_STREAM_PUB_PREFIX not defined"); + if ((pub_prefix = clicon_option_str(h, "CLICON_STREAM_PUB")) == NULL){ + clicon_err(OE_CFG, ENOENT, "CLICON_STREAM_PUB not defined"); goto done; } diff --git a/test/lib.sh b/test/lib.sh index 4404bbee..00cb63c9 100755 --- a/test/lib.sh +++ b/test/lib.sh @@ -176,24 +176,24 @@ expectwait(){ input=$2 expect=$3 wait=$4 - + # Do while read stuff echo timeout > /tmp/flag ret="" sleep $wait | cat <(echo $input) -| $cmd | while [ 1 ] ; do - read r + read -t 20 r # echo "r:$r" ret="$ret$r" match=$(echo "$ret" | grep -Eo "$expect"); if [ -z "$match" ]; then echo error > /tmp/flag - err $expect "$ret" + err "$expect" "$ret" else echo ok > /tmp/flag # only this is OK break; fi done - cat /tmp/flag +# cat /tmp/flag if [ $(cat /tmp/flag) != "ok" ]; then cat /tmp/flag exit diff --git a/test/test_event.sh b/test/test_stream.sh similarity index 72% rename from test/test_event.sh rename to test/test_stream.sh index ceede45f..3225bd2f 100755 --- a/test/test_event.sh +++ b/test/test_stream.sh @@ -32,6 +32,9 @@ cat < $cfg true true true + streams + https://localhost + http://localhost/pub EOF @@ -109,7 +112,33 @@ expectfn "curl -s -X GET http://localhost/restconf/data/ietf-restconf-monitoring new "restconf subscribe RFC8040 Sec 6.3, get location" expectfn "curl -s -X GET http://localhost/restconf/data/ietf-restconf-monitoring:restconf-state/streams/stream=EXAMPLE/access=xml/location" 0 '{"location": "https://localhost/streams/EXAMPLE"}' +# Restconf stream subscription RFC8040 Sec 6.3 - Native solution +new "restconf monitor event nonexist stream" +expectwait 'curl -s -X GET -H "Accept: text/event-stream" -H "Cache-Control: no-cache" -H "Connection: keep-alive" http://localhost/streams/NOTEXIST' 0 'invalid-valueapplicationerrorNo such stream' 2 + +# Need manual testing +new "restconf monitor streams native NEEDS manual testing" if false; then + # url -H "Accept: text/event-stream" http://localhost/streams/EXAMPLE + # Expect: + # data: 2018-10-21T19:22:11.381827faultEthernet0major + # + # data: 2018-10-21T19:22:16.387228faultEthernet0major + +new "restconf monitor event ok stream" +expectwait 'curl -s -X GET -H "Accept: text/event-stream" -H "Cache-Control: no-cache" -H "Connection: keep-alive" http://localhost/streams/EXAMPLE' 0 'foo' 2 +fi +# Restconf stream subscription RFC8040 Sec 6.3 - Nginx nchan solution +# Need manual testing +new "restconf monitor streams nchan NEEDS manual testing" +if false; then + # url -H "Accept: text/event-stream" http://localhost/streams/EXAMPLE + # Expect: + echo foo +fi +# Netconf stream subscription +# Switch here since subscriptions takes time +if true; then new "netconf EXAMPLE subscription" expectwait "$clixon_netconf -qf $cfg -y $fyang" 'EXAMPLE]]>]]>' '^]]>]]>20' 5 @@ -118,10 +147,10 @@ expectwait "$clixon_netconf -qf $cfg -y $fyang" "EXAMPLE]]>]]>" '^]]>]]>20' 5 -fi -#new "restconf monitor event stream RFC8040 Sec 6.3" -#expectfn "curl -H \"Accept: text/event-stream\" -s -X GET http://localhost/streams/EXAMPLE" 0 '2018-10-14T14:17:50.875370faultEthernet0major' +new "netconf NONEXIST subscription" +expectwait "$clixon_netconf -qf $cfg -y $fyang" 'NONEXIST]]>]]>' '^invalid-valueapplicationerrorNo such stream]]>]]>$' 5 +fi new "Kill restconf daemon" sudo pkill -u www-data clixon_restconf diff --git a/yang/Makefile.in b/yang/Makefile.in index ae2a69b2..b595b5ea 100644 --- a/yang/Makefile.in +++ b/yang/Makefile.in @@ -40,7 +40,7 @@ datarootdir = @datarootdir@ CLIXON_DATADIR = @CLIXON_DATADIR@ -YANGSPECS = clixon-config@2018-04-30.yang +YANGSPECS = clixon-config@2018-10-21.yang YANGSPECS += ietf-netconf@2011-06-01.yang YANGSPECS += ietf-netconf-acm@2018-02-14.yang YANGSPECS += ietf-inet-types@2013-07-15.yang diff --git a/yang/clixon-config@2018-04-30.yang b/yang/clixon-config@2018-04-30.yang index 03783e59..d395e543 100644 --- a/yang/clixon-config@2018-04-30.yang +++ b/yang/clixon-config@2018-04-30.yang @@ -38,9 +38,9 @@ module clixon-config { ***** END LICENSE BLOCK *****"; - revision 2018-09-30 { + revision 2018-04-30 { description - "Aligned to Clixon 3.8.0"; + "Released with Clixon 3.6"; } typedef startup_mode{ description @@ -117,16 +117,6 @@ module clixon-config { } } container config { - leaf-list CLICON_FEATURE { - description - "Supported features as used by YANG feature/if-feature - value is: :, where and - are either names, or the special character '*'. - *:an* means enable all features - :* means enable all features in the specified module - *: means enable the specific feature in all modules"; - type string; - } leaf CLICON_CONFIGFILE{ type string; description @@ -359,54 +349,5 @@ module clixon-config { type string; description "RFC8341 NACM external configuration file"; } - leaf CLICON_MODULE_LIBRARY_RFC7895 { - type boolean; - default true; - description "Enable RFC 7895 YANG Module library support as state - data. If enabled, module info will appear when doing - netconf get or restconf GET"; - } - leaf CLICON_MODULE_SET_ID { - type string; - default "0"; - description "If RFC 7895 YANG Module library enabled: - Contains a server-specific identifier representing - the current set of modules and submodules. The - server MUST change the value of this leaf if the - information represented by the 'module' list instances - has changed."; - } - leaf CLICON_STREAM_DISCOVERY_RFC5277 { - type boolean; - default false; - description "Enable event stream discovery as described in RFC 5277 - sections 3.2. If enabled, available streams will appear - when doing netconf get or restconf GET"; - } - leaf CLICON_STREAM_DISCOVERY_RFC8040 { - type boolean; - default false; - description "Enable event stream discovery as described in RFC 5277 - sections 3.2. If enabled, available streams will appear - when doing netconf get or restconf GET"; - } - leaf CLICON_STREAM_URL_PREFIX { - type string; - default "https://localhost/streams"; - description "See RFC 8040 Sec 9.3 location leaf: - 'Contains a URL that represents the entry point for - establishing notification delivery via server-sent events.' - Prepend this constant to name of stream. - Example: https://localhost/streams/NETCONF. Note this is the - external URL, not local behind a reverse-proxy"; - } - leaf CLICON_STREAM_PUB_PREFIX { - type string; - default "http://localhost/pub"; - description "For stream publish using eg nchan, the base address - to publish to. - Example: http://localhost/pub/NETCONF. Note this may - be local URL behind reverse-proxy"; - } } } diff --git a/yang/clixon-config@2018-10-21.yang b/yang/clixon-config@2018-10-21.yang new file mode 100644 index 00000000..31718ab2 --- /dev/null +++ b/yang/clixon-config@2018-10-21.yang @@ -0,0 +1,421 @@ +module clixon-config { + + prefix cc; + + organization + "Clicon / Clixon"; + + contact + "Olof Hagsand "; + + description + "Clixon configuration file + ***** BEGIN LICENSE BLOCK ***** + Copyright (C) 2009-2018 Olof Hagsand and Benny Holmgren + + This file is part of CLIXON + + Licensed under the Apache License, Version 2.0 (the \"License\"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an \"AS IS\" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + Alternatively, the contents of this file may be used under the terms of + the GNU General Public License Version 3 or later (the \"GPL\"), + in which case the provisions of the GPL are applicable instead + of those above. If you wish to allow use of your version of this file only + under the terms of the GPL, and not to allow others to + use your version of this file under the terms of Apache License version 2, + indicate your decision by deleting the provisions above and replace them with + the notice and other provisions required by the GPL. If you do not delete + the provisions above, a recipient may use your version of this file under + the terms of any one of the Apache License version 2 or the GPL. + + ***** END LICENSE BLOCK *****"; + + revision 2018-10-21 { + description + "Released in Clixon 3.8"; + } + typedef startup_mode{ + description + "Which method to boot/start clicon backend. + The methods differ in how they reach a running state + Which source database to commit from, if any."; + type enumeration{ + enum none{ + description + "Do not touch running state + Typically after crash when running state and db are synched"; + } + enum init{ + description + "Initialize running state. + Start with a completely clean running state"; + } + enum running{ + description + "Commit running db configuration into running state + After reboot if a persistent running db exists"; + } + enum startup{ + description + "Commit startup configuration into running state + After reboot when no persistent running db exists"; + } + } + } + typedef xmldb_format{ + description + "Format of TEXT xml database format."; + type enumeration{ + enum xml{ + description "Save and load xmldb as XML"; + } + enum json{ + description "Save and load xmldb as JSON"; + } + } + } + typedef cli_genmodel_type{ + description + "How to generate CLI from YANG model, + eg list a{ key x; leaf x; leaf y;}"; + type enumeration{ + enum NONE{ + description "No extra keywords: a "; + } + enum VARS{ + description "Keywords on non-key variables: a y "; + } + enum ALL{ + description "Keywords on all variables: a x y "; + } + } + } + typedef nacm_mode{ + description + "Mode of RFC8341 Network Configuration Access Control Model. + It is unclear from the RFC whether NACM rules are internal + in a configuration (ie embedded in regular config) or external/OOB + in s separate, specific NACM-config"; + type enumeration{ + enum disabled{ + description "NACM is disabled"; + } + enum internal{ + description "NACM is enabled and available in the regular config"; + } + enum external{ + description "NACM is enabled and available in a separate config"; + } + } + } + container config { + leaf-list CLICON_FEATURE { + description + "Supported features as used by YANG feature/if-feature + value is: :, where and + are either names, or the special character '*'. + *:an* means enable all features + :* means enable all features in the specified module + *: means enable the specific feature in all modules"; + type string; + } + leaf CLICON_CONFIGFILE{ + type string; + description + "Location of configuration-file for default values (this file)"; + } + leaf CLICON_YANG_DIR { + type string; + mandatory true; + description + "Location of YANG module and submodule files."; + } + leaf CLICON_YANG_MODULE_MAIN { + type string; + default "clicon"; + description + "Option used to construct initial yang file: + [@]"; + } + leaf CLICON_YANG_MODULE_REVISION { + type string; + description + "Option used to construct initial yang file: + [@]"; + } + leaf CLICON_BACKEND_DIR { + type string; + description + "Location of backend .so plugins. Load all .so + plugins in this dir as backend plugins"; + } + leaf CLICON_BACKEND_REGEXP { + type string; + description + "Regexp of matching backend plugins in CLICON_BACKEND_DIR"; + default "(.so)$"; + } + leaf CLICON_NETCONF_DIR { + type string; + description "Location of netconf (frontend) .so plugins"; + } + leaf CLICON_RESTCONF_DIR { + type string; + description + "Location of restconf (frontend) .so plugins. Load all .so + plugins in this dir as restconf code plugins"; + } + leaf CLICON_RESTCONF_PATH { + type string; + default "/www-data/fastcgi_restconf.sock"; + description + "FastCGI unix socket. Should be specified in webserver + Eg in nginx: fastcgi_pass unix:/www-data/clicon_restconf.sock"; + } + leaf CLICON_RESTCONF_PRETTY { + type boolean; + default true; + description + "Restconf return value pretty print. + Restconf clients may add HTTP header: + Accept: application/yang-data+json, or + Accept: application/yang-data+xml + to get return value in XML or JSON. + RFC 8040 examples print XML and JSON in pretty-printed form. + Setting this value to false makes restconf return not pretty-printed + which may be desirable for performance or tests"; + } + leaf CLICON_CLI_DIR { + type string; + description + "Location of cli frontend .so plugins. Load all .so + plugins in this dir as CLI object plugins"; + } + leaf CLICON_CLISPEC_DIR { + type string; + description + "Location of frontend .cli cligen spec files. Load all .cli + files in this dir as CLI specification files"; + } + leaf CLICON_CLISPEC_FILE { + type string; + description "Specific frontend .cli cligen spec file."; + } + leaf CLICON_CLI_MODE { + type string; + default "base"; + description + "Startup CLI mode. This should match a CLICON_MODE set in + one of the clispec files"; + } + leaf CLICON_CLI_GENMODEL { + type int32; + default 1; + description + "Generate code for CLI completion of existing db symbols. + Example: Add name=\"myspec\" in datamodel spec and reference + as @myspec"; + } + leaf CLICON_CLI_GENMODEL_COMPLETION { + type int32; + default 1; + description "Generate code for CLI completion of existing db symbols"; + } + leaf CLICON_CLI_GENMODEL_TYPE { + type cli_genmodel_type; + default "VARS"; + description "How to generate and show CLI syntax: VARS|ALL"; + } + leaf CLICON_CLI_VARONLY { + type int32; + default 1; + description + "Dont include keys in cvec in cli vars callbacks, + ie a & k in 'a k ' ignored"; + } + leaf CLICON_CLI_LINESCROLLING { + type int32; + default 1; + description + "Set to 0 if you want CLI to wrap to next line. + Set to 1 if you want CLI to scroll sideways when approaching + right margin"; + } + leaf CLICON_SOCK_FAMILY { + type string; + default "UNIX"; + description + "Address family for communicating with clixon_backend + (UNIX|IPv4|IPv6)"; + } + leaf CLICON_SOCK { + type string; + mandatory true; + description + "If family above is AF_UNIX: Unix socket for communicating + with clixon_backend. If family is AF_INET: IPv4 address"; + } + leaf CLICON_SOCK_PORT { + type int32; + default 4535; + description + "Inet socket port for communicating with clixon_backend + (only IPv4|IPv6)"; + } + leaf CLICON_SOCK_GROUP { + type string; + default "clicon"; + description "Group membership to access clixon_backend unix socket"; + } + leaf CLICON_BACKEND_PIDFILE { + type string; + mandatory true; + description "Process-id file of backend daemon"; + } + leaf CLICON_AUTOCOMMIT { + type int32; + default 0; + description + "Set if all configuration changes are committed automatically + on every edit change. Explicit commit commands unnecessary"; + } + leaf CLICON_XMLDB_DIR { + type string; + mandatory true; + description + "Directory where \"running\", \"candidate\" and \"startup\" are placed"; + } + leaf CLICON_XMLDB_PLUGIN { + type string; + mandatory true; + description + "XMLDB datastore plugin filename + (see datastore/ and clixon_xml_db.[ch])"; + } + leaf CLICON_XMLDB_CACHE { + type boolean; + default true; + description + "XMLDB datastore cache. + If set, XML candidate/running parsed tree is stored in memory + If not set, candidate/running is always accessed via disk."; + } + leaf CLICON_XMLDB_FORMAT { + type xmldb_format; + default xml; + description "XMLDB datastore format."; + } + leaf CLICON_XMLDB_PRETTY { + type boolean; + default true; + description + "XMLDB datastore pretty print. + If set, insert spaces and line-feeds making the XML/JSON human + readable. If not set, make the XML/JSON more compact."; + } + leaf CLICON_XML_SORT { + type boolean; + default true; + description + "If set, sort XML lists and leaf-lists alphabetically and uses binary + search. Unless ordered-by user is used. + Only works for Yang specified XML. + If not set, all lists accessed via linear search."; + } + leaf CLICON_USE_STARTUP_CONFIG { + type int32; + default 0; + description + "Enabled uses \"startup\" configuration on boot. It is called + startup_db and exists in XMLDB_DIR. + NOTE: Obsolete with 1.3.3 and CLICON_STARTUP_MODE"; + } + leaf CLICON_STARTUP_MODE { + type startup_mode; + description "Which method to boot/start clicon backend"; + } + leaf CLICON_TRANSACTION_MOD { + type boolean; + default false; + description "If set, modifications in validation and commit + callbacks are written back into the datastore"; + } + leaf CLICON_NACM_MODE { + type nacm_mode; + default disabled; + description "RFC8341 network access configuration control model + (NACM) mode: disabled, in regular (internal) config + or separate external file given by CLICON_NACM_FILE"; + } + leaf CLICON_NACM_FILE { + type string; + description "RFC8341 NACM external configuration file"; + } + leaf CLICON_MODULE_LIBRARY_RFC7895 { + type boolean; + default true; + description "Enable RFC 7895 YANG Module library support as state + data. If enabled, module info will appear when doing + netconf get or restconf GET"; + } + leaf CLICON_MODULE_SET_ID { + type string; + default "0"; + description "If RFC 7895 YANG Module library enabled: + Contains a server-specific identifier representing + the current set of modules and submodules. The + server MUST change the value of this leaf if the + information represented by the 'module' list instances + has changed."; + } + leaf CLICON_STREAM_DISCOVERY_RFC5277 { + type boolean; + default false; + description "Enable event stream discovery as described in RFC 5277 + sections 3.2. If enabled, available streams will appear + when doing netconf get or restconf GET"; + } + leaf CLICON_STREAM_DISCOVERY_RFC8040 { + type boolean; + default false; + description "Enable event stream discovery as described in RFC 5277 + sections 3.2. If enabled, available streams will appear + when doing netconf get or restconf GET"; + } + leaf CLICON_STREAM_PATH { + type string; + default "streams"; + description "Stream path appended to CLICON_STREAM_URL to form + stream subscription URL."; + } + leaf CLICON_STREAM_URL { + type string; + default "https://localhost"; + description "Prepend this to CLICON_STREAM_PATH to form URL. + See RFC 8040 Sec 9.3 location leaf: + 'Contains a URL that represents the entry point for + establishing notification delivery via server-sent events.' + Prepend this constant to name of stream. + Example: https://localhost/streams/NETCONF. Note this is the + external URL, not local behind a reverse-proxy. + Note that -s command-line option to clixon_restconf + should correspond to last path of url (eg 'streams')"; + } + leaf CLICON_STREAM_PUB { + type string; + default "http://localhost/pub"; + description "For stream publish using eg nchan, the base address + to publish to. + Example: http://localhost/pub/NETCONF. Note this may + be local URL behind reverse-proxy"; + } + } +}