We are working with the MPAT project, http://mpat.eu

The Telecom ParisTech team has a few specific contributions on github:

1. mpat-explorer plugin, to explore the depths and details of the MPAT instance
2. mpat-importexport plugin, to import and export pages and exchange them between content editors
3. mpat-getinfofrompage, to recreate an importable package from a published MPAT page
4. mpat-tvdebugserver, to help with debugging MPAT application even on relatively old TVs
5. mbop-remover, to help with a bug relevant for old MPAT users

There is an updated version of my NSD implementation on github, tagged as v2.

What’s new is:

• v2 follows the latest draft of NSD
  • in particular, v2 uses promises (from cujojs/when.js)
  • it even uses promises for onserviceavailable and onserviceunavailable, which is sort of in advance of the latest draft
• v2 uses the require.js packaging model for JS modules
• v2 was cleaned up and refactored somewhat, in particular previous fixes have been better integrated and function names have been rationalized, and logging can be configured from a properties file

There is a new version of my NSD API implementation on github.

New features are:

• implementation on top of socket.io in addition to WebSocket, so as to allow the use with some smart TVs:
  • start org.coltram.nsd.websocket.ConsoleApplication for a WebSocket-based service, and org.coltram.nsd.socketio.ConsoleApplication for the socket.io based version.
  • the webapps are almost identical: for WebSocket, include script nsdLib.js; for socket.io, include socket.io.js and nsdLibSocketIO.js, and that is it.
• implementation of a UPnP-style event mechanism, on top of both UPnP and Bonjour
• exposing a service is simpler as there is no more need to provide both an interface description and a service implementation. The service implementation is automatically analysed to create an interface description.

There are new samples for all variants. The names are self-explanatory:

• callBonjour.socketio.html
• callBonjour.websocket.html
• callBonjourEvent.socketio.html
• callUPnP.socketio.html
• callUPnP.websocket.html
• callUPnPEvent.socketio.html
• discovery.socketio.html
• discovery.websocket.html
• exposeBonjour.socketio.html
• exposeBonjour.websocket.html
• exposeBonjourEvent.socketio.html
• exposeUPnP.socketio.html
• exposeUPnP.websocket.html
• exposeUPnPEvent.socketio.html

The socket.io library used is netty-socketio (MIT license).

Here are binary versions: websocket and socket.io.

Enjoy.

Abstract

This post describes my implementation of Network Service Discovery API for any modern browser. At this time, there is the bare minimum information, since the target audience is specialists.

Introduction

I have implemented the Network Service Discovery API [link] on top of UPnP and Bonjour as a Java proxy for any modern browser, coupled with a JS library. The JS library handles the connection with the Java proxy through WebSocket and JSON messages.

This has been tested on PCs and Macs, with recent versions of Chrome, Firefox, Safari and Opera. It also works on Windows 7/IE10. Linux should not be a problem, but was not tested.

Compliance

As far as I understand, the UPnP implementation of NSD is compliant but for the name field which I chose not to implement as the UPnP serviceId, because that does not work in practice, but as the concatenation of the important part of the service type (after :service: and before :1) with the device friendly name. If the device friendly name is not included somewhere in one of the fields of NetworkService, there is no way to present to the user a meaningful list of services, i.e. a list which allows to discriminate services of similar types running on different devices.

The Bonjour implementation is compliant but for the config field, which is set to the device friendly name because I do not have access to the string value mentioned in NSD (or I did not understand the reference).

Extensions: NSD++

A messaging API is implemented in a way that is completely independent from the underlying protocol. It matches the UPnP messaging capability but does not cover events. When a service is discovered, NSD++ can “bind” to that service and returns a service implementation object, which contains one function per service action.

service = NSDPlusPlus.bindService(networkServices[i].id)
...
service.lightSwitch(true); // from Example 2, call action lightSwitch

For UPnP, the extended messaging is implemented on top of UPnP messaging. It is compatible with any UPnP service, even from outside.

For Bonjour, the extended messaging is implemented on top of TCP Sockets and JSON. It is restricted to communicating with Bonjour services advertized by NSD++ web apps.

A service advertizing API is implemented with a service description similar to SDCP but in JSON. An example follows:

var localService = {
 "type": "communicationtest",
 "protocol": "upnp",
 "actionList": [{ 
   "name": "lightSwitch", 
   "args": [{ "name": "newTargetValue", "dir": "in" },
            { "name": "response", "dir": "out" }]
 }]};

For UPnP, the JSON description is translated to SDCP.

For Bonjour, the JSON description is used as is: it is placed in a web server, and its URL is stored in the Desc property of the DNS record of the service.

Installing the Software

Here is a link to the executable jar for the NSD proxy. Run the agent in a terminal with the command line:

java -jar nsd-agent.jar

Running the agent in the terminal is necessary if you want to see the error log. Run the agent before you load any of the examples in a browser.

Here is a link to the examples. Unpack the zip in the document root of a web server. In the following, I assume the URL for the examples is “http://localhost/nsd/”.

Compiling the Software

The project structure is compatible with Maven, and a pom.xml is provided.

Example 1: Discovery @ http://localhost/nsd/discovery.html

The discovery example allows you to discover services. There is a service type field in which you enter the type of service you want to look for. Then you choose to press the blue “Get” button for an exact match of the type you entered, or the orange “get*” button to match a substring of the actual service type. The blue “Get” button triggers NSD.getNetworkServices (the standard API). The entered service type has to start with “upnp:” for a UPnP service, and “zeroconf:” for a Bonjour service.

The orange “Get*” button triggers an extension of this API that cannot be implemented directly with the standard API. I believe the extended API is more easily usable for web application authors and allows the design of web applications that are independent of whether the actual service is implemented in UPnP or Bonjour.

Example 2: Expose a Bonjour Service @ http://localhost/nsd/exposeBonjour.html

This web application shows a light bulb (initially off) and expose a Bonjour service with one lightSwitch action. The service type is zeroconf:_communicationtest._tcp.local.

Example 3: Expose a UPnP Service @ http://localhost/nsd/exposeUPnP.html

This web application shows a light bulb (initially off) and expose a UPnP service with one lightSwitch action, which sends a reply in the form of the time when the action was executed.

The service type is upnp:urn:coltram-org:service:communicationtest:1

Example 4: Call a Bonjour Service @ http://localhost/nsd/callBonjour.html

This web application shows two buttons to control the Bonjour service of Example 2.

Example 5: Call a UPnP Service @ http://localhost/nsd/callUPnP.html

This web application shows a text and two buttons to control the UPnP service of Example 3.

The text is “No service found” initially. It changes to “Service found” when the appropriate service has been found. After clicking on one of the two buttons, the UPnP service replies with the time when the action is executed, and that time replaces the text.

Troubleshooting

In all examples, the lower part of the screen is devoted to a debug log.

Messages appears at the top and older messages scroll down.

The first message is “connecting to undefined”: the web application is connecting to the agent. If the agent is not started yet or not accessible, this is the last thing you see.

The second message is “connected” or “UPnP/Bonjour service ‘X’ exposed”: the web app has established a connection to the agent, and NSD can now be used.

Then, in Call* examples, getNetworkServices is called with a callback called CB. When this callback is called, it prints “CB <nb of services>”. Typically, you get a message sequence of “CB 0” (initially, no service found), then “onserviceavailable callback” (NSD tell you one service was found and you need to call getNetworkServices again), then “CB 1” (one service was found).

Other typical problems:

1. If you are on a Wifi network, chances are that UPnP and Bonjour are not allowed. Then you will not be able to discover any service. If you have control over the Wifi configuration, you need to enable the UPnP and Bonjour protocols. If you do not have control over the Wifi, you need to negotiate with the Wifi owner.
2. If you are using a PC with Windows and you are not admin, then you cannot use the agent, unless you get an admin to allow your java installation to open any port (required for UPnP, Bonjour and WebSocket)

Source

This project is open source. Here is a link to a zip of the source code and here is the github link. The license is LGPL v3. 

This NSD agent uses open source libraries:

• Cling, a UPnP implementation from Christian Bauer at Teleal (LGPL).
• JMDNS, a Bonjour implementation from Arthur van Hoff at Strangeberry (Apache v2 License).
• Java_WebSocket, a WebSocket server and client implementation from TooTallNate (MIT License).
• JSON, a JSON codec implementation from json.org (JSON License).
• pygmy, a small web server implementation from Charlie Hubbard (Artistic License).

The examples use Bootstrap (Apache v2 license) and jQuery (MIT License).

Introduction

I have a working prototype for the following scenarii:

• an HTML widget A discovering another HTML widget B on another machine (with different OS): B exposes itself on the network, A discovers B, A calls an action declared by B (changing the color of an HTML element in B).
• an HTML widget discovering and communicating with a Intel Tools test UPnP service: discovering the Network Light and turning it on and off
• an HTML widget discovering and communicating with an MPEG-U Widget (in GPAC), the HTML widget acting as a switch for a virtual light in the MPEG-U Widget
• an MPEG-U Widget (in GPAC) discovering and communicating with an HTML widget , the MPEG-U widget acting as a switch for a virtual light in the HTML Widget

This is an extension of our work on communicating widgets [1][2][3][4] which should work in any recent web browser. The only requirements are: the platform must allow Java applications (hence no iOS), and the web browser needs to implement Web Sockets.

This work started with my work within the W3C Home Network Task Force. We worked on requirements for a platform allowing the convergence of the TV world with the Internet world, opening the TV to the Internet, and making multi-screen scenarii possible. After creating a set of requirements [5] in the task force, we “passed the buck” to the W3C Device API WG. The Device API group is working on two technologies that may answer our requirements: Web Intents [6] and Discovery API [7].

Actually, neither of these technologies is really solving my problem. I need more than just web applications/widgets and these technologies. I need:

•  a way to package web applications and exchange them on the net: W3C Widgets (check)
• a way to discover devices, services, and possibly other web applications: Discovery API (check), but Web Intents does not fullfil this (needs a discovery proxy of some sort)
• a way to exchange messages between web applications and services: it is possible to implement the full UPnP messaging protocol (SOAP and all) in JS and use Web Sockets to communicate with the UPnP services; however, since the UPnP proxy is already required for discovery, and already implement UPnP messaging, it seems a better idea to use the UPnP proxy for messaging.
• a way for web applications to expose themselves as services, in order to be discovered by other web applications or services: not provided.

Principle

The principle is:

• there is a UPnP proxy running beside the web browser. The proxy is implemented as an external Java application. It handles the UPnP discovery, communication with UPnP services, and service advertising. The Java application also handles Web Sockets connections to/from the HTML page.
• there is a JS library handling the communication from the HTML page to the external UPnP proxy through Web Sockets.

API

Communication

Architecture of the solution

Code

Source code and test files and jars are there.

The proxy uses the Cling UPnP library from teleal (LGPL), and the Web Sockets Library from http://java-websocket.org/ (MIT license), as well as a JSON Java library from json.org.

Tests

I have tested the following configurations:

• HTML widget to HTML widget on the same machine (sharing the same proxy) : to test that, run the console proxy and load testWidgets/coltram1.html and testWidgets/coltram2.html on Chrome on the same machine.
• an HTML widget A discovering another HTML widget B on another machine (with different OS): B exposes itself on the network, A discovers B, A calls an action declared by B (changing the color of an HTML element in B). To test that, run a console proxy and load testWidgets/coltram1.html on one machine, run console proxy and load testWidgets/coltram2.html on another machine.
• an HTML widget discovering and communicating with a Intel Tools test UPnP service: discovering the Network Light and turning it on and off: to test that, run a console proxy and testWidgets/coltram3.html on a machine, and an Intel network light on this or another machine.
• an HTML widget discovering and communicating with an MPEG-U Widget (in GPAC), the HTML widget acting as a switch for a virtual light in the MPEG-U Widget
• an MPEG-U Widget (in GPAC) discovering and communicating with an HTML widget , the MPEG-U widget acting as a switch for a virtual light in the HTML Widget

Complete tests were done with the Chrome navigator. A partial test was successful in Firefox.

References:

[1] J. C. Dufourd, C. Concolato and J. Le Feuvre, SVG Communicating Widgets, SVG Open, Mountain View, CA, USA, October 2009 [PDF]

[2] C. Concolato, J. Le Feuvre and J. C. Dufourd, Declarative Interfaces for Dynamic Widgets Communications, Document Engineering, Munich, Germany, September 2009, pp. 241-244 [PDF] [DOI 10.1145/1600193.1600245]

[3] J. Le Feuvre, C. Concolato and J. C. Dufourd, Widgets Mobility, International Conference on Mobile Technology, Applications and Systems, Nice, France, September 2009 [PDF] [DOI 10.1145/1710035.1710060]

[4] C. Concolato, J. C. Dufourd, J. Le Feuvre, K. Park and J. Song, Communicating and migratable interactive multimedia documents, Multimedia Tools and Applications, May 2011 [PDF] [DOI 10.1007/s11042-011-0805-2]

[5] Requirements for Home Networking Scenarios, W3C Interest Group Note 01 December 2011, http://www.w3.org/TR/hnreq/

[6] Web Intents, W3C Working Draft 26 June 2012, http://www.w3.org/TR/web-intents/

[7] Networked Service Discovery and Messaging, W3C Working Draft 07 August 2012,  http://www.w3.org/TR/discovery-api/

Here is the release of a carousel optimizer used to write the following paper:

J. C. Dufourd, J.-C. Moissinac and J. Le Feuvre, Loading Time Optimization of Broadcast TV Applications, EuroITV, Berlin, Allemagne, July 2012.

Here is a screen shot:

It is written in Java.

The source is here.

The executable jar is here.

When you launch it, it pops a file dialog, select a directory in which all files belong to the HbbTV app you want to optimize, then press “choose”.

It then pops a window like the one above, and lets you fiddle with the different options.

On the left is a resource table. On each line, the file name, radio buttons to choose whether to send the resource in broadband (BB), or in one of the three carousels, the resource size, and the resource contribution to the overal carousel sier in percent then as a blue bar. If a resource is compressed (.gz) its compressed size is used; if not compressed, the software compresses it in memory, and uses the compressed size, to simulate the behaviour of the DSM-CC carousel.

On the right, you can edit:

• the bandwidth, in kilobits/s
• the P2 multiple, defined as “resources in the carousel 1 are sent P2 times more often than resources in the carousel 2”
• the P3 multiple, defined as “resources in the carousel 1 are sent P3 times more often than resources in the carousel 3”

You can also choose:

• to sort the resource table by size or by contribution to the overall carousel size
• a policy for the optimizer tool
• only one carousel, all resources in carousel 1, P2 and P3 ignored
• two carousels, with images by default put in carousel 2, P3 is ignored
• script caching policy, as described in the article referenced above
• any manual fiddling places the optimizer in the customized state

On the right top is a table with the current status, size of each carousel, number of resources in each carousel, max waiting time for a resource in each carousel.

On the right bottom:

• image size is the total size of images not flagged as BB,
• script size is the total size of scripts not flagged as BB,
• and below are the max waiting times of the fast carousel for each of the implemented policies.

This software is released under the LGPL v2 license.

Ideas for this software were developed during the openHbb project.

A destination des élèves du projet PACT, voici un petit exemple de lecture d’un son issu de calculs (plutôt que lu à partir d’un fichier) ainsi que de capture de son.

I am gathering here all relevant references about our work here on HbbTV, the Home Network, usage of second/third screens, communicating widgets, etc.

Standards

HbbTV, Hybrid Broadband Broadcast TV, ETSI TS 102 796

The HbbTV Consortium web site

W3C Web and TV Interest Group, working on requirement and use cases for Web and TV convergence within the home network: wiki with in particular home network discussions

openHbb, French national project on implementing HbbTV, players, authoring tools and extensions, commercial and open source.

Papers

J. C. Dufourd, Requirements for Web and TV convergence, Second W3C Web and TV Workshop, Berlin, January 2011 [PDF]. Presentation: PPT

C. Concolato, J. C. Dufourd, J. Le Feuvre, K. Park and J. Song, Communicating and migratable interactive multimedia documents, Multimedia Tools and Applications, May 2011 [PDF].

J. C. Dufourd, C. Concolato and J. Le Feuvre, SVG Communicating Widgets, SVG Open, Mountain View, CA, USA, October 2009 [PDF].

C. Concolato, J. Le Feuvre and J. C. Dufourd, Declarative Interfaces for Dynamic Widgets Communications, Document Engineering, Munich, Germany, September 2009, pp. 241-244 [PDF].

J. Le Feuvre, C. Concolato and J. C. Dufourd, Widgets Mobility, International Conference on Mobile Technology, Applications and Systems, Nice, France, September 2009 [PDF].

Blog entries and videos

The Widgets Project

Video 1: presentation for the “gui” widget manager

Video 2: presentation of the “iphone” widget manager

Video 3: presentation of widgets localization (a W3C widget feature)

Video 4: presentation of the “tv” widget manager

Video 5: presentation of discovery and communication between widgets within a widget manager

Video 6: presentation of widget migration

Video 7: presentation of discovery and communication between widget and native app, or between widgets running in separate players.

To reproduce these demos on your machine, please read this post.

To reproduce the demos shown in the videos, you need a working version of the GPAC player: get the most recent installer for Windows, MacOSX or Windows Mobile here. For another system, please download and compile sources. If you compile GPAC yourself, please make sure that modules platinum and widgetman are included in the build.

Then please load and uncompress this file and this file in a directory.

In that directory, run (on Windows):

MP4Client.exe gui\iphone_wm_gui.svg

Make sure you have MP4Client.exe on your path. Click on “scan”. Navigate to the current directory, then “widgets”, then click on 

The icons of available widgets appear.

On a Mac, you need to do something like:

/Applications/Osmo4.app/Contents/MacOS/Osmo4 ~/test/gui/iphone_wm_gui.svg

if “~/test” is your current directory.

If you have two machines in the same local network (not Wifi, since Wifi stations often filter UPnP broadcasts), you can run two GPAC players on two machines. Otherwise, just open two terminals, and run two instances of the player in the two terminals.

UPnP discovery may take a few seconds to a minute to happen.

The command line ends with the name of the widget manager you want to execute:

• iphone_wm_gui.svg for the “iphone” widget manager
• tv_wm_gui.svg for the “tv” widget manager
• mpegu-wm.xmt for the “gui” widget manager