Eye / Gaze tracking system  Software Developer’s Manual 

Document Version 1.2.3 22 June 2014 xLabs Pty Ltd, Australia http://www.xlabsgaze.com/ http://www.xlabs.com.au/ [email protected] Eye / Gaze tracking system Software Developer’s Manual 1 1. Introduction
This document will assume that you have read the Client Software User Manual. (If not, go and read it…). The purpose of this document is to provide a greater level of technical information for interested users and software developers. A high level of transparency about how the system works is also part of our commitment to user privacy and security. 1.1 Developing under EULA Licencing is designed to encourage widespread adoption. In particular, we provide an End­User Licence Agreement (EULA) at no cost, and charge for Commercial or Developer Licences. You can use this document and the client software to develop your own programs and websites that use our client software. The EULA does not permit you to distribute a software program or provide a service that uses or interacts with our software. When you get to the stage where you wish to launch or deploy your software, you need to contact us to arrange a separate licence. If you create a website or program that uses or interacts with our system on an end­user’s computer, you must obtain a separate licence from xLabs. But don’t be put off: Our intent is very much to help you succeed with your software, and we might even offer to help promote your work. A full set of licences, terms and conditions can be viewed at: http://xlabsgaze.com/license 1.2 Developing for commercial purposes We would like to enable you to provide products and services that utilise our technology at a reasonable price for the scale of your business. Please contact us to discuss commercial licencing: [email protected] 1.3 Table of Contents Section Page 1. Introduction 2 2. Architecture 4 3. Javascript Interface 7 4. HTTP Interface[s] 17 5. How­to 21 2 6. Installer Actions 24 7. Directory Structure 25 8. Transient / Realtime settings 27 9. Administration 30 3 2. Architecture
2.1 Client Software and Browser Extension Due to current security models within web browsers, we need to install a program onto the end­user’s computer, that communicates with the camera when needed. We provide the software as a single installation package. We call this the “xLabs Client software”. The Eye/Gaze tracking system consists of two parts (see figure 1). The first part is the “client software”, which must be installed by the end user on their computer. The second part is a browser extension, that provides the user interface. 2.2 Processes Three processes are involved in the Eye/Gaze tracking system: ● The web browser (with our Browser Extension installed) [currently, Chrome only] ● A persistent web service (implemented as a Daemon, Service or Scheduled Task) ● A transient, realtime process that connects to the camera As the names imply, the persistent web service process runs at startup as a system service or daemon. It consumes minimal resources when not in use. The “transient” or “realtime” process is launched by the persistent process when needed. It connects to the camera and processes images from the camera in realtime (i.e. as fast as images can be captured). In future we will add throttling of this program. The transient / realtime camera program only runs when eye/gaze tracking is needed. At other times the camera is free for use by other programs. 2.3 Ports The processes created by xLabs have HTTP interfaces for configuration and transfer of data. We do not use port 80 (the default for HTTP) because users may have installed other services that require this port. Process Type Port Persistent Java 9522 Transient/realtime Native binary 9523 We picked ports 9522 and 9523 because 9522 = ‘xlab’ on a telephone keypad, and this port range is not listed as having any other purpose. (9523 is 9522 +1, of course). These rules may be a helpful memory aid... 4 Figure 1: The eye/gaze tracking system is installed as two parts: A client software package, and a browser extension that provides the user interface within the web browser. The client software consists of two processes: A persistent web service (invoked as a service or daemon) and a transient program that is invoked on demand by the persistent service. The transient process connects to the camera while running, and performs “realtime” image processing. Both processes have HTTP interfaces on ports 9522 (persistent) and 9523 (transient/realtime). 2.4 Hosts The HTTP interfaces are provided on localhost and therefore can be addressed as localhost or e.g. 127.0.0.1 in IPv4. For user convenience, we add a hosts entry redirecting the host “xlabs” to localhost. This allows users to access the system with URLs like: http://xlabs:9522 2.5 Interfaces All processes are configured by configuration files on disk (at startup) and by HTTP interfaces (during operation). We provide two interfaces for developers to communicate with the system: ● Javascript interface ● HTTP interface[s] We recommend third party systems use either the Javascript interface (provided by the browser extension) or the persistent HTTP interface to interact with the Eye/Gaze tracking system. The Javascript interface is easiest for web developers, because much of the interaction between web­page and the persistent service is already automated. Both interfaces are documented in this document. Section 3 describes the Javascript interface. Section 4 documents the HTTP interfaces. 5 The browser extension performs all actions via calls to the persistent HTTP interface. Therefore, both Javascript and HTTP interfaces can be used simultaneously. All system state is maintained in the persistent HTTP interface (see figure 2). The transient / realtime process communicates with the persistent process via HTTP. However, the transient / realtime process is not always running. With the exception of live video streaming, most developers will not need to communicate directly with the transient/realtime process. Figure 2: Relationship between HTTP and Javascript interfaces. All Javascript functions map to HTTP interface features. All state is maintained via the persistent HTTP interface. Therefore, developers can use the Javascript and HTTP interfaces simultaneously without problems due to replication of system state across interfaces. 2.6 Email vs Username The system has two properties ­ email and username ­ that sound more similar than they really are. We ask the user for an email address to register the system. This email address is then not used elsewhere. We allow multiple usernames to be used on the same computer, but only one copy of the client software has been installed and registered, with a single email address. Usernames are principally used to associate calibration data with specific individuals. In future versions, usernames will also be used to label face­recognition profiles. Third party websites are unable to access the registered email address via the Javascript API to protect user privacy. However, they can access the username to identify which user is present. Third party websites can negotiate sharing of email addresses with users directly. 6 3. Javascript Interface
3.1 Browser Extension The Javascript interface is the recommended method of integrating third­party web­based applications into the Eye/Gaze tracking system. The interface is provided by the xLabs Browser Extension, which users are prompted to install along with the Client Software. Once installed, the browser extension makes some features available to the user (as documented in the user manual) and provides some features to any web page. The browser extension’s content­script will accept messages from the embedding web page, allowing the page to modify the state of the Eye/Gaze tracking system. The page can also add listeners to detect specific events ­ firstly, the presence and setup of the Eye/Gaze tracking system, and secondly, updates to the state of the Eye/Gaze tracking system. The latter are pushed to the page as CustomEvent objects, to reduce the latencies associated with polling. 3.2 Debugging with the TrackMouse feature For debugging while using the xLabs system, there is a secret “trackMouse” feature which replaces gaze­coordinates from the tracking system with gaze­coordinates from the mouse position. The feature can be toggled on and off programmatically via the embedding web page, as described in examples below. 3.3 Simple Example It’s easiest to start with a simple example that should be easy for Javascript developers to follow. The example is provided as a HTML file in the client software install, named: XLABS_HOME/www/sdk­demo.html The contents of this file are as follows: 01<html> 02 <body> 03 <h1>Software Developer Kit ­ Demo page</h1> 04 <p>This page checks for browser extension and marks gaze on screen.</p> 05 <h3 id="target" style=" color: red; margin: 0; padding: 0; 06 position: fixed; left: 200; top: 200; " >X</h3> 07 <script type="text/javascript"> 08 var sdkDemo = { 09 documentOffsetX : 0, // offset of HTML document origin from screen origin 10 documentOffsetY : 0, 11 12 setup : function() { 13 window.addEventListener( "mousemove", function(e) { 14 sdkDemo.documentOffsetX = e.screenX ­ e.clientX ­ window.screenX; 15 sdkDemo.documentOffsetY = e.screenY ­ e.clientY ­ window.screenY; 16 } ); 17 }, 18 7 19 onApiState : function( state ) { 20 var x0 = ­( sdkDemo.documentOffsetX + window.screenX ); 21 var y0 = ­( sdkDemo.documentOffsetY + window.screenY ); 22 var xt = x0 +state.kvTrackingScreenX; 23 var yt = y0 +state.kvTrackingScreenY; 24 25 var targetElement = document.getElementById( "target" ); 26 targetElement.style.left = xt; 27 targetElement.style.top = yt; 28 }, 29 30 onApiReady : function() { 31 window.postMessage( {target:"xLabs", payload:{overlayEnabled:1}}, "*" ); 32 window.postMessage( {target:"xLabs", payload:{overlayMode:0}}, "*" ); 33 window.postMessage( {target:"xLabs", payload:{clicksEnabled:1}}, "*" ); 34 window.postMessage( {target:"xLabs", payload:{trackingEnabled:1}}, "*" ); 35 window.postMessage( {target:"xLabs", payload:{trackMouse:1}}, "*" ); 36 } 37 }; 38 39 document.addEventListener( "xLabsApiReady", function() { 40 sdkDemo.onApiReady(); 41 } ); 42 43 document.addEventListener( "xLabsApiState", function( event ) { 44 sdkDemo.onApiState( event.detail ); 45 } ); 46 47 sdkDemo.setup(); 48 </script> 49 </body> 50</html> Note that the file contains some basic HTML and an inlined Javascript code block. The file has the following functionality: ● It listens for a custom event indicating that the xLabs Javascript API is available ● It listens for custom events providing updated state information from the xLabs Javascript API. ● It sends commands to the Eye/Gaze tracking system to enable & configure it ● It displays the current gaze coordinates on screen as a red ‘X’. Now let’s look at how this functionality was achieved. 3.3.1 Detecting the xLabs API Lines 08 to 37 define an object named sdkDemo. This object wraps up the state and functions we need for our demo. On line 39 we ask the document to add an event listener, defined as an anonymous function. The anonymous function calls sdkDemo.onApiReady(). The anonymous function will be called when the relevant event is received. 8 The listener is asked to listen for the event “xLabsApiReady”. This custom event is sent by the xLabs browser extension’s content­script ONCE after it has finished loading and all setup functions. The browser extension content­script itself loads after the rest of the page has finished loading. When we receive this event, it means that both the page and the Eye/Gaze tracking system are ready for use. If we don’t receive the event, the browser extension and/or the Eye/Gaze tracking system are not installed or not correctly configured. On page load, you may want to display a warning to users, asking them to install the xLabs Client Software. The warning can be hidden when you receive the API ready event. 3.3.2 Sending commands to the Eye/Gaze tracking system The custom event sent by the xLabs Javascript API triggers us to call sdkDemo.onApiReady(). This function is defined starting at line 30. Since the API is now ready, we can use this opportunity to configure the Eye/Gaze tracking system in the way we want. Note we should not send these commands until this time, otherwise unexpected behaviour might result. We can continue to send these commands at any time after the API ready event has been received. Each command is implemented as a window.postMessage() function call. The first argument to the function call is an anonymous javascript object with two properties: “target” and “payload”. The message will be intercepted by the browser extension content­script. The content­script ignores commands that do not have the property “target” with the value “xLabs”. The “payload” property specifies which command we want to apply and any values associated with the command. Section 3.5 provides a list of available commands and values. In this example, we request 5 commands, that turn on the gaze­tracking graphics, set it to toolbar mode, enable capture of clicks and tracking, and turn on the secret mouse­tracking feature described in section 3.2. Note then that this won’t do real gaze tracking, it will just follow the mouse. This is easier to debug. 3.3.3 Getting and displaying gaze data Lines 43­45 add another event listener. This time we listen for “xLabsApiState” messages. When we receive them we will call sdkDemo.onApiState and pass the detail property of the event. The detail contains a complete copy of the Eye/Gaze tracking system state. In sdkDemo.onApiState (lines 19­28) we do two things. First, we work out where the current gaze coordinates are within the document client area. This is slightly complicated because the gaze coordinates (extracted from the state in lines 22 and 23) are screen coordinates. Next, on lines 25­27 we modify the CSS style of our “target” element (which contains an “X”) to move it to the current gaze coordinates. 9 Section 3.6 contains a complete list of the structure of the state object passed as event.detail and used in our sdkDemo.onApiState function. 3.4 Getting real gaze data So far we created a fake­gaze tracking page that tracks the mouse around the screen. This configuration proves that you can successfully use the xLabs API, but doesn’t do real gaze tracking. For real gaze tracking we just need to initialize the system in a different way, by sending a different set of commands after receiving the API ready event. (Note that we assume the user has previously calibrated the system and that it is ready to use). To enable gaze tracking on a previously calibrated system, replace lines 31­35 with: 01 window.postMessage( {target:"xLabs", payload:{overlayEnabled:1}}, "*" ); 02 window.postMessage( {target:"xLabs", payload:{overlayMode:0}}, "*" ); 03 window.postMessage( {target:"xLabs", payload:{clicksEnabled:1}}, "*" ); 04 window.postMessage( {target:"xLabs", payload:{trackingEnabled:1}}, "*" ); 05 window.postMessage( {target:"xLabs", payload:{realtimeEnabled:1}}, "*" ); 06 window.postMessage( {target:"xLabs", payload:{calibrationEnabled:1}}, "*" ); Note that this time we’re asking for the transient / realtime process to be invoked using the “realtimeEnabled” command. We are also asking for the system to be calibrated for use with “calibrationEnabled”. When the realtime process has prepared itself, connected to the camera, detected the face, started tracking and finished calibrating based on historical data, then it will set calibrationEnabled to 0. After this time our page will be tracking user gaze and marking it with an ‘X’. In the sdkDemo.onApiState function we can look for the change in calibrationEnabled if we want to check that everything is ready. 3.5 Table of Javascript commands The following table lists all the commands that can be sent as messages from the page to the content­script. Note that the messages should be formed like this: {target:”xLabs”, payload:{PROPERTY:VALUE}} Property Legal Values Notes overlayEnabled int, 0 or 1 Toggles whether the browser extension features are visible in the web page. overlayMode int, range 0 to 5 inc. Sets the graphics mode for the overlay. Values are: 0: Toolbar only 10 1: Feedback view (camera video) 2: Calibration screen 3: Spotlight mode 4: Sequence mode 5: Heatmap mode overlaySpotlightMode int, range 0 to 3 inc. Sets the way the Spotlight graphics mode is rendered. Possible values: 0: Default searchlight 1: Hotspot 2: Whiteout 3: Laser trackMouse int, 0 or 1 Replaces real gaze calculated from the camera with mouse coordinates. For debugging. trackingScreenX* int (unlimited) If trackMouse is set to 1, will set the gaze coordinate X value to the value provided. trackingScreenY* int (unlimited) If trackMouse is set to 1, will set the gaze coordinate Y value to the value provided. clicksEnabled int, 0 or 1 Toggles whether the browser extension will use mouse events for calibration purposes. captureKeyboard int, 0 or 1 Toggles whether the browser extension will capture user keyboard events or allow them to
pass through to the page below. captureMouse int, 0 or 1 Toggles whether the browser extension will capture user mouse events or allow them to pass through to the page below. realtimeEnabled int, 0 or 1 Toggles activation of the transient / realtime process and associated features. Process is invoked if not running. When value is set to 0, process will be terminated. trackingEnabled int, 0 or 1 Activates tracking and associated features. pinpointEnabled int, 0 or 1 Needed for gaze tracking. If only head pose required, turn this off (0) for less CPU burden. calibrationEnabled int, 0 or 1 Requests a calibration using all available data for the current user. calibrationClear None. Deletes all calibration data associated with current user. 11 validationEnabled int, 0 or 1 Toggles activation of validation module. *Note: trackingScreenX and trackingScreenY must be delivered together as a single payload. 3.6 Format of system state data When receiving system state data from the browser extension, the event.detail object will have properties and structure as detailed in the table below. See section 3.3 for sample code to receive this data. NB: kvXXXX means Key­Value, because all state properties are copies of the persistent Key­Value HTTP service (defined in section 4). Properties can be accessed from the xLabsApiState event by property name, e.g. to access the kvUser property: var user = event.detail.kvUser; Property (Type) Legal Values Notes kvUser (String) Any. Default is “unknown”. kvEmail (String) Value either “OK” or empty. Default is empty string. The system will not operate unless an email is provided. Email value is hidden from page for user security. kvOverlayEnabled (int) 0 or 1 If value is set to 1, browser extension graphics will be overlaid on the web page. kvOverlayMode (int) The graphics mode for the browser extension overlay. Values are: 0: Toolbar only 1: Feedback view (camera video) 2: Calibration screen 3: Spotlight mode 4: Sequence mode 5: Heatmap mode kvOverlaySpotlightMode (int) Sets the way the Spotlight graphics mode is rendered. Possible values: 0: Default searchlight 1: Hotspot 2: Whiteout 3: Laser 12 kvOverlayCaptureKeyboard (int) 0 or 1 Toggles whether the browser extension will capture user keyboard events or allow them to pass through to the page below. kvOverlayCaptureMouse (int) 0 or 1 Toggles whether the browser extension will capture user mouse events or allow them to pass through to the page below. kvTrackMouse (int) 0 or 1 Replace real gaze with mouse pointer location for debugging. kvClicksEnabled (int) 0 or 1 Whether mouse press events are used for calibration. kvClickMinDuration (int) Minimum duration of a mouse press for use in calibration. kvTrackingScreenX (int) Current gaze coordinates on screen. Only valid when realtime is active. kvTrackingScreenY (int) Current gaze coordinates on screen. Only valid when realtime is active. kvTrackingEnabled (int) 0 or 1 Set value to 1 to enable gaze tracking and other face post­processing features. kvPinpointEnabled (int) 0 or 1 Set value to 0 for faster head/face pose but with insufficient accuracy for gaze tracking. kvRealtimeActive (int) 0 or 1 This value is 1 when the realtime process is active, and 0 otherwise. kvRealtimeEnabled (int) 0 or 1 Set value to 1 to enable the realtime
process and get images from camera. Set value to 0 to turn off. kvCameraRecording (int) 0 or 1 Set value to 1 to save frames to disk. Note, this will use a lot of space and slow the computer if used excessively. kvCalibrationEnabled (int) 0 or 1 Set this value to 1 to trigger calibration. Set value to 0 to cancel 13 calibration. If realtimeActive = 1, this
process will set the value to 0 when calibration has completed. kvCalibrationStatus (int) 0 to 3 inclusive. The status of calibration. This value changes constantly. It indicates whether the current user pose is calibrated. Possible values: 0: No calibration 1: Good calibration 2: Uncalibrated current pose 3: Bad calibration kvCalibrationClicks (int) 0 to unbounded The total number of user clicks suitable for calibration that have been logged over all time, for the current user. kvCalibrationClicksUncalibrated (int) 0 to unbounded The number of user clicks suitable for calibration that have been logged
since the last calibration event. kvValidationEnabled (int) 0 or 1 Whether the validation module is enabled. Needs realtimeActive = 1 to work. kvValidationStatus (int) 0=bad or 1=OK Overal status of validation, including
all errors. Easier than testing all error conditions individually. kvValidationErrors (String). A subset of D,B,F,R,U A set of error codes, each 1 character. Empty string means no error. Values are: D: too Dark B: too Bright F: no Face R: face Resolution low U: Uneven lighting kvValidationRegions (String) e.g. Names of the regions in which "Eye[L],Eye[R],Fac statistics are computed to e" determine image quality. kvValidationExcessYaw (int) 0=OK or 1=bad Typically indicates a problem tracking user’s face features if value is 1. 14 kvValidationNoFrames (int) 0=OK or 1=bad Indicates a camera problem if value is 1. kvValidationFaceRect (String) defining a rectangle e.g. "350,197,111,94" The format of the value is x,y,w,h in original camera image coordinates. Value only updates when validation enabled and realtime is active. kvHeadX (float) Unscaled. No canonical reference frame (use for deltas only, to detect movements i.e. changes in pose). kvHeadY (float) Unscaled. See above. kvHeadZ (float) Unscaled. See above. kvHeadRoll (float) Radians. See above. kvHeadPitch (float) Radians. See above. kvHeadYaw (float) Radians. See above. 3.7 Dependencies between features Some dependencies between features mean that not all functionality is available in all configurations of the system. This section will attempt to describe these dependencies. All features dependent on image processing will only function when the transient / realtime process is running. This includes face detection, image validation, gaze tracking, and calibration. 3.7.1 Validation Most of the image validation features, and face­detection, are available whenever images are being received from the camera by the realtime process. Validation is necessary to update the error icons and detect that the image of the user is acceptable. 3.7.2 Tracking The “tracking” feature refers to all post­processing of the detected user face, including gaze­tracking, face­pose and face­gestures. Calibration, although performed only on demand, is only available when tracking is enabled. Clicks for use in calibration data are only collected when tracking is enabled. No post­processing is performed when either tracking is disabled or the user has not registered the system by entering their email address. 3.7.3 Pinpoint 15 Pinpoint feature localization is a feature that allows more precise tracking of facial features and is necessary for gaze tracking. However, it can be disabled for reduced CPU consumption and faster frame rates, when using applications that only require head and face pose tracking, and do not require gaze data. 3.7.4 Consent Web pages are not able to send any commands to the system via the browser extension if the user does not consent. Consent is checked when the first command is issued. The page cannot ask the user more than once, unless the page is reloaded. 3.7.5 Calibration Buffer There may be a delay between calibration clicks being completed and their appearance in the system because a temporary buffer discards data in the absence of corresponding images and clicks. The period in which data is stored in this buffer is equal to the longest allowed click (several seconds). 16 4. HTTP Interface
This section documents the HTTP interfaces that control the system and make data available to user interfaces and third party programs. The interfaces presented by each process are listed separately. Note that most third party programs will only need to communicate with the persistent process. 4.1 Persistent Process (9522) The following services are all accessible from the same root URL: http://xlabs:9522/servlets/URL … where URL is the entry given in the table below. For example, to access the kv (Key­Value) service, use: http://xlabs:9522/servlets/kv Most functionality is achieved using the Key­Value servlet. Dedicated functionality to support calibration is provided by the click and watch servlets (which log data) and the calibration/* servlets. The persistent service also provides HTTP commands to start and stop the transient / realtime process. URL Method Parameters Response Notes click GET None CSV file containing all user clicks eligible for calibration. Used by the realtime service to obtain all calibration click data. click POST t1: Time start t2: Time stop xs: Screen X ys: Screen Y None Used to send calibration clicks for association with camera image measurements. watch GET None CSV file containing all image measurements eligible for calibration. Used by realtime service to obtain all calibration image measurement data. Contents encrypted*. 17 watch POST Encrypted* None Used by realtime service to log measurements obtained from camera images. kv GET Any parameter will be interpreted as a key. Values will be returned for all specified keys. A JSON object containing keys and associated values. Used as a general key­value store backed by a .properties file. kv POST Parameters must None have values assigned. Any parameters allowed. Value will be stored for any parameter given. kv/write POST As per kv POST, but None values are persisted to disk and will remain after service restart. Value will be permanently stored for any parameter given. realtime/start POST None None Will cause the realtime process to be launched, if not already running. realtime/stop GET None None Will request a graceful shutdown of the realtime process. realtime/stop POST None None As above. calibration/clear POST None None Deletes all calibration data associated with the current user. None A JSON object Allows quantification with keys of available “clickCount” and calibration data. “watchCount” calibration/count GET *Note that to protect the implementation of proprietary gaze­tracking algorithms, the content of the watch data file is encrypted. However, it does not contain data that can be used to reconstruct the appearance of the user’s face or other personally­identifiable information. 18 4.2 Transient / Realtime Process (9523) These services are only available when the transient / realtime process is running. This can be tested using the kv servlet in the persistent web service: http://xlabs:9522/servlets/kv?realtimeActive Similarly, the persistent web service can be used to start the transient / realtime web service. The most likely reason a third party application would need to access the transient service is to access the camera video stream. This is provided as an MJPEG stream. URLs are formed as follows: http://xlabs:9523/URL … where URL is the entry given in the table below. For example, to access the video image use: http://xlabs:9523/video Several of the transient/realtime HTTP commands are used to query and modify individual settings files. Changes to these files do not persist beyong the duration of the realtime process and will be reset when the process is restarted. The set of HTTP parameters to these commands matches the keys in the settings files. Many of these settings are ignored once the realtime process is running. Exercise caution in modifying these settings. URL Method Parameters Response Notes video GET None A MJPEG stream (multipart) response containing a sequence of JPEG images. ping GET None OK Lightweight way to check responsiveness of process stop GET* None OK Triggers a graceful shutdown of the process. 19 stop POST None OK Triggers a graceful shutdown of the process. settings/online GET or POST* HTTP parameters are optional, but if provided, should be valid setting keys from this setting file. If parameters are provided, the associated values will be set. JSON object containing all settings and their values. Content of settings files listed in section 8. settings/camera GET or POST* See above. JSON object containing all settings and their values. Content of settings files listed in section 8. settings/recorder GET or POST* See above. JSON object containing all settings and their values. Content of settings files listed in section 8. settings/validation GET or POST* See above. JSON object containing all settings and their values. Content of settings files listed in section 8. * For developers’ convenience, we allow a GET to stop the realtime service so the command can be executed by entering the URL in a browser. 20 5. How-to
The following How­To examples are based on the basic Javascript API example provided in section 3. Ensure that you fully understand that before attempting these tasks. These additional tips are intended to help familiarize developers with the system. 5.1 How to detect faces The face can easily be detected by querying the Key­Value servlet in HTTP: http://xlabs:9522/servlets/kv?validationErrors The resulting JSON object will contain a property validationErrors with a string value. If the value contains the character “F” then no face is detected. If “F” is not present, then the image currently contains a face. In Javascript the same functionality can be achieved by looking at the kvValidationErrors property of the state object passed in the xLabsApiState event. Note that the value stored in the Key­Value servlet is only updated when the realtime service is running (realtimeActive=1) and validation is enabled (validationEnabled=1). At other times the value of validationErrors is undetermined (basically left since last update). 5.2 How to embed video Video from the camera can easily be embedded in a HTML page using the <img> element, for example: <img src="http://xlabs:9523/video" /> An example is provided in: http://127.0.0.1:9523/static/stream.html (Use view­source to see the relevant HTML code). Note that video will only be available when the transient / realtime process is running (realtimeActive=1). Video can be started and stopped on demand by modifying the src attribute of the element in Javascript (give the img element an ID so it can easily be accessed). Set src to an image file or null to terminate video; set src to the correct URL to start video. However, it has been reported that some browsers will continue to maintain an open stream from the camera unless a page navigation event occurs. To avoid this inconvenience, consider placing the img element in an iframe. 21 Note that video streaming may affect system performance as compression and decompression of JPEG images is computationally demanding. 5.3 How to show gaze coordinates See the first example in section 3: Javascript interface. This draws the gaze coordinates on screen. 5.4 How to find the user’s face in video A bounding box around the user’s face is provided as a Key­Value (kv) property and is therefore available in both Javascript and HTTP interfaces. In HTTP GET: http://xlabs:9522/servlets/kv?validationFaceRect The result is a JSON object with property validationFaceRect and value is a String containing x,y,w,h of the face bounding box. In Javascript, the kvValidationFaceRect property of the state object passed in the xLabsApiState event contains the same string. The face coordinates assume the same image resolution and origin as provided in the streaming video images. Therefore, you can overlay the face bounding box on the video feed without scaling. However, you will need to scale the bounding box if the video images are resized. Note that the value stored in the Key­Value servlet is only updated when the realtime service is running (realtimeActive=1) and validation is enabled (validationEnabled=1). At other times the value of validationFaceRect is undetermined (basically left since last update). When no face is detected (validationErrors contains a “F”) the value is also undetermined. 5.5 How to detect gaze tracking status We cannot be absolutely sure that gaze tracking is working, but it should be achieving the accuracy levels reported in our whitepaper (see http://www.xlabsgaze.com/documentation for a copy) as long as no errors are detected. The following properties should be checked to verify gaze tracking is working perfectly: ● realtimeActive=1 ● validationEnabled=1 ● validationStatus=1 ● validationExcessYaw=0 ● validationNoFrames=0 ● calibrationStatus=1 (if 2 then pose is not ideal) 22 However, even when all is not perfect the system will do its best to generate gaze data. Note that errors will persist in the system when the realtime process is not running or when validation is not enabled. 23 6. Installer Actions
This section lists all the actions taken by our client software installer programs. Action Notes Creates environment variable XLABS_HOME The environment variable XLABS_HOME is created and added to the system profile. The value of the variable is the root folder that will contain all the files belonging to the Eye/Gaze tracking system. Copies files to XLABS_HOME All files are extracted from the installer archive and copied to the
directory specified by XLABS_HOME Adds entry ‘xlabs’ to HOSTS file The HOSTS file resolves URLs to IP addresses. We add an entry for the URL “xlabs” that points to the local computer (127.0.0.1 in IPv4, aka localhost). This makes it easier for users
to access the local web services. Add Service or Daemon and schedule to run on startup We add a scheduled task (on Windows) or daemon (OSX and Linux) that runs on user login. The service/daemon provides the
local web pages and a HTTP interface to control the Eye/Gaze tracking system. It consumes very few resources when the system is not in use. The service/daemon launches a separate process to access the camera and perform processing on images when the system is active. The only file that is not contained within the XLABS_HOME folder is the definition of the service/daemon. On OSX the daemon definition is placed into: /Library/LaunchDaemons/ On Linux systems the daemon is placed into: /etc/X11/Xsession/ On Windows systems the service is added as a scheduled task and the usual Windows programs can be used to administer it (e.g. Task Scheduler). The service / daemon is run once­off by the installer script. Open local webpages in default web browser After all other steps have completed, the local web service is used to display the local web pages. The user’s default browser
is opened to the URL http://xlabs:9522 24 7. Directory Structure
All files are contained within the directory specified by XLABS_HOME. This location is modifiable by the user during installation. Within XLABS_HOME the directory structure is as follows: 7.1 Subdirectories Subdirectory Notes on content XLABS_HOME/3rd_party Files needed to run the system that were created by third
parties. Specifically, files for Face Detection (OpenCV) and ASM Face­Fitting (STASM). XLABS_HOME/cpp Home of executable and setting files for the transient “realtime” process that connects to the camera and performs image processing. XLABS_HOME/cpp/realtime_calib See above. XLABS_HOME/cpp/realtime_calib/ Settings files that configure the realtime process that settings connects to the camera. Some can be modified by expert users. XLABS_HOME/data Data files generated by the Eye/Gaze tracking system. These files are modified during normal operation of the system. Any user data would be stored here. XLABS_HOME/java Home of the persistent web service that is run as a service or daemon (depending on operating system). This includes the .jar files for the program itself and configuration files. XLABS_HOME/jre If the installer included a bundled Java Runtime Environment, it will be included in this directory. XLABS_HOME/www Static content for web services. SDK demo files are included here. XLABS_HOME/www/sdk­demo.ht Demo file showing how to make a website gaze­enabled.
ml XLABS_HOME/www/persistent Static files to support the web interface of the persistent web service. XLABS_HOME/www/realtime Static files to support the web interface of the realtime (camera program) web service. 7.2 User Data 25 The client software does not transmit any data to other computers except system usage information. System usage information is transmitted as a single HTTP POST request that contains the email address used to register the system and a timestamp. Application software developed by third parties that uses our client software may collect other data about users. For the purposes of calibration, we store files describing user clicks and the appearance of the user. The latter file is encrypted, but even when decrypted cannot be used to produce an image of the user. The software is capable of recording camera images to disk. This feature is not currently available in the browser extension or local web pages user interfaces. However, developers can make use of it. No mechanism is provided to transmit images to another computer. File containing user data Description of content XLABS_HOME/data/<USER>.clicks.csv <USER> = username specified in browser extension, not registered email address. A CSV file containing screen coordinates and timestamps of clicks by this user. The clicks are not associated with any web history. XLABS_HOME/data/<USER>.watch.csv An encrypted CSV file containing a geometric description of user appearance. This data is used for calibration. XLABS_HOME/data/session_YYYY­MMM­DD When video recording is enabled, frames _HH­MM­SS from the camera are saved to directories having this naming structure. A separate directory is created each time the camera program is executed. 7.3 Un­install instructions The system can be safely removed by deleting the XLABS_HOME directory and all subdirectories. 26 8. Transient / realtime settings
Settings files are used to configure the transient / realtime process, but also allow modification of more advanced features for which there is currently no graphical user interface. Beware that modification of settings files can seriously impair or prevent proper functioning of the system. 8.1 Settings files Settings files are located in the transient / realtime process directory: XLABS_HOME/cpp/realtime_calib/settings/*.txt XLABS_HOME/cpp/realtime_calib/settings_gaze_tracking/*.txt The settings_gaze_tracking directory contains a set of setting overrides for some properties. These are read after the original settings directory and will override any changes made there. Settings files are user­editable text files. One setting is specified per line. C­style // comments are allowed and are ignored. Lines containing only whitespace or comments are ignored. Each setting entry has the following format: TYPE::KEY=VALUE TYPE should be a basic C++ type i.e. one of char, int, long, float, double, bool, string. KEY should be a string without whitespace that is recognized by the program. User­defined settings will be ignored. VALUE should be some value that can be deserialized into the specified TYPE. Most settings files are only inspected during initialization of the realtime process. Some settings files are exposed via a HTTP interface (HTTP commands with URLs containing “settings”). This allows runtime modification of these settings. 8.2 Camera settings The client software has a graphical interface to modify some camera settings (basically selecting the camera device and changing resolution). However, additional camera configuration features can be achieved via the settings files. Camera settings are stored in these files: XLABS_HOME/cpp/realtime_calib/settings/camera_settings.txt XLABS_HOME/cpp/realtime_calib/settings_gaze_tracking/camera_settings.txt The former file is overwritten automatically by the client software camera configuration page. The latter file overrides the former one, and can be manually modified. 27 8.2.1 Selecting a different camera device To change the camera device, modify the deviceIdx setting: int::deviceIdx = 0 Any integer device index can be used. Since the realtime program uses the OpenCV camera interface and drivers, the only way cameras can be identified is via an integer index. Check the camera image feed to work out which is which. 8.2.2 Changing image dimensions The default image dimensions may yield distorted images on some cameras. Other cameras may not provide an image at all unless the image dimensions are acceptable. To fix these problems, modify the frame width and height settings. Note that all these properties are passed through to the OpenCV camera interface, hence the OpenCV property names are used. To modify width and height of the captured frames change these settings: double::CV_CAP_PROP_FRAME_WIDTH = 1024 double::CV_CAP_PROP_FRAME_HEIGHT = 768 Any resolution equal to or greater than 640x480 should work for Eye/Gaze tracking. Depending on imaging artefacts introduced, greater resolution may or may not improve performance. 8.2.3 Changing image format Other properties of the OpenCV camera interface can be modified by entering them in the camera settings files. This includes properties not currently listed in the settings files. Frame colour and pixel format can be specified by adding the settings used by OpenCV with suitable values. A full list of OpenCV image capture properties can be found at: http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html#video
capture The Eye/Gaze tracking system works with any image format that can yield a single channel 8­bit greyscale image (8UC1). 8.4 Video recording settings The system is able to record images from the camera to disk. This functionality is not currently available from the graphical user interface. To enable recording, set: bool::recorderThreadIsRecording = 1 … in file: XLABS_HOME/cpp/realtime_calib/settings/recorder_settings.txt 28 Note that this setting can also be remotely modified during execution to turn recording on and off if needed by a third party application. The location of the folder in which images are stored can also be modified from within this file. 8.5 Working from offline image files To enable repeated experimentation or testing, the system can work from recorded video provided as a sequence of image files. To enable offline processing, set: bool::offline = 1 … in file: XLABS_HOME/cpp/realtime_calib/settings_gaze_tracking/program_options_settings.txt Then set: string::dir = PATH … in file: XLABS_HOME/cpp/realtime_calib/settings_gaze_tracking/offline_settings.txt … where PATH is the absolute path to the images to be processed. NB: The video will loop endlessly unless you also set: bool::looping = 0 29 9. Administration
This section will allow advanced users and developers to administer the services or daemons installed as part of the client software. 9.1 Microsoft Windows Scheduled Task The xLabs client software persistent web service is installed as a Windows Scheduled Task. To administer it, go to: Control Panel \ System and Security \ Administrative Tools \ Task Scheduler Alternatively, enter “Task Scheduler” into the search box in the Start menu. When the Task Scheduler program loads, click on the “Task Scheduler Library” tree in the left panel to expand it. The main panel should now show a list of tasks in the library. Look for the one named “xLabsWebServices”. Double­click this entry to edit the service. 9.2 Apple OSX Daemon The daemon that runs the persistent web services on OSX systems is configured and controlled via the launchctl program. The configuration file for the daemon is: /Library/LaunchDaemons/au.com.xlabs.client.plist The daemon is named: au.com.xlabs.client To administer the daemon (check, start and stop) use the following commands: sudo launchctl start au.com.xlabs.client sudo launchctl list | grep xlabs sudo launchctl stop au.com.xlabs.client 9.3 Linux Daemon The Linux Daemon is a bash script named ‘xlabs’ located in /etc/X11/Xsession , because the X­server is needed to query the screen dimensions during calibration. Therefore, an /etc/init.d daemon cannot be used. The daemon can be controlled as follows: cd XLABS_HOME sudo ./xlabs_daemon_amd64.sh start sudo ./xlabs_daemon_amd64.sh status sudo ./xlabs_daemon_amd64.sh stop The arguments are relatively self­explanatory… 30