The G String

I reached the second milestone of my project to create a virtual reality model of the Conic Bellophone, an instrument comprised of 144 cone-shaped bells being developed for a PhD project. Moving on from the early prototype, which has six bells, I have managed to complete a version that has 144 bells, creating a ghostly harmonic when played.

Wide shot of Conic Bellophone in Xj3D

The first challenge was dealing with the size and complexity of the model. I was supplied VRML97 exported from CATIA, describing a much more complicated and detailed model. In this model, the author had gone to the detail of individual screws, brackets, stands and bells, and rather than defining the geometry as primitive shapes, the exporter program naively creates large numbers of IndexedFaceSets; defining each point on a plane individually. Consequently, the supplied VRML97 was about 15MB in size, some 40,000 lines of code. This instantly presented problems with the tools I was using; Netbeans would regularly crash and the load-up time was taking about 4-5 minutes, even on my MacBook. At any rate, to deliver such a file on the web would be useless. So my first task, before I could really begin, was to reduce the filesize of the model. To accomplish this, I had to first depart from the arcane VRML97 format and into X3D, an XML version of VMRL that I could work with.

I noticed from inspecting the file that the exporter had introduced a large number of seemingly useless group nodes, such as:

<Group>
  <Group>
    <Group>
      <Group USE="somegroup">
        ...
      </Group>
    </Group>
  </Group>
</Group>

I decided to remove them with a simple parser written in Java. I created an application that read in the XML file, performed a recursive operation to remove the nodes and write out the result.

private void deleteUseless(Node n)
{
     Node parent = n.getParentNode();

     NodeList children = n.getChildNodes();
     for(int p = 0; p < children.getLength(); p++)
     {
       Node child = children.item(p);

       if(!child.getNodeName().equals("#text"))
       {
         deleteUseless(child);
       }
     }

     if(n.getNodeName().equals("Group")
             && n.getAttributes().getNamedItem("USE") == null
             && n.getAttributes().getNamedItem("DEF") == null)
     {
       for(int p = 0; p < children.getLength(); p++)
       {
         Node child = children.item(p);
         if(!child.getNodeName().equals("#text"))
         {
             parent.insertBefore(child, parent.getFirstChild());
         }
       }
       parent.removeChild(n);
       replacecount++;
     }
}

This method removed 2,500 unnecessary nodes. This still wasn’t enough to make the scene workable, so I began to simplify the scene by replacing IndexedFaceSets with primitive geometry.

The most important element in the scene were the bells themselves. I created a new node to represent a bell, that would be scaled to the desired size and re-used multiple times. In X3D jargon, this new node is called a prototype. A prototype defines a new node, and has its own scope for ROUTEs and DEFs. A prototype can be parametrised by defining a number of fields, which permit re-use, towards a component-based method of building complex scenes.

At this stage each bell had three requirements, other than rendering the bell in 3D. Firstly, the bell had to sound when clicked, as if the bell were stuck in real-life. Secondly, the bell had to change colour when clicked, to visualise what parts of the instrument were playing. Thirdly, the bell had to be able to play when the mouse was hovering over the bell, so that a run of bells could be played in a glissando effect. The behaviour between a single-bell sound and a glissando would be toggled by pressing the ‘G’ key on the keyboard.

Single sound of AudioClip

The first requirement is the most straightforward to implement. It is a text-book use of the TouchSensor node, along with a Sound and AudioClip node. Keeping all the nodes grouped together, it is important that the TouchSensor is a sibling of the grouping node of the geometry, for the TouchSensor acts upon the child nodes of its parent. Thus:

<Group>
  <TouchSensor />
  <Sound>
    <AudioClip />
  </Sound>

  <!-- Geometry -->
  <Shape>
    ...
  </Shape>
</Group>

The AudioClip defines the sound to play with its url parameter. X3D allows multiple URLs to be specified, to allow for local and remote access. ROUTEs are used to implement the sound behaviour:

<ROUTE fromNode="singleTouch" fromField="touchTime" toNode="chime" toField="stopTime"/>
<ROUTE fromNode="singleTouch" fromField="touchTime" toNode="chime" toField="startTime"/>

N.B. I include both stopTime and startTime as I had desired the bell-sound to restart when clicked again, much like a bell would if hit successive times. According to the X3D specification, passing the same time event for the stop and the start should result in the audio clip restarting. However I found mixed results with this, and mostly I see start/stop behaviour instead.

Colour Highlight with ColorInterpolator

Like all complicated concepts, using interpolators in X3D is straightforward once you understand it. I wanted to choose a colour for the bell’s material, which would change like a roll-over in CSS. In X3D, its easier to consider the problem as charting the node’s material to pass through a series of colour values, which is precisely what ColorInterpolator does. In this sense its more akin to a robot walking between waypoints than a roll-over paradigm.

An interpolator (or the similar sequencer nodes) have two main variables. The key, which defines the points in the series, and the keyValue, which define the corresponding values at those points. The key is always between 0 and 1, representing the start and end of the cycle, with any number of fractions in between, seperated by whitespace. For my roll-over I defined 0, 0.5 and 1, so that I can pass through three colours, finally returning to the original colour at the end of the cycle. For the corresponding values, in this case a colour, I would need three-pairs of RGB values to correspond to the colours at 0, 0.5 and 1.

In order to cause succession between the points, in X3D a TimeSensor is used to generate time events, so that the browser moves through the series, like a sequencer. When used alongside the touchsensor, these nodes can now be routed to achieve the roll-over effect.

<TimeSensor DEF="colourTimer" />
<ColorInterpolator DEF="flash" key="0 0.5 1" keyValue="0 1 0 1 1 1 0.8 0.8 0.8"/>
<TouchSensor DEF="glissTouch" enabled="false">

<ROUTE fromField='touchTime' fromNode='singleTouch' toField='set_startTime' toNode='colourTimer'/>
<ROUTE fromField='fraction_changed' fromNode='colourTimer' toField='set_fraction' toNode='flash'/>
<ROUTE fromField='value_changed' fromNode='flash' toField='set_diffuseColor' toNode='material'/>

Glissando effect with AudioClip

The best result of the model was the glissando, simultaneously playing several samples of micro-tonal bells, creating a haunting and harmonic sound when the mouse cursor passes over a bell. This effect was a slight change from the single-touch, using a different field to drive the behaviour. touchTime would be too late; for this is the time value when the touch sensor’s geometry was clicked. Instead, TouchSensor provides an isOver event, which is set to true when the mouse cursor intersects the geometry. As I required a time value rather than a boolean value, I needed another node to convert the values, called TimeTrigger. TimeTrigger is an invisible node that acts like a black-box; accepting a boolean input and returning a timestamp output, triggerTime. I used this to route between the TouchSensor and the AudioClip to obtain the glissando effect, and could even use this to drive the colour flash.

<ROUTE fromNode="glissTouch" fromField="isOver" toNode="gliss" toField="set_boolean" />
<ROUTE fromNode="gliss" fromField="triggerTime" toNode="chime" toField="set_startTime" />
<ROUTE fromField='triggerTime' fromNode='gliss' toField='set_startTime' toNode='colourTimer'/>

Toggle play mode with keyboard button press

Finally, I needed to switch between glissando and the single-click modes by pressing a keyboard key. To keep the behaviours seperate, I defined two seperate TouchSensors, each routed differently to achieve the different behaviours. I defined the TouchSensor associated with the glissando effect to be disabled, with the enabled=”false” attribute on the touch sensor. The process of toggling thus became an alternation of the enabled attribute of the TouchSensor. The current state of the toggle is stored using a BooleanToggle node.

The browser picks up keyboard events when you define a KeySensor within your scene. The KeySensor can then be routed to a script node, for processing. Below I wrote a short script to detect whether the letter ‘G’ had been pressed, and routed the result to the touch sensor.

<KeySensor DEF="glissMode" />
<BooleanToggle DEF="playMode" />

<Script DEF="logic" directOutput="true">
  <field name='keyInput' type='SFString' accessType='inputOnly'/>
  <field name='toggleGliss' type='SFBool' accessType='outputOnly'/>
  <![CDATA[ecmascript:
  function keyInput (inputValue)
  {
    if(inputValue == 'g' || inputValue == 'G')
    {
      toggleGliss = true;
    }
  }
  ]]>
</Script>

<ROUTE fromNode="logic" fromField="toggleGliss" toNode="playMode" toField="set_boolean" />
<ROUTE fromNode='glissMode' fromField='keyPress' toNode='logic' toField='keyInput'/>
<ROUTE fromNode="playMode" fromField="toggle_changed" toNode="glissTouchSensor" toField="enabled" />

Putting it all together

Now I could define a prototype and re-use the bell logic and geometry. I would embed each prototype inside a transform element, so that I could scale and translate each individually. The only alteration I needed to make was with the button toggle; for the state of the toggle had to be held within the main scene, not within each prototype, because each prototype has its own scope. I modified the prototype to accept the boolean value of the BooleanToggle as a field, as well as the URL for the sound to play.

<ProtoDeclare name='Bell'>
  <ProtoInterface>
    <field accessType='initializeOnly' name='soundURL' type='MFString' />
    <field accessType='inputOnly' name='playMode' type='SFBool'/>
  </ProtoInterface>
  <ProtoBody>
    <Group DEF="BellMain">
      <TimeSensor DEF="colourTimer" />
      <ColorInterpolator DEF="flash" key="0 0.5 1" keyValue="0 1 0 1 1 1 0.8 0.8 0.8"/>
      <TouchSensor DEF="glissTouch" enabled="false">
        <IS>
          <connect nodeField='enabled' protoField='playMode'/>
        </IS>
      </TouchSensor>
      <TouchSensor DEF="singleTouch" />
      <TimeTrigger DEF="gliss" />
      <Sound maxFront="100000" maxBack="100000">
        <AudioClip DEF="chime">
          <IS>
            <connect nodeField='url' protoField='soundURL'/>
          </IS>
        </AudioClip>
      </Sound>

      <!-- Bell Geometry -->

     <BooleanToggle DEF="playMode">
       <IS>
         <connect nodeField='set_boolean' protoField='playMode'/>
       </IS>
     </BooleanToggle>

     <!-- Single-touch sound -->
     <ROUTE fromNode="singleTouch" fromField="touchTime" toNode="chime" toField="stopTime"/>
     <ROUTE fromNode="singleTouch" fromField="touchTime" toNode="chime" toField="startTime"/>

     <!-- Glissando -->
     <ROUTE fromNode="glissTouch" fromField="isOver" toNode="gliss" toField="set_boolean" />
     <ROUTE fromNode="gliss" fromField="triggerTime" toNode="chime" toField="set_startTime" />
     <ROUTE fromField='triggerTime' fromNode='gliss' toField='set_startTime' toNode='colourTimer'/>

     <!-- Colour flash -->
     <ROUTE fromField='touchTime' fromNode='singleTouch' toField='set_startTime' toNode='colourTimer'/>
     <ROUTE fromField='fraction_changed' fromNode='colourTimer' toField='set_fraction' toNode='flash'/>
     <ROUTE fromField='value_changed' fromNode='flash' toField='set_diffuseColor' toNode='material1'/>
  </ProtoBody>
</ProtoDeclare>

Rather than rigging up 144 route statements to pass each prototype the playmode value, I picked up this neat trick from the X3D mailing list to dynamically add routes via the SAI. In order for the script to work, all of the prototypes were grouped together in a single group, so that they could be passed to the script.

<Script DEF="logic" directOutput="true">
   <field name="touchs" type="SFNode" accessType="initializeOnly">
     <Group USE="Touchable"/>
   </field>
   <field name="playMode" type="SFNode" accessType="initializeOnly">
     <BooleanToggle USE="playMode" />
   </field>

   <![CDATA[ecmascript:
   function initialize()
   {
    scene = Browser.currentScene;

    for (i = 0; i < 144; i++)
    {
      bell = touchs.children[i].children[0].children[0];
      scene.addRoute(playMode, 'toggle_changed', bell, 'playMode');
    }
   }
  ]]>
</Script>

Next steps

I was left only to rig-up the geometry for the stands and the bases. Overall, the model is working well. After the optimisations, the file was reduced to 1,500 lines of code, and renders at an acceptable framerate. The next challenge will be to reduce the file-size of the audio-clips: currently 10 second .wav files, before it can be delivered onto the web proper.

The remaining area of functionality for this project is to work with MIDI to read-in compositions and play them using the new instrument, in effect, writing a basic MIDI sequencer in X3D. I’m currently working on this now, so watch this space! You can listen to a sample of the sounds being played on my music blog.

Thanks to all those in the X3D Mailing List whose comments have helped me progress with this project.

Close up of Bells playing glissando