1.4 TO 1.5 UPGRADE

Daisy 1.5 was released on August 16, 2006.

Read this document carefully, especially the upgrading instructions. Take your time to do the upgrade. This release contains various changes that will make future updates a lot easier.


Some important changes that affect deployment or compatibility: READ THIS SECTION

  • Daisy Wiki: the configuration and data used by the Daisy Wiki is now stored in a separate wikidata directory, similar to the data directory of the repository server. The data directory contains things such as the skins, the site definitions, the doctype XSLs, the logs, ... In earlier Daisy versions, these items were simply stored inside the Daisy Wiki webapp. The new structure is also illustrated in the updated deployment diagram.
    The introduction of the wikidata directory has several advantages:
    • Clear separation of data and application
    • Clear view on official extension points
    • Upgrading and backups become easier: it will no longer be needed to copy over your site definitions, skins, etc. after upgrading.
  • OpenJMS is replaced with ActiveMQ as default JMS implementation.
    • We embedded the ActiveMQ inside the repository server so from now on, the OpenJMS server doesn't need to be started anymore.
    • The ActiveMQ project is much more alive then OpenJMS, provides more ways to connect to it, ...
  • A new default skin
    • Much more CSS-based, the layout is better controllable through CSS only.
    • The rather big default.css file has been split up in multiple smaller CSS files
    • Aesthetic improvements:
      • lighter look
      • fixed width layout (better readability on wide screens)
      • ...
    • See also the compatibility notes if you have created your own skin(s) based on the old default skin.

Other changes:

  • Various query language improvements. See this mail and the updated query documentation.
  • The content of the blobstore directory is now organised hierarchically. This should scale better on most file systems.
  • A new field type, link, to allow structured linking between documents. See this mail for an overview.
  • Query-based selection lists. See this mail.
  • Publisher:
    • preparedDocuments will now execute a separate publisher request for the document and each document it includes. The publisher request to be used is determined based on a mapping definition which can test on things like the documentType (using the same expressions as in the ACL). This allows to retrieve additional information for certain sets of documents, e.g. related documents & queries. Together with things like the link field type or the ContextDoc function in the query language, this opens many new possibilities. See this documentation and the upgraded publisher docs.
    • The publisher request model has been made more flexible and supports a some new tags. See also the third star in this mail.
  • Document styling:
    • The input XML for the stylesheets now has a "isIncluded" attribute on the root <document> element. This attribute allows to render the document differently depending on whether it is included it another document.
    • The input XML for the stylesheets now includes the full Daisy Wiki page context (the <context> element) instead of just the user information.
  • Links to Attachment-documents now link directly to the attachment data. This behaviour is easily controllable via XSL. See this mail for more information.
  • The content of non-Daisy-HTML-parts can now be inlined during publishing, so that it is available to the document-styling XSLT. See this mail.
  • Image thumbnails and metadata extraction: details see this mail.
  • Auto-redirecting between sites: when requesting a document that does not occur in the navigation tree of the current site, the other sites are now considered to see if they are better suited (either because of a match in the navigation tree, or because the document belongs to the collection associated with the site). This behaviour can be disabled (see the compatibility section). See also the docs.
  • Added version info: the Daisy Wiki and the Daisy repository server now add an X-Daisy-Version header, the <context> element available in the Daisy Wiki skinning now includes version info, and most command line utilities support a -v and --version option.
  • Allow to switch between live and staging views in the Wiki. Up till now, the Wiki displayed by default the live versions of all documents. When switching to staging mode, the last versions of the documents are displayed by default. So it shows how everything would look if the last versions of all documents would become live. This also applies to navigation trees, queries embedded in documents, et cetera. See docs.
  • Navigation trees:
    • Nodes can now be hidden or visible-when-active. Hidden nodes allow to have the navigation tree expanded up to a certain point, and to control the URL path, without having to overload the navigation tree (this will probably mostly be done using queries). Visible-when-active is similar: the node is always hidden except when it is active.
    • Query nodes in the navigation tree can now create hierarchies instead of just a flat list of nodes. For example, if you enter a query like:
      select documentType, name where true order by documentType, name

      this will create group nodes per document type, and normal document nodes below that. This can of course work multiple levels deep. Make sure the order by clause is meaningful (thus: sort values in the same order as selected).

  • Books:
    • A new book publication process task, named "custom", allows to specify a Java class to do custom processing.
    • Publication types can now specify parts for which they need the binary content to be available.
    • The input for the doctype XSLs now has a /document/@isIncluded attribute, just as in the Wiki, to distinguish included documents.
    • Added two publication tasks: renderSVG and callPipeline, both of which are useful in combination with the new part-download capability (see the second bullet)
    • The publication task copyBookInstanceImages has been renamed to copyBookInstanceResources, though the old name still works. The new name also comes with some new functionality: not only img/@src resources are copied, but also resources linked to in a/@href, embed/@src, object/@data.
  • Faceted navigation: allow to limit the set of documents on which the faceted navigation is performed, through an <defaultConditions> element in the faceted navigation definition.
  • Removed the need to have absolute paths in the myconfig.xml by supporting property substitutions (where needed), such as ${daisy.datadir} and ${daisy.home}. The repository initialisation script now generates such myconfig.xml's without absolute paths. This allows to move the data directory around without having to adjust these paths. (DSY-271)
  • The homepage of a site, as defined in the siteconf.xml, can now be a custom path instead of only a document ID. This is for example useful to let the homepage point to some extension URL. See the sites documentation.
  • The recent changes page now has links to the RSS feeds (including link elements in the page header, so that the feeds can be detected automatically).
  • Editor: improvements to the image browser dialog:
    • the collection of the current site is automatically selected by default, so the user immediately sees the images in that collection. The list of collections has been changed to a drop-down list.
    • it is possible to select no collection, thus showing all images, without a collection constraint
    • the image browser now shows all documents that have an 'ImageData' part. Previously it showed all documents of type 'Image'. The benefit of this is that you can have custom document types for images, and as long as they reuse the 'ImageData' part for the rendered image, it will now show up in the image browser.
  • Document editor:
    • Allow to initialise new documents with some content before showing the editor (docs).
    • Allow to specify a custom location to redirect to after editing is done (useful for included/aggregated content).
  • Included documents now show edit links next to their title(if you have edit rights on the document) (similar to Wikipedia).
  • The document basket: a new Daisy Wiki feature that allow end-users to do things with sets of documents (such as aggregate them for easy printing), as described in this mail.
  • Added a schema uploader tool, which allows to create a Daisy repository schema (document, part, field types) from an XML file.
  • Added SVG support, through a new SvgDocument document type. This is roughly based on the scratchpad article, but with several enhancements: the SVGs embedded in HTML are automatically resized by some javascript, and the SVGs are also properly rendered in the books (in the PDF publication by default at 250dpi).
  • The document type specific XSLs, and the query styling XSLs are now put in the directory of the skin. This makes fallback between skins work for them.
  • Added a version reversion feature in the Wiki, accessible through the versions overview page.

To also see bug fixes or some non-functional changes, check the change lists for each individual 1.5 Milestone release.


Document type specific stylesheet

There is small difference in the output of the publisher requests which influences the document type specific stylesheets. Don't worry, it is a very minor change, depending on how your stylesheets are written they might even require no change at all.

Where the input previously contained this element hierarchy:

document > d:document

it has now become:

document > p:publisherResponse > d:document

Where "p:" is the publisher namespace, thus bound as:


If you want to have stylesheets that work with both releases, you can write them so that they first test for the presence of document/d:document and if not available use document/p:publisherResponse/d:document

Custom styling: attributes on links

The links in document content were previously annotated with the attributes daisyDocumentName, daisyFileName and daisyNavigationPath. These have been removed in favour of a more powerful system, as explained in this mail.

If you have custom document type specific XSLs or book publication types, you might want to do a search for @daisyDocumentName used in XPath expressions, and replace it by p:linkInfo/@documentName, the p prefix should be declared as:


Auto-redirecting between sites

The Daisy Wiki now has the ability to automatically redirect to another site if a document is better suited to be displayed over there. This is enabled by default, and can be disabled by adding the following element in the siteconf.xml of each site for which you want to disable this:

<siteSwitching mode="stay"/>

Field type in HTTP interface

A small change has been done to the field type XML. If you have any programs making direct use of the HTTP interface, you'll need to adjust them.

Where you previously had a <d:selectionList> element for specifying a static selection list, the <d:selectionList> element now contains an extra <d:staticSelectionList> element which then contains the elements previously contained in <d:selectionList>. This change was done to allow for alternative selection list implementations.

Publisher XML Beans generated classes

Applies when using custom publisher requests build up via the Java objects (instead of written in XML).

The structure of the XML Schema for the publisher requests has changed a little bit, which has caused some small changes in the nesting of the classes generated by XMLBeans for this XML Schema. Adjusting your code for this is straightforward.


[todo: add helpful notes here on updating custom skins based on the old default skin]

Custom book publication types

If you made custom book publication types or document type specific XSLs for use in books, they might need some small changes due to the introduction of the wikidata directory.

In the query-styling.xsl of the publication type, if you want to import the default query styling XSL, the include should now be:

<xsl:include href="wikidata:books/publicationtypes/common/default-query-styling.xsl"/>

If in the custom document type specific XSL you included the default book-document-to-html.xsl file, the import should now be:

<xsl:include href="wikidata:books/publicationtypes/common/book-document-to-html.xsl"/>


MySQL 4.0 is not supported anymore. You might still be able to get it to work, but it wasn't worth the effort anymore for us to support it in the default configuration.

JmsClient API

If you were using the remote Java API and constructing a JmsClientImpl to enable remote cache invalidation through JMS, you'll notice the constructor has changed: it takes a new parameter clientID (the JMS client ID, which is a string of your choice, but should be different for each client), and the topicConnectionFactory and queueConnectionFactory parameters have merged into just one connectionFactory argument (since we're now using JMS 1.1 APIs internally, as a result of the switch to ActiveMQ).


Important: these are instructions for upgrading from Daisy 1.4 (or 1.4.1). To upgrade from older releases, e.g. Daisy 1.3, you first need to follow the instructions to upgrade to 1.4, and then these instructions (you don't need to actually download or install 1.4, just follow the instructions). To upgrade from Daisy 1.5-M1, follow the instructions for upgrading to 1.5-M2 (from 1.5-M2 to 1.5-final, there are no additional instructions). To upgrade from Daisy 1.5-M2, see the instructions (basically, nothing).

Before starting

Shutdown Daisy (the Repository Server, the Daisy Wiki, and the OpenJMS server)

Make backups! More specifically:

daisy-backup-tool -b -d $DAISYDATA_DIR -l $DAISY_BACKUP_DIR -o $DAISY_HOME/openjms -a additional-entries.xml

The '-a additional-entries.xml' is not required to make a backup, but if specified you can backup daisywiki specific files.  If you want to backup these files you must first create the addition-entries.xml file, an example can be found in the backup documentation.

If you can't get the backup tool to work (it was buggy in Daisy 1.4 on Windows), you can also do a manual backup:

  • make a copy of the daisy data directory
  • do a dump of the database:
    mysqldump daisyrepository -uuser -ppassword > daisyrepo.sql
  • make a copy of at least daisy/daisywiki/webapp/WEB-INF/cocoon.xconf, and daisy/daisywiki/webapp/daisy/sites, preferably keep a copy of the whole daisy/daisywiki/webapp untill you have successfully upgraded and moved over all config.

Download and extract installation files

  1. Rename your existing <DAISY_HOME> directory. During these instructions, we will refer to this location as <OLD_DAISY_HOME>.
  2. Download the Daisy 1.5 installation files from here.
  3. Extract the installation archive. Make sure that  your <DAISY_HOME> variable points to that newly created directory, if necessary, move or rename that directory (or adapt your <DAISY_HOME> environment variable).

Switching from OpenJMS to ActiveMQ

Make sure no messages are left in OpenJMS

Before moving to ActiveMQ, it is a good idea to check if there are no messages waiting to be processed in OpenJMS's queues. Usually JMS messages in Daisy are processed quickly, so unless you have had high activity right before shutting down the Daisy repository, there will normally be no more messages waiting to be processed.

To be sure, you can do a quick check like this:

  1. start OpenJMS
  2. go to OPENJMS_HOME/bin, and execute the "admin" script. This will launch a GUI window. In its menu, choose Actions, Connections, Online. It will ask for a username and password. The username is normally "admin", the password can be found in the file OPENJMS_HOME/config/openjms.xml (look for <User name="admin" password="something"/>). Once connected, you see the topics with their durable subscribers, and the queues. Next to them a number is displayed, they should all be zero.

If there would still be messages, just start the repostory server again, and wait till they have all been processed.

Creating a database for ActiveMQ

Start the MySQL client:

mysql -uroot

and execute:

GRANT ALL ON activemq.* TO activemq@'%' IDENTIFIED BY 'activemq';
GRANT ALL ON activemq.* TO activemq@localhost IDENTIFIED BY 'activemq';

ActiveMQ will automatically create its database tables the first time the repository server is launched.

ActiveMQ configuration

Create the ActiveMQ configuration :

cp <DAISY_HOME>/repository-server/conf/activemq-conf.xml.template <DAISY_DATA>/conf/activemq-conf.xml
cp <DAISY_HOME>/repository-server/conf/login.config <DAISY_DATA>/conf/
cp <DAISY_HOME>/repository-server/conf/users.properties <DAISY_DATA>/conf/
cp <DAISY_HOME>/repository-server/conf/groups.properties <DAISY_DATA>/conf/

Update myconfig.xml (<DAISY_DATA>/conf/myconfig.xml):

Look for the following element:

<target path="/daisy/jmsclient/jmsclient">

and replace the <configuration> element inside it with:

    <credentials username="admin" password="jmsadmin"/>
      <property name="java.naming.provider.url" value="vm://DaisyJMS?brokerConfig=xbean:file:${daisy.datadir}/conf/activemq-conf.xml"/>
      <property name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
      <property name="queue.fullTextIndexerJobs" value="fullTextIndexerJobs"/>
      <property name="topic.daisy" value="daisy"/>

If you are using MySQL 5 (thus not 4.1), then you need to edit <DAISY_DATA>/conf/activemq-conf.xml. Look for the following line (here split on two lines for readability):

<property name="url" value="jdbc:mysql://localhost/activemq?relaxAutoCommit=true&amp;

In this, you should remove the "useServerPrepStmts=false&amp;", so that it becomes:

<property name="url" value="jdbc:mysql://localhost/activemq

Configuring the Driver Registrar component

This new component registers JDBC drivers that are used by datasources in the repository-server.

Background info: previously, the JDBC drivers were configured as part of the datasource, but since JDBC connections are now needed by multiple datasources in the same VM (both the repository server and ActiveMQ), their registration is now moved into a specific component.

Edit <DAISY_DATA>/conf/myconfig.xml, and add a new target :

<target path="/daisy/driverregistrar/driverregistrar">

Notice the upgrade from mysql-connector 3.1.7 to mysql-connector 3.1.12

You may now also remove the references to JDBC drives from the datasource component.  In the same file look for "/daisy/datasource/datasource".  In the configuration of this component look for the "driverClasspath" and "driverClassName" elements.  You may simply remove both these elements like this :

<target path="/daisy/datasource/datasource">
      <driverClasspath>REMOVE THIS LINE</driverClasspath>
      <driverClassName>REMOVE THIS LINE</driverClassName>

Upgrade blobstore to hierarchical format

The blobstore directory structure has changed from being flat to being hierarchical, therefore a conversion must be made.  There is a tool that will convert your existing data to the correct format.  It can be executed by running the following command :

<DAISY_HOME>/bin/daisy-blobstore-convertor <path-to-blobstore>

The <path-to-blobstore> is normally <daisy data dir>/blobstore

The tool will print lines like:

Upgrading 1014764c672c25335146498e33393b30642a4207

for each processed file. If the tool outputs nothing, then either something is wrong (you give the incorrect path to the blobstore), or your blobstore is empty (very unlikely, impossible if you've installed the wiki).

Run database upgrade script

cd <DAISY_HOME>/misc
mysql -Ddaisyrepository -udaisy -p<password>
[then on the mysql prompt]
\. daisy-1_4-to-1_5_M1.sql

Create pubreqs directory and update myconfig.xml

In the daisy data directory, create a new empty subdirectory called pubreqs (all lowercase). This directory becomes thus a sibling of the blobstore, indexstore, conf and logs directories.

Then open the following file in a text editor:

<daisy data dir>/conf/myconfig.xml

And add the following as a child of the root <targets> element (the order of this and the other <target> elements does not matter)

  <target path="/daisy/extensions/publisher/publisher">
      <repositoryUser login="internal" password="defaultpwd"/>

Change the value of the password attribute to the same value as the password attribute on the other <repositoryUser> elements in the same file.

Change the content of the <publisherRequestDirectory> to contain the absolute path to the new pubreqs directory you just created.

Updating wrapper configuration for the Repository Server

If you are using the service wrapper (or custom startup scripts) to start the repository server, then you will need to upgrade the configuration/scripts to pass some additional parameters. In dsy_repo_wrapper.conf, these have to be added (see also the full script):


Optional: remove absolute paths in myconfig.xml

Daisy now allows to use properties in the myconfig.xml to avoid absolute paths. If you want, you can update your myconfig.xml and change absolute references to the daisy data directory and daisy home directory by ${daisy.datadir} and ${daisy.home} respectively. Doing this will allow to move your data directory around without changing the myconfig.xml.

Start repository server

At this point, start the repository server, as this is needed for the remainder of the upgrade.

It is not needed anymore to start OpenJMS.

Upgrade Daisy Wiki schema

Run the daisy-wiki-init script to update the Daisy Wiki schema types:

cd <DAISY_HOME>/install

Create wikidata directory

From now on, configurations parameters, skins, doctype XSLs, the book publication types and book store, the wiki log files, in short all 'wiki-instance-specific' stuff is stored in a separate "wikidata directory" (not to be confused with the daisydata directory, which performs a similar role but for the repository server).

We will now create such a directory and copy your existing configuration to it.

To create the wikidata directory, go to <DAISY_HOME>/install and execute:


The program will ask you for a location for this directory. Just choose some location for this, for example put it next to (but not inside) the repository daisydata directory.

The program will also ask for the location of the repository daisydata directory, as it will take some configuration values from there (internal & jms user password) and put them in the wiki config.

It will also try to create the registrar user, but since you're doing an upgrade that user will already exist so it will simply skip that. However, you will need to enter the password for the registrar user. This can be found in <OLD_DAISY_HOME>/daisywiki/webapp/WEB-INF/cocoon.xconf, and look there for something like:

<registrarUser login="registrar" password="b6f84c35249a08d11ad6fa8279e2f9"/>

Just copy-and-paste that password to the input prompt.

Once the daisy-wikidata-init script has ended, we can copy over the remaining things:

  • If modified, copy <OLD_DAISY_HOME>/daisywiki/webapp/daisy/external-include-rules.xml to <wikidata dir>/external-include-rules.xml
  • Copy <OLD_DAISY_HOME>/daisywiki/webapp/daisy/sites/* (without the cocoon subdir) to <wikidata dir>/sites
  • If you created your own skins, copy <OLD_DAISY_HOME>/daisywiki/webapp/daisy/resources/skins/(yourskin) to <wikidata dir>/resources/skins
  • If you created your own doctype XSLs, then these also have to be copied into the wikidata directory, but at a new location, namely inside the directory of the skin. For example:
    <OLD_DAISY_HOME>/daisywiki/webapp/daisy/resources/document-styling/default/html/MyDoctype.xsl to <wikidata dir>/resources/skins/default/document-styling/html/MyDoctype.xsl
  • Ditto for query-styling XSLs: copy all contents from <OLD_DAISY_HOME>/daisywiki/webapp/daisy/resources/ to <wikidata dir>/resources/skins/<skin-name>/query-styling/
  • If you published any books, copy the from <OLD_DAISY_HOME>/daisywiki/webapp/daisy/bookstore to <wikidata dir>/bookstore
  • If you made any custom book publication types, you can copy these from <OLD_DAISY_HOME>/daisywiki/webapp/daisy/books/publicationtypes to <wikidata dir>/books/publicationtypes

If you made any configuration changes to the repository manager, JMS client or registrar component which were previously in the cocoon.xconf, then you can now find these in <wikidata dir>/daisy.xconf.

Updating wrapper configuration for the Daisy Wiki

If you are using the service wrapper script to launch the Daisy Wiki, you need to adjust the dsy_wiki_wrapper.conf to include the following parameter (see also the full script):


Daisy now ships with Jetty 5, which means that the classpath definition in the dsy_wiki_wrapper.conf needs to be updated. Just replace the existing wrapper.java.classpath entries with the new ones found in the updated dsy_wiki_wrapper.conf file.

The DAISYWIKI_DATADIR environment variable has to be declared in the shell script (dsy_wiki.sh or equivalent):

#! /bin/sh

DAISY_HOME=/home/daisy/daisy ; export DAISY_HOME
JETTY_HOME=$DAISY_HOME/daisywiki/jetty ; export JETTY_HOME
DAISYWIKI_DATADIR=/path/to/your/daisywikidata ; export DAISYWIKI_DATADIR


Start the Daisy Wiki

Now you can also start the Daisy Wiki.

After-upgrade notes

[optional] Image thumbnail generation

Daisy 1.5 can generate image thumbnails and extract their width and height to store them in fields of the Image document type. If you want to do this for all your current images, you can do so by running a document task. For this, first make sure you're logged in as Administrator, then go to the Document Tasks screen (the link is in the user menu) and create a new document task. Select the documents using the query:

select name where documentType = 'Image'

You might first want to try this with one document to see if everything is OK.

Then go to the next screen, and use the following script:

var document = repository.getDocument(variantKey, true);
document.setField("ImageWidth", new java.lang.Long(0));

This script "touches" the document by setting a field to a dummy value. Otherwise, the document wouldn't be changed and no save will be performed.

OpenJMS cleanup

Since OpenJMS is not used anymore, some cleanup you might want to to:

  • delete the openjms database
  • if globally set, remove the OPENJMS_HOME environment variable
  • remove the openjms wrapper script, if used

ActiveMQ authentication

With the procedure we followed above, ActiveMQ still uses default passwords. If the ActiveMQ port is publicly accessible, it is of course very much recommended to change these. Here is how. The password needs to be changed in the following locations:

Original setting:

In  this file:

in this line:

For the connection made by the Repository Server:

In this file:

in this line:
<credentials username="admin" password="jmsadmin"/>

For the connection made by the Daisy Wiki:

In this file:
<wikidata dir>/daisy.xconf

in this line:
<credentials username="admin" password="jmsadmin"/>

After making these changes, stop both the Daisy Wiki and the Repository Server, and then restart them (first the repository, then the wiki).

Note about static resources and caching

Since static resources are cached for about 5 hours by your browser, you might need to clear the browsers' cache.

In Firefox, this is done using Edit -> Preferences -> Privacy -> press the "Clear" button next to Cache

Internet Explorer users, see here.

If these instructions or unclear to you, or if you find an error in them, please share them with us on the Daisy mailing list or by leaving a comment.