Building eXist-db

1. Building a WAR from the Standard Distribution

If you would like to deploy eXist into an existing servlet engine (e.g. tomcat), you will need a .war file. We no longer distribute a .war build, but it is easy to create one from the standard distribution by repackaging (not building) it:

  1. Install the standard distribution

    Get a standard distribution from the download page. Install it into a directory of your choice. Make sure you include the source package.

  2. Generate the WAR

    Open a command line prompt and change to the directory where you just installed eXist. Call

    build.sh dist-war

    or

    build.bat dist-war

    once. This will generate a .war archive into directory dist.

Note that generating a WAR archive does only repackage the files in the distribution, so if the distribution worked properly for you, the WAR should as well. You don't need to recompile eXist or sign any jars.

2. Building eXist from SVN

Building eXist is quite easy and requires 3 simple steps:

  1. Check out from SVN

    The SVN URL for the development trunk is:

    https://exist.svn.sourceforge.net/svnroot../trunk/eXist

    You need an SVN client to check out the source code. Examples for SVN clients are Tortoise SVN on the windows platform or the command line SVN tools on Unix (see below). The oXygen XML editor ships with its own SVN client.

  2. Set JAVA_HOME

    Before starting the build, your JAVA_HOME environment variable should be set to point to the root directory of the JAVA JDK (note: JDK, not JRE!). Open a console or a DOS box on windows and type:

    set JAVA_HOME=c:\path\to\jdk

    on windows or

    export JAVA_HOME=/path/to/jdk

    on unix.

  3. Build eXist

    Change into the eXist directory you just checked out and call

    build.bat

    on windows or on unix:

    build.sh

3. Using the Build System

You can rebuild all eXist packages from the standard distribution or the subversion repository. You can even create a new distribution from the one you installed. For example, assume you installed eXist using the installer GUI, but you need to have an exist.war package for deployment within a different servlet engine. Just call the main build script (build.sh or build.bat) and pass it the target dist-war:

build.sh dist-war

This will create a fresh .war archive in the dist directory.

Note

To be able to use the build system, you need to include the "source" module when selecting installable packages in the installer GUI.

eXist relies on Ant for all compile and build tasks. Ant is included in the distribution (directory tools/ant). As already mentioned above, there are two shell scripts to start Ant: build.sh or build.bat. The shell scripts will take care to create a proper command line to launch Java with the Ant version included in the distribution.

Calling Ant without further argument will compile the sources and create the main jar-files (exist.jar, exist-optional.jar, start.jar).

build.sh -p

lists the available build targets. Passing one of these as argument to ant will call execute the corresponding target. The main targets are:

Useful Build Targets

all

Compile the sources and create the main jar-files (exist.jar, exist-optional.jar, start.jar).

backrest Create backrest zip
benchmark Performs the benchmark tests for eXist.
clean Cleans the files and directories generated by a previous build.
clean-all Cleanup deeper
clean-conf Cleanup config file
compile Compiles the source code
compile-tomcat-realm Compile the Tomcat realm
dist-tgz Packages eXist into a .tgz file
dist-war Packages eXist into a .war file for deployment within a servlet engine
dist-zip Packages eXist into a .zip file
docs Creates a static HTML version of the documentation into docs.zip
download-additional-jars Download optional third-party jar files. These are currently: icu4j and saxon. These are downloaded into lib/user/
download-xqts Download XQTS file.
download-xslts Download XSLTS file.
extensions Creates the necessary .jar files for eXist's extensions. Read extensions/modules/build.properties to select the extensions to build.
installer Creates a new installer package based on IzPack
installer-exe Create .exe from installer
jar The default: compiles the source and creates eXist's main .jar files
javadocs Creates the API docs into webapp/api
javadocs-uml

Creates the API docs along with UML diagrams into webapp/api

Note

This build target takes a long time to run, because it is generating graphics for every package in eXist.

jdepend JDepend traverses Java class file directories and generates design quality metrics for each Java package.
jetty-keygen Generate keystore for jetty.
jnlp-clean
jnlp-keygen Generate keystore for signing jars.
jnlp-pack200 Pack all jar files.
jnlp-sign-all This signs all of the jar files, so that the webstart of the eXist client Java application will launch without any signing errors.
jnlp-sign-core Sign all CORE jar files.
jnlp-sign-exist Sign all EXIST jar files.
jnlp-unsign-all This unsigns all of the jar files.
pmd

PMD scans Java source code and looks for potential problems like:

Possible bugs

empty try/catch/finally/switch statements

Dead code

unused local variables, parameters and private methods

Suboptimal code

wasteful String/StringBuffer usage

Overcomplicated expressions

unnecessary if statements, for loops that could be while loops

Duplicate code

copied/pasted code means copied/pasted bugs

rebuild Performs a clean and the all
samples Builds the examples and packs them into examples.jar
snapshot-installer Creates a snapshot installer
svn-diff Makes a patch file.
svn-download This downloads a jar file that places the current revision number of the working copy of eXist into VERSION.txt. This allows for the tagging of the build with the revision number.
svn-update Update project from SubVersion archive.
test Runs all tests in the test suite and outputs the HTML result into test/junit/html. This may take several minutes.
xquery Regenerate the XQuery parser code with ANTLR (only required if you change src/org../xquery/parser/XQuery.g).

Note

You might want to set some custom settings (in particular your HTTP proxy when external libraries are required) before running build.sh or build.bat. See build.properties.

4. Using the Subversion Repository

Since spring 2006, the entire code base of eXist is managed by the Subversion repository at Sourceforge. Before that we used CVS, which had a few issues, especially due to the separation into a developer's and anonymous server. Using Subversion should be much simpler, faster and more reliable.

Fortunately, you don't need to be a hardcore Java developer to use the SVN version of eXist. In general, we try to keep the development version in a consistent state. The code should always compile and run the test cases, though temporary failures are possible, especially while a larger update is in progress.

The repository root URL is:

https://exist.svn.sourceforge.net/svnroot/exist

Important

Please note you have to use https, not http.

The current development trunk is at:

https://exist.svn.sourceforge.net/svnroot../trunk/eXist

This is the URL you will normally use for a checkout of the main eXist development tree.

4.1. Checking Out from the Repository

There are many Subversion clients to choose from. If you want to actively contribute to eXist or otherwise work on eXist's source code, we recommend to use eclipse to check out the sources (see below) and setup a project. Otherwise, you can use any other SVN client. On Linux, I prefer the plain command line. For example, the following command checks out the main development trunk into a directory called eXist-trunk:

svn co https://exist.svn.sourceforge.net/svnroot../trunk/eXist eXist-trunk

A good choice on the Windows platform is Tortoise SVN. Tortoise is an extension to the windows shell. The following screenshot shows the checkout dialog:

The repository already contains project configuration files for eclipse, so setting up the project should be pretty simple: you normally don't need to change much in order to dive into the source code (though you may need some customizations to build and run directly from within eclipse).

To access a Subversion repository from within eclipse, you need to install the subclipse plugin first! Please follow the install instructions available there.

Once you installed subclipse, change to the perspective: "SVN Repository Exploring". In the "SVN Repository" tab of the left view, right click into the empty space and select New, then Repository Location. In the appearing dialog, enter the URL of the eXist development trunk:

The URL will now be listed in the "SVN Repository" view. Right click the URL and select Checkout. A progress window appears, saying that eclipse is "getting remote info". Once it finished, you should see the following dialog:

Just press Finish unless you want to specify a different directory or checkout a certain tag.

5. Signing jar files for webstart

The database can be easily accessed with the Java client, when the client is launched by clicking the Java Webstart "Launch" button on the right side of this page. Java webstart explicilty requires that all application code, the 'jar' files, are signed using a security certificate. Out of the box, when eXist-db is installed using the installer, all relevant jar files have already been signed, so webstart can be used directly. It is nessecary however to (re-)sign the jar files yourself if you want to (re-)build the eXist-db java code. This chapter explains how to do this.

5.1. Quickstart

To get started quickly, just execute the following command:

./build.sh -f build/scripts/jarsigner.xml

This command sequentially creates the key.store file and signs a selection of jar files:

Example: Jarsigner.xml Output

./build.sh -f build/scripts/jarsigner.xml
Starting Ant...

Buildfile: ..../eXist-db/build/scripts/jarsigner.xml

jnlp-prepare:

jnlp-keygen:
[genkey] Generating Key for exist

jnlp-sign-exist:
[signjar] Signing JAR: ..../eXist-db/exist-fluent.jar to ..../eXist-db/exist-fluent.jar as exist
[signjar] Signing JAR: ..../eXist-db/exist-optional.jar to ..../eXist-db/exist-optional.jar as exist
[signjar] Signing JAR: ..../eXist-db/exist.jar to ..../eXist-db/exist.jar as exist
[signjar] Signing JAR: ..../eXist-db/start.jar to ..../eXist-db/start.jar as exist

jnlp-sign-core:
[signjar] Signing JAR: ..../eXist-db/lib/core/antlr-2.7.7.jar to ..../eXist-db/lib/core/antlr-2.7.7.jar as exist
[signjar] Signing JAR: ..../eXist-db/lib/core/commons-collections-3.2.1.jar to ..../eXist-db/lib/core/commons-collections-3.2.1.jar as exist
....
[signjar] Signing JAR: ..../eXist-db/lib/core/xmlrpc-server-3.1.3.jar to ..../eXist-db/lib/core/xmlrpc-server-3.1.3.jar as exist

Note that this command needs to be executed just once. In subsequent builds newly generated jars are signed automatically.

5.2. Ant targets

An overview of all available ant targets can be retrieved by executing the following command:

./build.sh -projecthelp -f build/scripts/jarsigner.xml

The following ant targets are available for signing jar files:

jnlp-all

Create keystore file and sign all EXIST and CORE jar files. (Default task)

jnlp-keygen

Generate a new keystore file if not present.

jnlp-clean

Delete the keystore file.

jnlp-sign-core

Sign all CORE jar files in lib/core.

jnlp-sign-exist

Sign all EXIST jar files, e.g. exist.jar, exist-XX.jar and start.jar

jnlp-unsign-all

Unsign all jar files.

The target jnlp-keygen generates a new certificate file key.store. If this file is present the EXIST jar files are signed during each following build. The process of signing jar files can be stopped by removing this keystore file.

The ant script contains two additional targets: jetty-keygen which generates a keystore file for the jetty server and jnlp-pack200 which processes the signed jar files with pack200 technology. More about the latter in the next section.

5.3. Reducing jar sizes with pack200 compression

Pack200 is a compression technology introduced in Java 1.5.0. It has been designed for compressing jar files and works most efficiently on Java class files. Pack200 can reduce the size of a jar file up to 60% resulting into a significant reduction of the amount of downloaded webstart data.

The webstart jar files can be compressed with pack200 with the following command:

./build.sh -f build/scripts/jarsigner.xml jnlp-pack200
This command consecutively performs the following steps for all jar files:

Example: pack200 Output


./build.sh -f build/scripts/jarsigner.xml jnlp-pack200
Starting Ant...

Buildfile: .../eXist-db/build/scripts/jarsigner.xml

jnlp-pack200:

jnlp-unsign-all:
[unsignjar] Unsigning exist-fluent.jar
[unsignjar] Unsigning exist-optional.jar
[unsignjar] Unsigning exist.jar
[unsignjar] Unsigning start.jar
[unsignjar] Unsigning antlr-2.7.7.jar
...
[unsignjar] Unsigning xmlrpc-server-3.1.3.jar
[repack] Normalizing exist-fluent.jar
[repack] Normalizing exist-optional.jar
[repack] Normalizing exist.jar
[repack] Normalizing start.jar
[repack] Normalizing antlr-2.7.7.jar
...
[repack] Normalizing xmlrpc-server-3.1.3.jar

jnlp-prepare:

jnlp-keygen:

jnlp-sign-exist:
[signjar] Signing JAR: .../eXist-db/exist-fluent.jar to .../eXist-db/exist-fluent.jar as exist
[signjar] Signing JAR: .../eXist-db/exist-optional.jar to .../eXist-db/exist-optional.jar as exist
[signjar] Signing JAR: .../eXist-db/exist.jar to .../eXist-db/exist.jar as exist
[signjar] Signing JAR: .../eXist-db/start.jar to .../eXist-db/start.jar as exist

jnlp-prepare:

jnlp-keygen:

jnlp-sign-core:
[signjar] Signing JAR: .../eXist-db/lib/core/antlr-2.7.7.jar to .../eXist-db/lib/core/antlr-2.7.7.jar as exist
...
[signjar] Signing JAR: .../eXist-db/lib/core/xmlrpc-server-3.1.3.jar to .../eXist-db/lib/core/xmlrpc-server-3.1.3.jar as exist
[pack] Packing exist-fluent.jar
[pack] Packing exist-optional.jar
[pack] Packing exist.jar
[pack] Packing start.jar
[pack] Packing antlr-2.7.7.jar
...
[pack] Packing xmlrpc-server-3.1.3.jar

BUILD SUCCESSFUL
Total time: 1 minute 0 seconds

The pack200 compression feature works with Java SE 6 update 10 (Java6u10) or newer. Older Java6 versions will only receive the larger jar files.

5.4. References

5.5. Notes

September 2009
Wolfgang M. Meier
wolfgang at exist-db.org
Leif-Jöran Olsson
ljo at exist-db.org