PolePosition is a benchmark test suite to compare
database engines and object-relational mapping technology. It is by no means
complete. Database vendors and open source database project contributors are
invited to improve the test implementations and to write further scenarios
("Circuits"). The PolePosition framework will help to implement tests
quickly and to time and output the results as number series and graph
The authors of the current test implementations do not see themselves as
experts for SQL, JDO, JPA, Hibernate or Datanucleus. If
you can provide a faster implementation for a circuit, please
supply it to the PolePosition project.
show that there can not be one single "best product". Depending on
your application, some circuit results may be very important for you and some
may be irrelevant. If you consider basing your choice of database on
the sources and bring along time to look at what every circuit does.
We provide sample results from two different runs as PDF files:
with low resource consumption and low administration costs,
for instance for (mobile) devices.
Database setups with client/server functionality for scalable applications.
An explanation about each benchmark scenario ("Circuit") and what it does can be
You are encouraged to
the PolePosition suite and to run the tests yourself with your
favourite databases on your hardware, see Running the tests.
If you are evaluating databases for a concrete scenario, the best thing you can
do is to write your own customized circuit for this use case.
As already expressed above, there is no best product. The importance of
individual results will depend on the respective application. However we think
that we can draw some general conclusions from the results that may be valid
When you interpret the results, you should focus on significant differences,
where the results differ by an order of magnitude. If you see differences of
only 20% or 30% this may be an effect of Java garbage collection.
The longer the tests run (see the milliseconds values), the more
meaningful the results will be.
HSQLDB looks very good on many graphs, because much processing is done in
memory. We configured HSQLDB to use cached tables and with write_delay=false to be
fair to other databases that flush file buffers on commit. Before you consider to use HSQLDB for a complex database application, you
should take a close look at Complex:query. The query optimizer does not seem
to cope with a query over five tables.
The use of O-R mapping technology like Hibernate,
Datanucleus or OpenJPA has a
negative impact on performance. O-R mapping advocates may tell you that
caching will bring you back performance. Do not
forget: Data that comes from a cache can always be stale in a multi-user
application. If you want your data to be up-to-date, you have to reread from
As would be expected, object databases like db4o and Versant VOD provide best performance for
inheritance hierarchies, complex structures and navigational access to object
If you take a look at the sources of the tests, they are also a measurement,
how easy it is to work against a specific database. SQL statements look like
aliens in Java code, ugly, difficult to write, debug and refactor. O-R mappers and
object databases make life so much easier and productive for developers. The task of storing
objects is very similar among all the products tested. A common
object-oriented query language, like LINQ, would be great for Java.
- NoSQL technologies outperform ORM mappers.
- Within NoSQL MongoDb does great until you start to introduce concurrency at which point it
begins to fall in performance compared to other solutions.
All sources of the PolePosition test suite including some open source
database engines to run tests are available for download
Running the tests
We recommend you to use the Eclipse IDE
to work with the test source code. The distribution contains a .project
file for Eclipse to get you started quickly.
The full test suite with all engines takes a full night to run on a fast machine.
Before your first run you may
like to take a look at the configuration possibilities, so you can first check
that all configured databases run through successfully.
Changing org.polepos.Settings.DEBUG to true will run all benchmarks with less
The main class to start the PolePosition suite is org.polepos.RunSeasonAfterEnhancing..
To run with all engines in one go, you need quite a lot of RAM. We ran with
Here are the most important files, that you may want to customize:
org.polepos.RunSeason sets up the teams and the circuits. You can comment out
any circuit or team you don't want to run
configures the JDBC databases and connection parameters that are to be used and
which JDBC databases Hibernate is to run with.
configures the JDO databases and connection parameters that are to be used.
configures the JPA databases and connection parameters to be used.
For JPA you also have to edit /src/META-INF/persistence.xml
configures the settings for all circuits
is where the results of your test run will end up.
Open PolePosition.pdf from there when your test run is done.
is a simple Ant script to help you compile and run the tests, if you prefer
Ant to running from within Eclipse.
you may want to look for:
a set of timed test cases that work against the same data
a single (timed) test
a specific database category or engine that requires specific source code
a specialized implementation of a team
an implementation of a circuit for a team
Since JDO, Hibernate
and JPA modify classes at compile/run-time, they need their own
set of persistent classes for the tests. Accordingly we did not consider it a good idea to
pass a set of objects from the outside into the write methods. Instead we give
all implementations full freedom, to do all the work on their own. By
collecting checksums for every timed task, the benchmark test makes sure that
no implementation cheats by accident.
PolePosition source code is supplied under the