Content
About
PolePosition is a benchmark test suite to compare
database engines and object-relational mapping technology. It is by no means
complete. Database vendors and open source database project contributors are
invited to improve the test implementations and to write further scenarios
("Circuits"). The PolePosition framework will help to implement tests
quickly and to time and output the results as number series and graph
visualizations.
The authors of the current test implementations do not see themselves as
experts for SQL, JDO, JPA, Hibernate or Datanucleus. If
you can provide a faster implementation for a circuit, please
supply it to the PolePosition project.
The results
show that there can not be one single "best product". Depending on
your application, some circuit results may be very important for you and some
may be irrelevant. If you consider basing your choice of database on
this benchmark,
download
the sources and bring along time to look at what every circuit does.
Results
We provide sample results from two different runs as PDF files:
Embedded Results
Database setups
with low resource consumption and low administration costs,
for instance for (mobile) devices.
Client/Server Results
Database setups with client/server functionality for scalable applications.
An explanation about each benchmark scenario ("Circuit") and what it does can be
found here.
You are encouraged to
download
the PolePosition suite and to run the tests yourself with your
favourite databases on your hardware, see Running the tests.
If you are evaluating databases for a concrete scenario, the best thing you can
do is to write your own customized circuit for this use case.
Interpretation
As already expressed above, there is no best product. The importance of
individual results will depend on the respective application. However we think
that we can draw some general conclusions from the results that may be valid
for everyone:
-
When you interpret the results, you should focus on significant differences,
where the results differ by an order of magnitude. If you see differences of
only 20% or 30% this may be an effect of Java garbage collection.
-
The longer the tests run (see the milliseconds values), the more
meaningful the results will be.
-
HSQLDB looks very good on many graphs, because much processing is done in
memory. We configured HSQLDB to use cached tables and with write_delay=false to be
fair to other databases that flush file buffers on commit. Before you consider to use HSQLDB for a complex database application, you
should take a close look at Complex:query. The query optimizer does not seem
to cope with a query over five tables.
-
The use of O-R mapping technology like Hibernate,
Datanucleus or OpenJPA has a
negative impact on performance. O-R mapping advocates may tell you that
caching will bring you back performance. Do not
forget: Data that comes from a cache can always be stale in a multi-user
application. If you want your data to be up-to-date, you have to reread from
the database.
-
As would be expected, object databases like db4o and Versant VOD provide best performance for
inheritance hierarchies, complex structures and navigational access to object
members.
-
If you take a look at the sources of the tests, they are also a measurement,
how easy it is to work against a specific database. SQL statements look like
aliens in Java code, ugly, difficult to write, debug and refactor. O-R mappers and
object databases make life so much easier and productive for developers. The task of storing
objects is very similar among all the products tested. A common
object-oriented query language, like LINQ, would be great for Java.
- NoSQL technologies outperform ORM mappers.
- Within NoSQL MongoDb does great until you start to introduce concurrency at which point it
begins to fall in performance compared to other solutions.
Download
All sources of the PolePosition test suite including some open source
database engines to run tests are available for download
here.
Running the tests
We recommend you to use the Eclipse IDE
to work with the test source code. The distribution contains a .project
file for Eclipse to get you started quickly.
The full test suite with all engines takes a full night to run on a fast machine.
Before your first run you may
like to take a look at the configuration possibilities, so you can first check
that all configured databases run through successfully.
Changing org.polepos.Settings.DEBUG to true will run all benchmarks with less
objects.
The main class to start the PolePosition suite is org.polepos.RunSeasonAfterEnhancing..
To run with all engines in one go, you need quite a lot of RAM. We ran with
-Xmx1500m
Here are the most important files, that you may want to customize:
-
org.polepos.RunSeason sets up the teams and the circuits. You can comment out
any circuit or team you don't want to run
-
/settings/Jdbc.properties
configures the JDBC databases and connection parameters that are to be used and
which JDBC databases Hibernate is to run with.
-
/settings/Jdo.properties
configures the JDO databases and connection parameters that are to be used.
-
/settings/Jpa.properties
configures the JPA databases and connection parameters to be used.
For JPA you also have to edit /src/META-INF/persistence.xml -
/settings/Circuits.properties
configures the settings for all circuits
-
/results/
is where the results of your test run will end up.
Open PolePosition.pdf from there when your test run is done.
-
/build.xmll
is a simple Ant script to help you compile and run the tests, if you prefer
Ant to running from within Eclipse.
Source Code
What
you may want to look for:
-
Circuit
a set of timed test cases that work against the same data
-
Lap
a single (timed) test
-
Team
a specific database category or engine that requires specific source code
-
Car
a specialized implementation of a team
-
Driver
an implementation of a circuit for a team
-
Since JDO, Hibernate
and JPA modify classes at compile/run-time, they need their own
set of persistent classes for the tests. Accordingly we did not consider it a good idea to
pass a set of objects from the outside into the write methods. Instead we give
all implementations full freedom, to do all the work on their own. By
collecting checksums for every timed task, the benchmark test makes sure that
no implementation cheats by accident.
License
PolePosition source code is supplied under the
GPL.