Product Testing
Obviously, this is the most important part of our development process.
If you do buy a binary license from us, we want you to feel confident that it
will perform as expected. Our modified ExtSQL daemon should be just a simple
'drop-in' change to your existing MySQL or PostgreSQL environment.
To insure reliability almost the entire process is automated. If we
have done our job properly, you shouldn't notice any differences. Here
are some details on how our internal process works.
| Building/Testing
ExtSQL versions (using MySQL as an example) |
Here are the steps taken before a binary is released for a specific
combination of MySQL, i.e. a combination of MySQL Version, OS, and architecture:
- We start with a binary distribution from the
MySQL web site for the Architecture, OS,
& MySQL version we plan on building. This binary distirbution (0) executable is automatically tested using a combination of
the MySQL distribution benchmarks and some of our own tests which run
multiple queries in parallel from different hosts. Testing is
done against both MyISAM and Innodb engines. This binary data is recorded
to be used for comparison purposes later.
- Depending on the OS we next use a source distribution from
the main MySQL site.
We initially use the same configure options that were used in the MySQL
binary distribution. These are labelled as 'STANDARD' in our download system.
We then build a level (0.0) baseline executable on a system configured similarly to step
1.
- The (0.0) executable, similar to the baseline (0), is automatically tested using a combination of
the MySQL distribution benchmarks and some of our own tests which run
multiple queries in parallel from different hosts. The baseline build (0.0) should perform identically
to the binary (0) distribution. Testing is
done against both MyISAM and Innodb engines. The data is again recorded
to be used for comparison purposes.
- A comparison is done between the data collected for the (0) and (0.0)
versions. We expect to see practically no differences.
- We next do another clean build and apply our patches before any
'configure' is done. This (1.0) build is again done under automatic control
to help guarantee a correct configuration. With ExtSQL features
disabled, we again run the same tests used for the baseline build and record
the results.
- A comparison is then done between the baseline (0) and patch (1.0) results to
insure test success and also that no performance changes were noted. We
expect to see almost identical results (while disabled the performance
impact of the patches is < 1%).
- Extended statistics are then activated with a default set of variables
to be tracked. The prior benchmarks are again run and data recorded.
- A comparison is then made to check for not only success, but also to
measure performance impact, and to check that the statistics are reporting
correct results (regression testing). While tracking 18 different status
variables the increase in server load was measured at approximately 5%.
- The binary is then certified as ready for your use for that combination
of MySQL version, OS, and Architecture, e.g. MySQL 4.1.22 on RHEL 4, x86_64.
There is always the possibility of a surprise. Because we are
building hundreds of different combinations of MySQL, OS, and Architectures --
we take care to track configurations and bugs carefully. Our
key to reading versions helps you understand
how we name our downloads and how bugs are tracked. To give you confidence
in the product, our installation instructions make it
clear that we expect you to start with a working installation on either
Unix/Linux or Windows platforms. We also give you the ability to
vote for specific versions/configurations you would
like to see supported.
Installing the extensions means just shutting down your server, replace the
executable with our binary, and restarting the server (with extended statistics
disabled). If problems start to occur, you can just re-install your
original binary, and we know that there was a problem or configure difference
with our binary.
Let us know if you have any questions!
|
