The Python Experiment Suite

The Python Experiment Suite is an open source software framework written in
Python to execute and organize large scale automated software experiments with many useful features:
- Generic wrapper class with little overhead that works with any kind of experiment
- Multi-core support that automatically uses all the available hardware resources
- Automatically logs results in organized folder structure, ideal for a large number of experiments
- Experiments are defined in config files, with arbitrary parameter sets matching your experiments
- Parameter ranges can be evaluated without defining multiple experiments
- Different experiment types available, e.g. grid search for parameter combinations
- Supports experiment interruption and continuation, e.g. after a power failure
- Log file parser and result retrieval through the included Python API
- Various run-time options for progress indication, experiment selection, etc.
Download
The Python Experiment Suite is available for download on the
github repository.
Usage
Running experiments in the Python Experiment Suite only takes a few
easy steps.
- Create a class and derive it from the
PyExperimentSuite
class
- Add object creation and a call to the suite's
start()
method at the bottom of the script
- Implement the
reset()
and iterate()
methods
- Define your experiments and the parameters you want to use in the configuration file
- Run the script from the command line
Below is a very simple working example on how to implement an experiment. Create a file called "suite.py"
with the code below:
from expsuite import PyExperimentSuite
class MySuite(PyExperimentSuite):
def reset(self, params, rep):
""" for this simple example, nothing needs to be loaded or initialized."""
pass
def iterate(self, params, rep, n):
# access config file parameters through the params dictionary
alpha = params['alpha']
beta = params['beta']
# your experiment code goes here...
# return any results you want to store in log files
ret = {'rep':rep, 'iter':n, 'alpha':alpha, 'beta':beta}
return ret
if __name__ == '__main__':
# create object and call start method
mysuite = MySuite()
mysuite.start()
The configuration file "experiments.cfg" needs to be in the same directory and contain
the following lines:
[DEFAULT]
repetitions = 5
iterations = 10
path = results
[myfirstexp]
alpha = 1
beta = 0.5
Now you can run the script from the command line:
python suite.py
.
After execution, the script will have created a directory called "results" with a
sub-directory called "myfirstexp" (the experiment name from the config file) and
5 log files, each with 10 lines.
Another example is available in the Snippets section of this website.
It will show a few more features, like the optional restore support. For a full description of all features, check out
the
documentation pdf.