The Python Experiment Suite

The Python Experiment Suite is an open source software framework written in Python to execute and organize large scale automated software experiments with many useful features:

• Generic wrapper class with little overhead that works with any kind of experiment
• Multi-core support that automatically uses all the available hardware resources
• Automatically logs results in organized folder structure, ideal for a large number of experiments
• Experiments are defined in config files, with arbitrary parameter sets matching your experiments
• Parameter ranges can be evaluated without defining multiple experiments
• Different experiment types available, e.g. grid search for parameter combinations
• Supports experiment interruption and continuation, e.g. after a power failure
• Log file parser and result retrieval through the included Python API
• Various run-time options for progress indication, experiment selection, etc.

The Python Experiment Suite is available for download on the github repository.

Running experiments in the Python Experiment Suite only takes a few easy steps.

1. Create a class and derive it from the PyExperimentSuite class
2. Add object creation and a call to the suite's start() method at the bottom of the script
3. Implement the reset() and iterate() methods
4. Define your experiments and the parameters you want to use in the configuration file
5. Run the script from the command line

from expsuite import PyExperimentSuite

class MySuite(PyExperimentSuite):

  def reset(self, params, rep):
    """ for this simple example, nothing needs to be loaded or initialized."""
    pass

  def iterate(self, params, rep, n):
    # access config file parameters through the params dictionary
    alpha = params['alpha']
    beta = params['beta']

    # your experiment code goes here...

    # return any results you want to store in log files
    ret = {'rep':rep, 'iter':n, 'alpha':alpha, 'beta':beta}
    return ret        

if __name__ == '__main__':
  # create object and call start method
  mysuite = MySuite()
  mysuite.start()
  

[DEFAULT]
repetitions = 5
iterations = 10
path = results

[myfirstexp]
alpha = 1
beta = 0.5
  

web-phd  2008/09 © Thomas Rückstieß